This solves the issue of complex categorical modeling but makes slot-filling dependent on an intent detector.

we propose a framework that treats slot-filling as a fully intentagnostic span extraction problem.


Second, the train-test splits of NICHE dataset contain same CNs since the splitting has been done using one paraphrase for each HS and its all original CNs, while CROWD train-test splits have a similar property since an exact same CN can be found for many different HSs.

the non-pretrained transformer models, which are more prone to generating an exact sequence of text from the training set, show a relatively better performance with the standard metrics in comparison to the advanced pre-trained models.


In fact, we observed that, after the output CN, the over-generated chunk of text consists of semantically coherent brand-new HS-CN pairs, marked with proper HS/CN start and end tokens consistent with the training data representation.

on top of CN generation for a given HS, we can also take advantage of the over-generation capabilities of GPT-2, so that the author module can continuously output plausible HS-CN pairs without the need to provide the HS to generate the CN response.


Although it seems more intuitive to focus on precision since we search for an effective filtering over many possible solutions, we observed that a model with a very high precision tends to overfit on generic responses, such as "Evidence please?".

we aim to keep the balance between the precision and recall and we opted for F1 score for model selection.


The main goal of our effort is to reduce the time needed by experts to produce training data for automatic CN generation.

the primary evaluation measure is the average time needed to obtain a proper pair.


We performed some manual analysis of the selected CNs and we observed that especially for the Reviewer≥2 case (which was the most problematic in terms of RR and novelty) there was a significantly higher ratio of “generic” responses, such as “This is not true.” or “How can you say this about an entire faith?”, for which reviewers agreement is easier to attain.

the higher agreement on the generic CNs reveals itself as a negative impact in the diversity and novelty metrics.


In this case, generalizability of evaluation results becomes questionable.

our evaluation methodology needs to fulfill the following two requirements: (1) evaluation must not be performed on translational equivalents of the Source entries to which the model already had access during training (e.g., Sonnenschein and nuklear in our example from Figure 1); but, on the other hand, (2) a reasonable number of instances must be available for evaluation (ideally, as many as possible to increase reliability).


zh1 was created and is distributed using traditional Chinese characters, whereas the embedding model by Grave et al. (2018) employs simplified ones.

we converted zh1 into simplified characters using GOOGLE TRANSLATE 6 prior to evaluation.


Note that some variants also produce different top MT outputs (o), as they were trained using different architectures or decoding algorithms.

we have four sets of DA annotations collected for 400 segments for system variants with different MT outputs: standard Transformer, Transformer with diverse beam search, MoE and ensembling.


Each domain has a set of slots; each slot can be assigned a value of the right type, a special DONTCARE marker indicating that the user has no preference, or a special “?” marker indicating the user is requesting information about that slot.

we can summarize the content discussed up to any point of a conversation with a concrete state, consisting of an abstract state, and all the slot-value pairs mentioned up to that point.


For training time, ATS is roughly half of the multi-task methods on both Zh2En and En2Zh tasks.

compared with the multi-task methods, ATS can significantly reduce the model size and improve the training efficiency.


Furthermore, JNC does not provide full-text articles but only lead three sentences.

we take the latter strategy, removing non-entailment pairs from the supervision data for headline generation.


Recently, pretrained language models such as BERT (Devlin et al., 2019) show remarkable advances in the task of recognizing textual entailment (RTE) 8 .

we fine-tune pretrained models on the supervision data for entailment relation between source documents and their headlines.


However, no large-scale Japanese corpus for semantic inference (counterpart to MultiNLI) is available.

we created supervision data for entailment relation between lead three sentences and headlines (lead3headline, hereafter) on JNC.


Furthermore, we would like to confirm whether the filtering strategy can improve the truthfulness of the model.

we also report the support score, the ratio of entailment relation between source documents and generated headlines measured by the entailment classifiers (explained in Section 4.1), and human evaluation about the truthfulness.


Those methods construct the representations of the context and response with a single vector space.

the models tend to select the response with the same words .


We treat these as positive instances, making the tacit assumption that in the data the agent's reply is always relevant given a user utterance.

the data lacks negative examples of irrelevant agent responses.


The original data is formatted as (dialogue, question, answer), which is not directly suitable for our goal since chatbots only concern about how to respond contexts instead of answering an additional question.

we ask human annotators to rewrite the question and answer candidates as response candidates.


Furthermore, the divide-and-conquer strategy greatly reduces the search space but introduces the projectivity restriction, which we remedy with a transition-based reordering system.

the proposed linearizer outperforms the previous state-of-the-art model both in quality and speed.


Without enough training examples, the classifier can hardly tell which relation the entity participates in.

the extracted triples are usually incomplete and inaccurate.


For example, the relation "Work in" does not hold between the detected subject "Jackie R. Brown" and the candidate object "Washington".

the object tagger for relation "Work in" will not identify the span of "Washington", i.e., the output of both start and end position are all zeros as shown in Figure 2.


TRADE-OFF relations express a problem space in terms of mutual exclusivity constraints between competing demands.

tradeoffs play a prominent role in evolutionary thinking (Agrawal et al., 2010) and are the principal relation under investigation in a significant portion of biology research papers (Garland, 2014).


Negative samples are important because possible trigger words can be contiguous, e.g., the phrase 'negative correlation' denotes a TRADE-OFF relation, whereas 'correlation' by itself does not.

the annotation of training examples is harder, and lexical and syntactic patterns that correctly signify the relation are sparse (Peng et al., 2017).


Previous methods primarily encode two arguments separately or extract the specific interaction patterns for the task, which have not fully exploited the annotated relation signal.

we propose a novel TransS-driven joint learning architecture to address the issues.


Different from TransE, we could not directly utilize TransS to recognize discourse relations, for that each argument could not be reused in discourse.

we exploit TransS to mine the latent geometric structure information and further guide the semantic feature learning.


However, the results imply that with the more encoder layers considered, the model could incur the over-fitting problem due to adding more parameters.

we adopt three encoder layers to encode the arguments as our Baseline in section 3.3.


However, comparable studies have yet to be performed for neural machine translation (NMT).

it is still unclear whether all translation directions are equally easy (or hard) to model for NMT.


In summary, BLEU only allows us to compare models for a fixed target language and tokenization scheme, i.e. it only allows us to draw conclusions about the difficulty of translating different source languages into a specific target one (with downstream performance as a proxy for difficulty).

bLEU scores cannot provide an answer to which translation direction is easier between any two source-target pairs.


This small-scale manual analysis hints that DA scores are a valid proxy for CLDA.

we decided to treat them as reliable scores for our setup and evaluate our proposed metrics by comparing their correlation with DA scores.


showed that SANs in machine translation could learn word order mainly due to the PE, indicating that modeling cross-lingual information at position representation level may be informative.

we propose a novel cross-lingual PE method to improve SANs.


Our proposed model is motivated by the observation that although every sentence in the training data has a domain label, a word in the sentence does not necessarily only belong to that single domain.

we assume that every word in the vocabulary has a domain proportion, which indicates its domain preference.


We re- mark that the Transformer model, though does not have any explicit recurrent structure, handles the sequence through adding additional positional embedding for each word (in conjunction with sequential masking).

if a word appears in different positions of a sentence, its corresponding embedding is different.


Recall that the Transformer model contains multiple multi-head attention modules/layers.

our proposed model inherits the same architecture and applies the word-level domain mixing to all these attention layers.


This is because the domain proportions are determined by the word embedding, and the word embedding at top layers is essentially learnt from the representations of all words at bottom layers.

when the embedding of a word at some attention layer is already learned well through previous layers (in the sense that it contains sufficient contextual information and domain knowledge), we no longer need to borrow knowledge from other domains to learn the embedding of the word at the current layer.


Training without domain labels shows a slight improvement over baseline, but is still significantly worse than our proposed method for most of the tasks.

we can conclude that our proposed domain mixing approach indeed improves performance.


For example, in the law domain, we find that "article" often appears at the beginning of a sentence, while in the media domain, the word "article" may appear in other positions.

varying domain proportions for different positions can help with word disambiguation.


We need to explicitly "teach" the model where to copy and where to generate.

to provide the model accurate guidance of the behavior of the switch, we match the target text with input table values to get the positions of where to copy.


In real-world problems, retrieval response sets usually have many more than 10 candidates.

we further test the selection and binary models on a bigger reconstructed test set.


With NOTA options in the training data, the models learn to sometimes predict NOTA as the best response, resulting in more false-positive isNOTA predictions at inference time.

also, by replacing various ground truths and strong distractors with NOTa, the model has fewer samples to help it learn to distinguish between different ground truths and strong distractors/ it performs less well on borderline predictions (scores close to the threshold).


ConceptFlow learns to model the conversation development along more meaningful relations in the commonsense knowledge graph.

the model is able to "grow" the grounded concepts by hopping from the conversation utterances, along the commonsense relations, to distant but meaningful concepts; this guides the model to generate more informative and on-topic responses.


Softmax: We will discuss in §2.3 that annotators are expected to miss a few good responses since good and bad answers are often very similar (may only differ by a single preposition or pronoun).

we explore a ranking objective that calculates errors based on the margin with which incorrect responses are ranked above correct ones (Collins and Koo, 2005).


there can be at most 5 unique annotated responses for each question.


Also, due to the lack of human-generated references in SQuAD-dev-test, we cannot use other typical generation based automatic metrics.

we use Amazon Mechanical Turk to do human evaluation.


While outputting answer-phrase to all questions is trivially correct, this style of response generation seems robotic and unnatural in a prolonged conversation.

we also ask the annotators to judge if the response is a completesentence (e.g. “it is in Indiana”) and not a sentencefragment (e.g. “Indiana”).


This strong dependence on labeled data largely prevents neural network models from being applied to new settings or real-world situations due to the need of large amount of time, money, and expertise to obtain enough labeled data.

semi-supervised learning has received much attention to utilize both labeled and unlabeled data for different learning tasks, as unlabeled data is always much easier and cheaper to collect (Chawla and Karakoulas, 2011).


Despite the huge success of those models, most prior work utilized labeled and unlabeled data separately in a way that no supervision can transit from labeled to unlabeled data or from unlabeled to labeled data.

most semisupervised models can easily still overfit on the very limited labeled data, despite unlabeled data is abundant.


By model latency analysis 2 , we find that layer normalization (Ba et al., 2016) and gelu activation (Hendrycks and Gimpel, 2016) accounted for a considerable proportion of total latency.

we propose to replace them with new operations in our MobileBERT.


Progressive Knowledge Transfer One may also concern that if MobileBERT cannot perfectly mimic the IB-BERT teacher, the errors from the lower layers may affect the knowledge transfer in the higher layers.

we propose to progressively train each layer in the knowledge transfer.


Furthermore, none of the curves exhibit any signs of convergence even after drawing orders of magnitude more samples (Figure 3); the estimated model perplexities continue to improve.

the performance of these models is likely better than the originally reported estimates.


While this work helps clarify and validate existing results, we also observe that none of the estimates appear to converge even after drawing large numbers of samples.

we encourage future research into obtaining tighter bounds on latent LM perplexity, possibly by using more powerful proposal distributions that consider entire documents as context, or by considering methods such as annealed importance sampling.


Yet, in a semi-supervised learning setting where we already have GT labels, we need novel QA pairs that are different from GT QA pairs for the additional QA pairs to be truly effective.

we propose a novel metric, Reverse QAE (R-QAE), which is low if the generated QA pairs are novel and diverse.


However, QAE only measures how well the distribution of synthetic QA pairs matches the distribution of GT QA pairs, and does not consider the diversity of QA pairs.

we propose Reverse QA-based Evaluation (R-QAE), which is the accuracy of the QA model trained on the human-annotated QA pairs, evaluated on the generated QA pairs.


We tune each layer for n epochs and restore model to the best configuration based on validation loss on a held-out set.

the model retains best possible performance from any iteration.


However, a key limitation of prior work is that authorship obfuscation methods do not consider the adversarial threat model where the adversary is "obfuscation aware" (Karadzhov et al., 2017;Mahmood et al., 2019).

in addition to evading attribution and preserving semantics, it is important that authorship obfuscation methods are "stealthy" -i.e., they need to hide the fact that text was obfuscated from the adversary.


The quality and smoothness of automated text transformations using the state-of-the-art obfuscators differ from that of human written text (Mahmood et al., 2019).

the intuition behind our obfuscation detectors is to exploit the differences in text smoothness between human written and obfuscated texts.


we use neural language models with deep architectures and trained on large amounts of data which are better at identifying both long-term and short-term context.


The evaded documents are those where the modification strategy somehow crossed an implicit threshold for evading authorship attribution.

we surmise that the evaded documents are likely to be relatively less smooth.


We have no real world scenario to mimic in that we have not encountered any real world use of automated obfuscators and their outputs.

we make the datasets under a reasonable assumption that original documents are in the vast majority.


However, without the audio recordings, proficiency scoring must be performed based on the text alone.

robust methods for text-only speech scoring need to be developed to ensure the reliability and validity of educational applications in scenarios such as smart speakers.


Further research is needed to improve machine assessment at the upper and lower ends of the scoring scale, although these are the scores for which the least training data exists.

future work could include different sampling methods, generation of synthetic data, or training objectives which reward models which are less conservatively drawn to the middle of the scoring scale.


In our case, we would expect that when users look for academic papers, the papers they view in a single browsing session tend to be related.

accurate paper embeddings should, all else being equal, be relatively more similar for papers that are frequently viewed in the same session than for other papers.


We test different embeddings on the recommendation task by including cosine embedding distance 9 as a feature within an existing recommendation system that includes several other informative features (title/author similarity, reference and citation overlap, etc.).

the recommendation experiments measure whether the embeddings can boost the performance of a strong baseline system on an end task.


Moreover, current methods for KG construction often rely on the rich structure of Wikipedia, such as links and infoboxes, which are not available for every domain.

we ask if it is possible to make predictions about, for example, new drug applications from raw text without the intermediate step of KG construction.


While our goal is to require almost no human domain expertise to learn a good model, the size of validation data is much smaller than the size of the training data.


However, this fine-tuning for multimodal language is neither trivial nor yet studied; simply because both BERT and XLNet only expect linguistic input.

in applying BERT and XLNet to multimodal language, one must either (a) forfeit the nonverbal information and fine-tune for language, or (b) simply extract word representations and proceed to use a state-of-the-art model for multimodal studies.


In essence, it randomly samples multiple factorization orders and trains the model on each of those orders.

it can model input by taking all possible permutations into consideration (in expectation).


As the first elementZ M CLS represents the [CLS] token, it has the information necessary to make a class label prediction.

,Z M CLS goes through an affine transformation to produce a single real-value which can be used to predict a class label.


Similarly for XLNET category, the results for MulT (with XLNet embeddings), XLNet and MAG-XLNet are as follows: [84.1, 83.7] for MulT, [85.4, 85.2] for XLNet and [85.6, 85.7] for MAG-XLNet.

superior performance of MAG-BERT and MAG-XLNet also generalizes to CMU-MOSEI dataset.


One exception, the long-running TV show Whose Line Is It Anyway, has, despite a large number of episodes, surprisingly little continuous improvised dialogue, due to the rapid-fire nature of the program.

we set our objective as collecting yesand-type dialogue pairs (yes-ands) to enable their modeling by corpus-driven dialogue systems.


An adequate evaluation of our models requires assessing the main yes-and criteria: agreement with the context and the quality of the new relevant contribution, both of which are not feasible with the aforementioned metrics.

we ask human evaluators to compare the quality of the yes-ands generated by various models and the actual response to the prompt in SPOLIN that is used as the input.


This is due to the aforementioned fact that they often  take short-cuts to directly reach the goal, with a significantly short trajectory.

the success rate weighted by inverse path length is high.


Plus, a model that performs well on only one condition but poorly on others is not practically useful.

to measure the robustness among conditions, we calculate the variance of accuracy under all conditions in a task.


However, these methods require significant computational resources (memory, time) during pretraining, and during downstream task training and inference.

an important research problem is to understand when these contextual embeddings add significant value vs. when it is possible to use more efficient representations without significant degradation in performance.


In particular, we assume that the prior covariance function for the GP is determined by the pretrained embeddings, and show that as the number of observed samples from this GP grows, the posterior distribution gives diminishing weight to the prior covariance function, and eventually depends solely on the observed samples.

if we were to calculate the posterior distribution using an inaccurate prior covariance function determined by random embeddings, this posterior would approach the true posterior as the number of observed samples grew.


This encoder is also the most lightweight.

we use it for the majority of our experiments.


The resulting quantization function Q has no gradient towards the input query vectors.

we use the straight-through estimator (Bengio et al., 2013) to compute a pseudo gradient.


We simply compute the embedding vector for the j th dimension of the i th entity as: The final entity embedding vector e i is achieved by the concatenation of the embedding vectors for each dimension: Non-linear Reconstruction (NL): While the codebook lookup approach is simple and efficient, due to its linear nature, the capacity of the generated KG embedding may be limited.

we also employ neural network based non-linear approaches for embedding reconstruction.


A major limitation of deep learning is the need for huge amounts of training data.

when dealing with low resource datasets, transfer learning is a common solution.


Notice that z is a sequence of embedding vectors.

the output of the FCN is also a sequence of vectors, where each of them tries to estimate the embedding of the corresponding word in the input sentence.


Due to the incorporation of bidirectional attention, masked language model can capture the contextual information on both sides.

it usually achieves better performances when finetuned in downstream NLU tasks than the conventional autoregressive models.


Practically, this theorem suggests the failure of bootstrapping (Efron, 1982) for statistical hypothesis testing and constructing confidence intervals (CIs) of the expected maximum, since the bootstrap requires a good approximation of the CDF (Canty et al., 2006).

relying on the bootstrap method for constructing confidence intervals of the expected maximum, as in Lucic et al. (2018), may lead to poor coverage of the true parameter.


We find that across all runs, the LFR is 100% and the clean accuracy 92.3%, with a standard deviation below 0.01%.

we conclude that the position of the trigger keyword has minimal effect on the success of the attack.


We present Enhanced WSD Integrating Synset Embeddings and Relations (EWISER), a neural supervised architecture that is able to tap into this wealth of knowledge by embedding information from the LKB graph within the neural architecture, and to exploit pretrained synset embeddings, enabling the network to predict synsets that are not in the training set.

we set a new state of the art on almost all the evaluation settings considered, also breaking through, for the first time, the 80% ceiling on the concatenation of all the standard allwords English WSD evaluation benchmarks.


Since the general-language corpus is web-crawled, it obviously contains a certain amount of domainspecific texts as well; especially if a highly technical term is not ambiguous, the general-language corpus contains only such contexts.

the general-language and domain-specific contexts are maximally similar in these cases.


the ranker takes a (question, answer) pair and a review as its input and calculates a ranking score s.

it can rank all reviews for a given QA pair.


Product aspects usually play a major role in all of product questions, answers and reviews, since they are the discussion focus of such text content.

such aspects can act as connections in modeling input pairs of qa and r via the partially shared structure.


the ranker is trained based on the rewards from the generation, which is used for instance augmentation in S.

the training set S is updated during the iterative learning, starting from a pure (question, answer) set.


World Englishes exhibit variation at multiple levels of linguistic analysis (Kachru et al., 2009).

putting these models directly into production without addressing this inherent bias puts them at risk of committing linguistic discrimination by performing poorly for many speech communities (e.g., AAVE and L2 speakers).


One possible explanation for the SQuAD 2.0 models' increased fragility is the difference in the tasks they were trained for: SQuAD 1.1 models expect all questions to be answerable and only need to contend with finding the right span, while SQuAD 2.0 models have the added burden of predicting whether a question is answerable.

in SQuAD 1.1 models, the feature space corresponding to a possible answer ends where the space corresponding to another possible answer begins, and there is room to accommodate slight variations in the input (i.e., larger individual spaces).


The diminished effectiveness of the transferred adversaries at inducing model failure is likely due to each model learning slightly different segmentations of the answer space.

different small, local perturbations have different effects on each model.


NNS and VBG also happen to be uncommon in the original distribution.

we conjecture that the models failed (Section 4) because MORPHEUS is able to find the contexts in the training data where these inflections are uncommon.


Although we agree that adding a GEC model before the actual NLU/translation model would likely help, this would not only require an extra model-often another Transformer (Bryant et al., 2019)-and its training data to be maintained, but would also double the resource usage of the combined system at inference time.

institutions with limited resources may choose to sacrifice the experience of minority users rather than incur the extra maintenance costs.


For example, in Fig. 1, the sentiment word "good" is highlighted, but other useful clues such as "but" and "not" do not gain sufficient attentions, which may not be optimal for learning accurate text representations.

a dynamically learnable degree of "hard" or "soft" for pooling may benefit text representation learning.


In contrast, if p is smaller, the attentions are more distributed, which indicates the attentive pooling is "softer".

in this manner, our APLN model can automatically explore how "hard/soft" the attention should be when constructing text representations, which may help recognize important contexts and avoid the problem of over-emphasizing some features and not fully respecting other useful ones, both of which are important for learning accurate text representations.


Unfortunately, in most cases the training of APLN is unstable if we directly use it for pooling.

we propose two methods to ensure the numerical stability of the model training.


This may be because when p > 1, our model has the risk of gradient explosion.

the scale of input features should be limited.


This is probably because a large value of p will lead to sharp attentions on critical contexts, and other useful information is not fully exploited.


This is probably because the rating of a review is usually a synthesis of all opinions conveyed by it.

it may not be optimal for learning accurate text representations if only salient contexts are considered.


For a proper evaluation of different auxiliary datasets, hyperparameter search and training runs with multiple random seeds have to be performed for each auxiliary dataset individually.

the process takes even longer and uses even more computational resources.


Because the process of selecting the closest vector representation from the main dataset to the auxiliary dataset or vice versa can result in different combinations, the counts in the contingency table will be different depending on the direction.

for a symmetric similarity measure like NMI, two scores are obtained.


We speculate that long sentences often contain more ambiguous words.

compared with short sentences, long sentences may require visual information to be better exploited as supplementary information, which can be achieved by the multi-modal semantic interaction of our model.


Previous works impose a too strong constraint on the matching and lead to many counterintuitive translation pairings.

we propose a relaxed matching procedure to find a more precise matching between two languages.


This 1 to 1 constraint brings out many redundant matchings.

in order to avoid this problem, we relax the constraint and control the relaxation degree by adding two KL divergence regularization terms to the original loss function.


(2) As q grows larger, the average number of decoding steps ("Step") increases steadily because the model is misled that to generate then delete a repetitive segment is expected.


(4) The model achieves the best performance with q = 0.5.


Although accelerating the decoding process significantly, NAT suffers from the multimodality problem (Gu et al., 2018) which generally manifests as repetitive or missing tokens in translation.

intensive efforts have been devoted to alleviate the multi-modality problem in NAT.


For instance, in North America, "much less than 1% of SMS messages were spam" (Almeida et al., 2013).

the active learning model should be more sensitive to spam samples.


If automatic ICD coding models ignore such a characteristic, they are prone to giving inconsistent predictions.

a challenging problem is how to model the code hierarchy and use it to capture the mutual exclusion of codes.


Meanwhile, the graph has been proved effective in modeling data correlation and the graph convolutional network (GCN) enables to efficiently learn node representation (Kipf and Welling, 2016).

we devise a code co-occurrence graph (co-graph) for capturing Code Co-occurrence and exploit the GCN to learn the code representation in the co-graph.


Effectively learning the document information about multiple labels is crucial for MLC.

we propose to connect CNN and RNN in parallel to capture both local and global contextual information, which would be complementary to each other.


Compressing capsules into a smaller amount can not only relieve the computational complexity, but also merge similar capsules and remove outliers.

hyperbolic compression layer is introduced.


Since most of the labels are unrelated to a document, calculating the label-aware hyperbolic capsules for all the unrelated labels is redundant.

encoding based adaptive routing layer is used to efficiently decide the candidate labels for the document.


In addi-tion, NLP-CAP applies the non-linear squashing function for capsules in the Euclidean space, while HDR is designed for hyperbolic capsules, which take advantage of the representation capacity of the hyperbolic space.

hYPERCAPS outperforms NLP-CAP as expected.


However, the interpretability is very important in the CDS to explain how the diagnosis is generated by machines.

we propose the Bayesian network ensembles on top of the output of ECNN to explicitly infer disease with PGMs.


The top 100,000 frequent segmented words consist of the word vocabulary in the embedding layer of ECNN.

the size of the embedding layer is (100000, 100).


Since the feature representation of pairs in the same row or column tends to be closer, we believe that pairs in the same row and column with the current pair have a greater impact on the current pair.

we propose the cross-road 2D transformer, in which the multi-head 2D self-attention mechanism is replaced by the cross-road 2D selfattention, and the other parts remain the same.


The data in different domains usually shares certain background knowledge that can possibly be transferred from the source domain to the target domain.

we leverage external knowledge as a bridge between the source and target domains.


Based on our empirical observation, capturing the multi-hop semantic correlation is one of the most important parts for the overall performance of SEKT.

we also investigate the impact of the number of hops used in GCN.


During training, we greedily find the 1-best head for each word without tree constraints.

the processing speed is faster than the evaluation phase.


On the one hand, despite bringing performance improvements over existing MNER methods, our UMT approach still fails to perform well on social media posts with unmatched text and images, as analyzed in Section 3.5.

our next step is to enhance UMT so as to dynamically filter out the potential noise from images.


Second, due to the global structure, the test documents are mandatory in training.

they are inherently transductive and have difficulty with inductive learning, in which one can easily obtain word embeddings for new documents with new structures and words using the trained model.


Despite that recent language encoders achieve promising performance, it is unclear if they perform equally well on text data with grammatical errors.

we synthesize grammatical errors on clean corpora to test the robustness of language encoders.


We believe such absolute measurements to the significance of words may be playing a more crucial role (than attention weights) when understanding the attention mechanism.

unlike many previous research efforts, we will instead focus on the understanding of attention scores in this work.


To determine whether a prefix x [1:i] is promising, we can estimate where is the minimum ratio of all sentences with prefixx is greater than a pre-defined threshold, all sentences with prefix x [1:i] should be rejected.

we do not need to waste time to continue sampling.


However, a richer latent space does not guarantee a better probability estimation result.

in this part, we delve deeper into whether the decoder signal matching mechanism helps improve probability estimation.


As it is shown in Table 5, our method is less likely to perturb some easily-modified semantics (e.g. numbers are edited to other "forms", but not different numbers), while search tends to generate semantically different tokens to achieve degradation.

our agent can lead to more insightful and plausible analyses for neural machine translation than search by gradient.


However, there are still some problems with machine translation in the documentlevel context (Laubli et al. , 2018).

more recent work (Jean et al., 2017;Wang et al., 2017;Tiedemann and Scherrer, 2017;Maruf and Haffari, 2018;Bawden et al., 2018;Voita et al., 2019a;Junczys-Dowmunt, 2019) is focusing on the document-level machine translation.


The flat structure adopts a unified encoder that does not distinguish the context sentences and the source sentences.

we introduce the segment embedding to identify these two types of inputs.


Intuitively, the less the direction of accumulated gradients is moved by the gradients of a new minibatch, the more certainty there is about the gradient direction.

we propose that the magnitude of the angle fluctuation relates to the certainty of the model parameter optimization direction, and may therefore serve as a measure of optimization difficulty.


But after the direction of gradients has stabilized, accumulating more mini-batches seems useless as the gradient direction starts to fluctuate.

we suggest to compute dynamic and efficient batch sizes by accumulating gradients of mini-batches, while evaluating the gradient direction change with each new mini-batch, and stop accumulating more mini-batches and perform an optimization step when the gradient direction fluctuates.


Encoders and decoders are (partially) shared between L 1 and L 2 .

l 1 and l 2 must use the same vocabulary.


The LBUNMT model trained in the same language branch performed better than the single model because similar languages have a positive interaction during the training process as shown in Tables 2 and  3.

the distilled information of LBUNMT is used to guide the MUNMT model during backtranslation.


As the number of languages increases, the number of translation directions increases quadratically.

zero-shot translation accuracy is important to the MUNMT model.


Specifically, training with teacher forcing only exposes the model to gold history, while previous predictions during inference may be erroneous.

the model trained with teacher forcing may over-rely on previously predicted words, which would exacerbate error propagation.


Indeed, viral claims often come back after a while in social media, and politicians are known to repeat the same claims over and over again.

before spending hours fact-checking a claim manually, it is worth first making sure that nobody has done it already.


Previous work has argued that BERT by itself does not yield good sentence representation.

approaches such as sentence-BERT (Reimers and Gurevych, 2019) have been proposed, which are specifically trained to produce good sentence-level representations.


In general, there is a 1:1 correspondence, but in some cases an Input claim is mapped to multiple VerClaim claims in the database, and in other cases, multiple Input claims are matched to the same VerClaim claim.

the task in Section 3 reads as follows when instantiated to the PolitiFact dataset: given an Input claim, rank all 16,636 VerClaim claims, so that its matching VerClaim claims are ranked at the top.


we use ranking evaluation measures, namely mean reciprocal rank (MRR), Mean Average Precision (MAP), and MAP truncated to rank k (MAP@k).


Initially, we tried to fine-tune BERT (Devlin et al., 2019), but this did not work well, probably because we did not have enough data to perform the fine-tuning.

eventually we opted to use BERT (and variations thereof) as a sentence encoder, and to perform max-pooling on the penultimate layer to obtain a representation for an input piece of text.


For the purpose of comparison, we tried to filter out the text of the input tweet from the text of the article body before attempting the matching, but we still got unrealistically high results.

ultimately we decided to abandon these experiments.


Subsequently, larger values of λ reduced the BLEU scores, suggesting that excessive biased content word translation may be weak at translating function words

Therefore, we set the hyperparameter λ to 0.4 to control the loss of target content words in our experiments (Table 1).


Especially for the LEFT-ARC lt action, there is only about 0.43% in the total actions, turning out to be the most difficult action to learn given the relatively small training samples.

as shown in Figure 5(a), the accuracy for LEFT-ARC lt is 0, which drops the overall performance heavily.


Shown as Figure 3, based on late-fusion multimodal learning framework (Cambria et al., 2017; Zadeh et al., 2017), we add independent output units for three unimodal representations: text, audio, and vision

these unimodal representations not only participate in feature fusion but are used to generate their predictive outputs.


It is relatively obvious that new models learn more distinctive unimodal representations compare to original models.

unimodal annotations can help the model to obtain more differentiated information and improve the complementarity between modalities.


Different from joint training, meta-transfer learning computes the firstorder optimization using the gradients from monolingual resources constrained to the code-switching validation set.

instead of learning one model that is able to generalize to all tasks, we focus on judiciously extracting useful information from the monolingual resources.


In multimodal context, sarcasm is no longer a pure linguistic phenomenon, and due to the nature of social media short text, the opposite is more often manifested via cross-modality expressions.

traditional text-based methods are insufficient to detect multimodal sarcasm.


For example, in Fig.1b, we can not reason about sarcasm intention simply from the short text 'Perfect flying weather in April' until we notice the downpour outside the airplane window in the attached image.

compared to text-based methods, the essential research issue in multimodal sarcasm detection is the reasoning of cross-modality contrast in the associated situation.


Our work focus on the multimodal sarcasm detection using image and text modalities.

we compare our model with the only two existing related models using the same modalities.


The MLP+CNN model simply takes the multimodal sarcasm detection as a general multimodal classification task via directly concatenating multimodal features for classification.


CNN and BiLSTM just treat the sarcasm detection task as a text classification task, ignoring the contextual contrast information.

their performances are worse than MIARN, which focuses on textual context to model the contrast information between individual words and phrases.


After removing the D-Net, the model only accepts the text and ANPs inputs.

we further incorporate image information via directly concatenating image encoding in the final fusion layer (see row 2).


In Fig.4b, our model pays more attention to the textual phrase 'these lovely books' with stupid sign, strange sign, and bad sign ANPs which refer to the emoji in the attached image.

it is easy for our model to detect the sarcasm intention that the books are NOT 'lovely' at all.


In multimodal sarcastic tweets, we expect our model to focus more on the opposite between different modality information.

we reinforce discrepancy between image and text, and on the contrary, weaken their commonality.


We have already extracted multiple ANPs as the visual semantic information, which is beneficial to model multi-view associations between image and text according to different views of ANPs.

we propose the ANP-aware cross-modality attention layer to align textual words and ANPs via utilizing each ANP to query each textual word and computing their pertinence.


However, the attention weights are difficult to learn, and the attention weights of SimulSpeech model are more difficult to learn than that of the simultaneous ASR and NMT models since SimulSpeech is much more challenging.

we propose to distill the knowledge from the multiplication of the attention weights of the simultaneous ASR and NMT, as shown in Figure 2b and Figure 3.


We add attention-level knowledge distillation (Row 5 vs. Row 3) to the model and find that the accuracy can also be improved.

we combine all the techniques together (Row 6, SimulSpeech) and obtain the best BLEU scores across different wait-k, which demonstrates the effectiveness of all techniques we proposed for the training of Simul-Speech.


As shown in Figure 5, simultaneous ASR model makes a mistake which further affects the accuracy of downstream simultaneous NMT model, while SimulSpeech is not suffered by this problem.

simulspeech outperforms cascaded models.


Our proposed approach aims to exploit speech signal to word encoder learnt using an architecture similar to Speech2Vec as lower level dynamic word representations for the utterance classifier.

our system never actually needs to know what word it is but only word segmentation information.


We found there was not a big difference in encoder output quality with higher dimensions.

we use a 50 dimensional LSTM cell, thus the resulting encoder output becomes 100 (Bidirectional last hidden states) + 100 (cell state) = 200 dimensions.


One challenge is that SSWE and Speech2Vec generally needs large amount of transcribed data to learn high quality word embeddings.

we first train SSWE on a general speech corpus (here, LibreSpeech (Libre)) before fine-tuning it on our classifier training data (results with * show this experiment).


We hypothesize that it can be due to the fact that our behavior code prediction data was split to minimize the speaker overlap.

it becomes easier to overfit when we fine-tune it on some speaker-related properties instead of generalizing for behaviour code prediction task.


SeqGFMN has a stable training because it does not concurrently train a discriminator, which in principle could easily learn to distinguish between one-hot and soft one-hot representations.

we can use soft one-hot representations that the generator outputs during training without using the Gumbel softmax or REINFORCE algorithm as needed in GANs for text.


A natural task that fits into this problem formulation is commonsense reasoning.

it will be the main focus of the present paper.


For example, on HellaSwag, the target hypothesis mode is only 8% better than the hypothesis only mode (58.8% versus 50.8%), which confirms that on this setting our zero-shot method is mainly taking advantage of the bias in the hypotheses.

we refrain from doing more zero-shot experiments on both datasets.


The time complexity of function f 3 is O(k 2 d) because there are k 3 dot product terms r x , h y , t w in total.

the scoring function f 3 needs k 3 times of dot product to compute the score of a triple (h, r, t).


For the space complexity, the dimension of entity and relation embeddings is d, and there are no other parameters in our SEEK framework.


And (5) another deep programming logic (DPL) method, GPT+DPL , is complicated, and the source code is not provided.

we directly used the results from the original paper and did not evaluate it on BERT.


It can be seen that BERT-HA+STM outperformed the base model BERT-HA by a large margin in terms of all the metrics.

the evidence extractor augmented with STM pro-vided more evidential information for the answer predictor, which may explain the improvements of BERT-HA+STM on the two datasets.


we compute the adjacency matrix A qcomp for graph G qcomp and A qcell for G qcell .


Though a significant amount of parameters are introduced for incorporating phrase representation into the Transformer model, our approach ("+Max+Attn+TA") improved the performance of the Transformer Base model by +1.29 BLEU on the WMT 14 En-De news task, and the proposed Transformer model with phrase representation still performs competitively compared to the Transformer Big model with only about half the number of parameters and 1/3 of the training steps.

we suggest our improvements are not only because of introducing parameters, but also due to the modeling and utilization of phrase representation.


The tweets collected with these hashtags may contain reported sexist acts towards both men and women.

we collected around 205, 000 tweets, among which about 70, 000 contain the specific hashtags.


To this date there have been no proposals for a dynamic oracle for CCG parsing with F1 metric over CCG dependency structures and it is not even clear if there is a polynomial solution to this problem.


For example, texts containing some demographic identity-terms (e.g., "gay", "black") are more likely to be abusive in existing abusive language detection datasets.

models trained with these datasets may consider sentences like "She makes me happy to be gay" as abusive simply because of the word "gay."


Because of such a phenomenon, models trained with the dataset may capture the unintended biases and perform differently for texts containing various identity-terms.

predictions of models may discriminate against some demographic minority groups.


However, "perform similarly" is indeed hard to define.

we pay more attention to some criteria defined on demographic groups.


Attention flow can indicate a set of input tokens that are important for the final decision.

we do not get sharp distinctions among them.


The lack of explicit claims by research may cause misinformation to potential users of the technology, who are not versed in its inner workings.

clear distinction between these terms is critical.


On average, the Diversity LSTM model provides 53.52 % (relative) more attention to rationales than the vanilla LSTM across the 8 Text classification datasets.

the attention weights in the Diversity LSTM are able to better indicate words that are important for making predictions.


While the system is still uncertain, the users often receive inappropriate (e.g., too hard or too easy) exercises.

they get the impression that the system does not work properly, which is especially harmful during the inception phase of an application, as the community opinion largely defines its success.


Less motivated learners or learners who suffer from distractions, interruptions, or frustration, however, may show different paces in their learning speed or even deteriorate in their proficiency.

we study four prototypical types of learner behavior: -Static learners (STAT) do not improve their skills over the course of our experiments.


In GEC, it is important to evaluate the model with multiple datasets .

we used GEC evaluation data such as W&I-test, CoNLL-2014 (Ng et al., 2014), FCE-test and JFLEG (Napoles et al., 2017).


Following Ye et al. (2018), we regard the paragraph start with “our court identified that” and end with “the above facts” as the fact description. Burges et al. (2005) shows that training on ties makes little difference

we could consider only defendant pairs (A, B) such that A plays a more important role than B and label it 1.


For humans, the most natural way to communicate is by natural language.

future intelligent systems must be programmable in everyday language.


Utterances that were labeled as non-teaching in the first stage also run through the third stage, except for signature synthesis.

we only construct scripts for this type of utterances.


Most encouragingly, the average rank of the correct element is near 1.

our scoring mechanism succeeds in placing the right elements on top of the list.


However, the vast majority of current datasets do not include the preceding comments in a conversation and such context was not shown to the annotators who provided the gold toxicity labels.

systems trained on these datasets ignore the conversational context.


To investigate whether adding context can benefit toxicity detection classifiers, we could not use CAT-SMALL, because its 250 comments are too few to effectively train a classifier.

we proceeded with the development of a larger dataset.


TAPAS predicts a minimal program by selecting a subset of the table cells and a possible aggregation operation to be executed on top of them.

tAPAS can learn operations from natural language, without the need to specify them in some formalism.


All of the end task datasets we experiment with only contain horizontal tables with a header row with column names.

we only extract Wiki tables of this form using the <th> tag to identify headers.


A scalar answer s that also appears in the table (thus C 6= ∅) is ambiguous, as in some cases the question implies aggregation (question 3 in Figure 3), while in other cases a table cell should be predicted (question 4 in Figure 3).

in this case we dynamically let the model choose the supervision (cell selection or scalar answer) according to its current policy.


The Krippendorff’s α of the original 3,600 annotations of response appropriateness is 0.431, which is considered not good according to the interpretation of the number in Table 5

we decided to remove the outliers to improve the inter-annotator agreement.


Moreover, multi-modal input helps the model to understand the intent and the sentiment of the speaker with more certainty.

in the context of a dialogue, multi-modal data such as video (acoustic + visual) along with text helps to understand the sentiment and emotion of the speaker, and in turn, helps to detect sarcasm in the conversation.


Whereas, in the case of visual modality, it majorly contains the image of the speaker along with sentiment and emotion information.

visual will not have a similar kind of problem as acoustic.


As we described in Section 2, accurately locating the previous statements about the claim is a very challenging problem.

instead of directly searching for a possible previous statement, we search for related context, where the source are describing a statement related to the claim.


However, an article can include multiple different statements about the same claim with different opinions, and multiple articles can refer to the same statement about the claim from a common source.

the majority vote by opinions in article level is not good enough, since it suffers from (1) opinions which are too coarse-grained and (2) overcounting the opinions from the same source, which is also known as collusion or dependency of sources problem in truth finding (Pochampally et al., 2014).


MPQA dataset is originally developed for identifying sources for the given opinion, and the opinion sometimes can be a noun phrase or an entity, while in our problem we are to extract sources for claims.

we only leave the opinions which are sentences as the query claim, and perform 10-fold cross validation to evaluate the performance of our models and the baselines.


The 'quality' of extracted rationales will depend on their intended use.

we propose an initial set of metrics to evaluate rationales that are meant to measure different varieties of 'interpretability'.


When deploying a CliniRC system to a new environment (e.g., a new set of clinical records, a new hospital, etc.), it is infeasible to create new QA pairs for training every time

an ideal CliniRC system is able to generalize to unseen documents and questions after being fully trained.


We draw several observation from the evidence selection results: (1) AIR vs. unsupervised methods -AIR outperforms all the unsupervised baselines and previous works in both MultiRC (row 9-15 vs. row 23 in table 1) and QASC(rows 0-6 vs. row 18).

highlighting strengths of AIR over the standard IR baselines.


We cannot use Webster et al.’s GAP dataset directly, because their data is constrained that the “gender” of the two possible antecedents is “the same”7 ; for us, we are specifically interested in how annotators make decisions even when additional gender information is available.

we construct a dataset called Maybe Ambiguous Pronoun (MAP) follow-ing Webster et al.'s  approach, but we do not restrict the two names to match gender.


As it is shown in Fig. 3, there might be multiple changes for each output words during the translation, and we only start to calculate the latency for this word once it agrees with the final results.

it is necessary to locate the last change for each word.


Many speculate that these representations encode a continuous analogue of discrete linguistic properties, e.g., part-of-speech tags, due to the networks’ impressive performance on many NLP tasks (Belinkov et al., 2017).

of this speculation, one common thread of research focuses on the construction of probes, i.e., supervised models that are trained to extract the linguistic properties directly Conneau et al., 2018;Peters et al., 2018b;Zhang and Bowman, 2018;Naik et al., 2018;Tenney et al., 2019).


Since TurkCorpus was adopted as the standard dataset for evaluating SS models, several system outputs on this data are already publicly available (Zhang and Lapata, 2017;Zhao et al., 2018;Martin et al., 2020).

we can now assess the capabilities of these and other systems in scenarios with varying simplification expectations: lexical paraphrasing with TurkCorpus, sentence splitting with HSplit, and multiple transformations with ASSET.


The two images illustrate different NC concepts (i.e., HIGH JUMP and POLE VAULT) which are different configurations of the same elementary objects (i.e., PERSON, ROD, BLEACHERS).

nC concepts require complex image understanding, integrating a fair amount of common sense knowledge.


For each label, these weights are given by its learned correlation with all the other labels.

the prediction score of each label is affected by the prediction score of the other labels, based on the correlation between label pairs.


As a side-effect, we observe that older interpretability methods for static embeddings-while more mature than those available for their dynamic counterparts-are underutilized in studying newer contextualized representations.

we introduce simple and fully general methods for converting from contextualized representations to static lookup-table embeddings which we apply to 5 popular pretrained models and 9 sets of pretrained weights.


Through a human study, we show that our manipulated attention-based explanations deceive people into thinking that predictions from a model biased against gender minorities do not rely on the gender.

our results cast doubt on attention's reliability as a tool for auditing algorithms in the context of fairness and accountability.


Thus, the models (trained on unanonymized data) make use of gender indicators to obtain a higher task performance.

we consider gender indicators as impermissible tokens for this task.


Since the feasible space is the same for both kinds of constraints, the performance difference is due to the randomness of the ILP solver picking different solutions with the same objective value.

the entity and relation experiments in this section demonstrate that our approach can recover the designed constraints and provide a way of interpreting these constraints.


In contrast, our method for learning constraints uses general constraint features, and does not rely on domain knowledge.

our method is suited to tasks where little is known about the underlying domain.


Such a z' is a negative example for the constraint learning task because z' has a lower objective value than z.

it violates at least one of the constraints in Eq.


This feature looks at a pair of entities and focuses on the two relation labels between them, one in each direction.

our running example will give us two positive examples with features (OrgBasedIn, NoRel) and (NoRel,OrgBasedIn).


Despite the high correlation, we also find that the estimated FAR scores may vary in range compared to the ground-truth FAR.

we further use the estimations of different sentence regression approaches to train a linear regression model to fit the ground-truth FAR (denoted as AutoFAR).


On the other hand, the agreement between rhetorical relations tends to be lower and more ambiguous.

we do not encode rhetorical relations explicitly in our model.


Equipped with the metrics for abstractiveness above, we want to further understand how abstractive the generated summaries are, and whether the amount of abstractiveness is a result of the training data or the model.

we compute abstractiveness scores for both the reference summaries and summaries generated from a diverse set of models on two datasets.


Our analysis above shows that the number of unfaithful sentences increases significantly as more abstractive summaries are generated.

the key challenge to faithfulness evaluation is to verify highly abstractive sentences against the source document, where surface similarity matching would fail.


Given the right corpus, we argue that a language model's probability can be modified into a Fluency Score.

we adapt a language model into the Fluency Model.


Speakers distill their past experience of language use into what we call "meaning" here, and produce new attempts at using language based on this; this attempt is successful if the listener correctly deduces the speaker's communicative intent.

standing meanings evolve over time as speakers can different experiences (e.g. McConnellGinet, 1984), and a reflection of such change can be observed in their changing textual distribution (e.g. Herbelot et al., 2012; Hamilton et al., 2016).


Nonetheless, given the immense volume of scientific literature, the relative ease with which one can track citations using services such as Google Scholar (GS), and given the lack of other easily applicable and effec-tive metrics, citation analysis is an imperfect but useful window into research impact.

citation metrics are often a factor when making decisions about funding research and hiring scientists.


Citation analysis can also be used to gauge the influence of outside fields on one's field and the influence of one's field on other fields.

it can be used to determine the relationship of a field with the wider academic community.


However, scientometric researchers estimated that it included about 389 million documents in January 2018 (Gusenbauer, 2019)-making it the world's largest source of academic information.

there is growing interest in the use of Google Scholar information to draw inferences about scholarly research in general (Howland, 2010; Orduna-Malea et al. , 2014; Khabsa and Giles, 2014; Mingers and Leydesdorff, 2015; Mart´ın-Mart´ın et al., 2018) and on scholarly impact in particular (Priem and Hemminger, 2010; Yogatama et al., 2011; Bulaitis, 2017; Ravenscroft et al., 2017; Bos and Nitza, 2019; Ioannidis et al., 2019).


However, algorithms also provide systematic ways to reduce bias, and some see the mitigation of bias in algorithm decisions as a potential opportunity to move the needle positively (Kleinberg et al., 2018).

we can apply frameworks of contemporaries in human behavior to machines (Rahwan et al., 2019), and perhaps benefit from a more scalable experimentation process.


The language of demographic groups systematically differs from each other for syntactic attributes.

models trained on samples whose demographic composition (e.g., age and ethnicity) differs from the target perform significantly worse.


In contrast, the per-worker PEA scores for our annotations are shifted towards the right, indicating better agreement than the random baseline.

we interpret our annotations as showing "moderate agreement" under the PEA metric.


In most cases, Equation 2 is not equal to the ordinary conditional distribution P (Y | T = t) since the latter is simply filtering to the sub-population and the former is changing the underlying data distribution via intervention.

for observational studies that lack intervention, one needs an identification strategy in order to represent P (Y | do(T = t)) in terms of distributions of observed variables.


For instance, studies have shown that pre-processing decisions dramatically change topic models (Denny and Spirling, 2018;Schofield et al., 2017); embeddings are sensitive to hyperparameter tuning (Levy et al., 2015) and the construction of the training corpus (Antoniak and Mimno, 2018); and fine-tuned language model performance is sensitive to random restarts (Phang et al., 2018).

reporting sensitivity analysis of the causal effects from these decisions seems crucial: how robust are the results to variations in modeling specifications?


However, as Gentzel et al. (2019) discuss, synthetic data has no “unknown unknowns” and many researcher degrees of freedom, which limits their effectiveness.

we encourage researchers to evaluate with constructed observational studies or semi-synthetic datasets, although measuring latent confounders from text increases the difficulty of creating realistic datasets that can be used for empirical evaluation of causal methods.


Our key insight in this paper is that the context or relations through which specific information is propagated among different players in the legislative process (e.g., money donors and legislators), can be leveraged to further improve the performance.

we build a shared relational architecture that models the text of a bill and its context into a graph; Our model captures the behavior of individual legislators, language of bills, and influence of contributions on the decision to identify demographic cleavages.


A close look reveals that the legislative process cannot be captured in a simple graph as there can be multiple relations between a pair of nodes (e.g., sponsorship and vote between legislators and bills), and the graph consists of several nodes types with different attributes and labels (e.g., bills with competitive labels).

we model the process using a heterogeneous multi-relational graph, as follows: Node attributes: The nodes in our proposed legislative graph come with a rich set of features and information: (1) Bill nodes contain title, description, and full text of the house and senate state bills.


In NLP, dropped pronouns can cause loss of important information, such as the subject or object of the central predicate in a sentence, introducing ambiguity to applications such as machine translation (Nakaiwa and Shirai, 1996; Wang et al., 2016; Takeno et al., 2016), question answering (Choi et al., 2018; Reddy et al., 2019; Sun et al., 2019; Chen and Choi, 2016) and dialogue understanding (Chen et al., 2017; Rolih, 2018).

zero pronouns have recently received much research attention (Liu et al., 2017; Yin et al., 2018a,b)


Compared to semantic frames (Fillmore and Baker, 2001), the meanings projected by pragmatic frames are richer, and thus cannot be easily formalized using only categorical labels.

as illustrated in Figure 1, our formalism combines hierarchical categories of biased implications such as intent and offensiveness with implicatures described in free-form text such as groups referenced and implied statements.


In the Post round, users are given the same data, but they are also equipped with explanations of the model predictions for the original inputs.

any improvement in performance is attributable to the addition of explanations.


As explained, given end-task supervision only, modules may not act as intended, since their parameters are only trained for minimizing the end-task loss.

a straightforward way to improve interpretability is to train modules with additional atomic-task supervision.


Note that since a single proposed bounding box can align with multiple annotated bounding boxes, it is possible for the numerator to exceed the denominator.

these two choices for a common numerator have issues, and we avoid these issues by defining the numerators of precision and recall separately.


Specially, DST-SC is designed with a slot connecting mechanism to establish the connection between the target slot and its source slot explicitly.

it can take advantage of the source slot value directly instead of reasoning from preceding turns.


As claimed in Section 1, connecting the target slot with its source slot helps to decrease the reasoning difficulty.

we enhance the copyaugmented encoder-decoder model with a slot connecting mechanism to model slot correlations directly.


Their applications are usually limited in a single domain.

several open vocabulary approaches in generative fashion (Xu and Hu, 2018;Wu et al., 2019;Ren et al., 2019) are proposed to handle unlimited slot values in more complicated dialogues.


A wide range of NLP tasks have greatly benefited from the pre-trained BERT model.

we also finetune the pre-trained BERT-Large model on our task through sequence pair classification schema.


In Example 4, BERT-ft prefers A but the answer is C. The reason why BERT-ft chooses A may be that "enjoy life" happens in the context, but summarizing the next sentence is necessary to achieve the correct answer.

it is necessary to improve the ability of BERT to represent meaning at the sentence level beyond representing individual words in context.


A potential artifact type for our task is whether we could detect distractors without passages.

we finetune BERT-Large as a binary classifier, the input of which is just distractors and other correct candidates.


With |B| = 5 blanks and |C| = 7 candidates, the size of answer space, |A|, is number of permutations |B| objects taken |C| at a time, i.e., P(7, 5) = 2520.

the probability of answering all blanks correctly is 1/2520 = 0.03% What are the chances of getting answers partially correct?


While fine-tuning for span-based QA, every utterance as well as the question are separated encoded and multi-head attentions and additional transformers are built on the token and utterance embeddings respectively to provide a more comprehensive view of the dialogue to the QA model.

our model achieves a new state-of-the-art result on a span-based QA task where the evidence documents are multiparty dialogue.


One important reason behind this is that, due to the vague definition of commonsense knowledge, we are not clear about what the essential knowledge types are and thus we are unclear about how to represent, acquire, and use them.

we can only treat commonsense knowledge as a black box and try to learn it from limited training data.


If at least four annotators think the reason is plausible, we will accept that reason.


Note that each reason may contain inference over multiple knowledge types.

for each reason, we invite five different annotators to provide annotations.


Each annotators are provided with detailed instruction of the job, descriptions of each candidate category, and examples for the category.


Besides the above analysis, we are also interested in how different models perform on questions that require complex reasoning types.

we divide all WSC questions based on how many knowledge types are required to solve these questions and show the result in Table 5.


One possible reason is that even though the designers of WSC are trying to avoid any statistical correlation between the answer and the trigger word, such statistical correlation still exists.

pre-trained language representation models can learn such correlation from large-scale training corpus and thus can answer WSC questions without fully understanding the reasons behind.


Research in Cyber Argumentation has shown that incorporating both stance polarity and intensity information into online discussions improves the analysis of discussions and the various phenomena that arise during a debate, including opinion polarization (Sirrianni et al., 2018), and identifying outlier opinions (Arvapally et al., 2017), compared to using stance polarity alone.

automatically identifying both the post's stance polarity and intensity, allows these powerful analytical models to be applied to unstructured debate data from platforms such as Twitter, Facebook, Wikipedia, comment threads, and online forums.


The difference between strong opposition and weak opposition is often expressed through subtle word choices and conversational behaviors.

to accurately predict agreement intensity, a learned model must understand the nuances between word choices in the context of the discussion.


The authors were instructed on how to annotate their posts, but the annotations themselves were left to the post's author's discretion.

including author information into our models would likely improve the stance polarity and intensity prediction results.


Our full model outperforms state-ofthe-art unsupervised fine-tuning approaches and partially supervised approaches using crosslingual resources in 8/11 tasks.

our results provide a strong lower bound performance on what future semi-supervised or supervised approaches are expected to produce.


The maximization is guaranteed to converge to the unique maximum likelihood estimator in finite steps under the assumption that in every possible partition of the items into two nonempty subsets, some subject in the second set beats some subject in the first set at least once (Hunter, 2004).

a pairwise comparison experiment is restricted in two ways: (i) The matrix formed by the comparisons must construct a strongly connected graph; (ii) The comparisons between the partitions cannot all be won by subjects from the same group, i.e., no item has losses or wins exclusively.


However, this strategy suffers from the major drawback that for some step sizes, the resulting graph has multiple unconnected components, thus violating the restriction that the comparison matrix must form a strongly connected graph.

complex combinations of different step sizes are needed, resulting in needlessly complicated experimental setups.


By example, going from x = 1, k = 16 to x = 1, k = 4 ends up at the same number of comparisons as x = 2, k = 8, but has a slightly higher ranking accuracy.

it is more economical to increase the sampling rate until the required accuracy is met than collecting multiple judgments.


However, the specific choice of k depends on scale and domain of the data as well as trustworthiness of comparisons.

we refrain from making a general suggestion for the choice of k. Thus, if the model is to be adapted to drastically different domains or item counts, exploratory studies are advised to estimate the quality tradeoff for a specific use case.


While this could hint at a data bias, with crowd workers just voting for longer texts in the comparison but not actually reading all of it, the effect is much less pronounced when only measuring the correlation in texts longer than 100 words (n = 869).

much of the pronounced effect can be explained by short texts receiving justified low scores rather than longer texts being voted higher regardless of content.


The first step of the PCA accounts for 73% of the data variance, and is equally influenced by all three quality dimensions.

evidence is given towards the hypothesis.


To account for this, the score distributions are equally shifted into the positive domain.

a standardized scalar value for overall argument quality can be calculated.


While this paradigm works to a certain extent, it usually retrieves knowledge facts only based on the entity word itself, without considering the specific dialogue context.

the introduction of the context-irrelevant knowledge facts can impact the quality of generations.


To summarize, Felicitous Fact mechanism can alleviate the first two issues, and the next two techniques solve the last issue.

our approach can improve the utilization rate of knowledge graphs, as well as can promote the diversity and informativeness of the generated responses.


In the field of natural language processing (NLP) which widely employs DNNs, practical systems such as spam filtering (Stringhini et al., 2010) and malware detection (Kolter and Maloof, 2006) have been broadly used, but at the same time the concerns about their security are growing.

the research on textual adversarial attacks becomes increasingly important.


Instead, the successful uses of neural networks in computational linguistics have replaced specific pieces of computational-linguistic models with new neural network architectures which bring together continuous vector spaces with structured representations in ways which are novel for both machine learning and computational linguistics.

the great progress which we have made through the application of neural networks to natural language processing should not be viewed as a conquest, but as a compromise.


With parameters shared across entities and sensitive to these properties and relations, learned rules are parameterised in terms of these structures.

transformer is a deep learning architecture with the kind of generalisation ability required to exhibit systematicity, as in (Fodor and Pylyshyn, 1988).


Since most of the common neighbors would be popular entities, they will be neighbors of many other entities.

it is still challenging to align such entities.


However, not all one-hop neighbors contribute positively to characterizing the target entity.

considering all of them without careful selection can introduce noise and degrade the performance.


On another scenario, recent analysis also reveals that state-of-the-art sequential neural language models still fail to learn certain long-range syntactic dependencies (Kuncoro et al., 2018).

it is an interesting problem to explore the relation between language models and syntax and investigate whether syntax can be integrated to enhance neural language models.


Their differences are such that they can not be easily collapsed into a single meta-tag.

we do not penalize the model for producing any variation of equally valid analyses given the surface form, and for each model we adjust the evaluation for syncretism in a post-processing step.


Chinese is an ideographic language and lacks word delimiters between words in written sentences.

chinese word segmentation (cWS) is often regarded as a prerequisite to downstream tasks in chinese natural language processing.


Source domain data and target domain data generally have different distributions.

models built on source domain data tend to degrade performance when they are applied to target domain data.


Chinese word segmentation is typically formalized as a sequence tagging problem.

traditional machine learning models such as Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs) are widely employed for CWS in the early stage (Wong and Chan, 1996;Gao et al., 2005;Zhao et al., 2010).


Furthermore, the system outputs pseudo-tags, and the mapping from pseudo-tags to paradigm slots is unknown.

we propose to use best-match accuracy (BMAcc), the best accuracy among all mappings from pseudo-tags to paradigm slots, for evaluation.


Hiring crowd-sourcing workers to perform these annotations is very costly.

we propose automated data augmentation methods to expand existing well-annotated dialog datasets, and thereby train better dialog systems.


In other words, a paraphrased dialog utterance needs to serve the same function as the original utterance under the same dialog context.

we propose to construct dialog paraphrases that consider dialog context in order to improve dialog generation quality.


These patterns can be seen as the "template" to produce the questions.

we can use the patterns as the prior to regularize the QG model and obtain better results accordingly.


These methods often need a lot of labeled data for training, but the data is expensive to obtain.

we are inspired to apply our generated results to enrich the training set for the task of MRC-QA.


Slightly modified queries such as iPhone X charger and case for iPhone X refer to different products.

it is hard for distributed representations to capture the nuances.


BLEU scores do not indicate whether or not a generated queries is a "realistic" modification of the original query.

we also had 2500 generated pairs annotated by human experts who were specifically trained to decide if a query-item pair is matched or not


To account for these differences between query strings and item titles, we separately train word embeddings using word2vec (Mikolov et al., 2013) on anonymized query logs and item titles.

the same word can have two embeddings, one for the query and one for the title.


Then, the evaluation of model uncertainty is conducted on D U to mirror the confidence over the current curriculum.

our approach reserves the efficiency in CL, in the meanwhile, guiding the duration of each curriculum in a self-adaptive fashion.


This phenomenon reveals that the most "simple" and "complex" sentences quantified by different measures are relatively similar, and the main diversity lies in those sentences of which the difficulties hardly to be distinguished.

we argue that the improvements of the proposed method may mainly contribute by the differences in these two steps.


We noticed that the Spanish tokenizer sometimes merges multi-word expressions into a single token joined with underscores for contiguous words.

some tokens cannot be aligned with the corresponding entity annotations.


We restrict our analysis to Spanish since the data is labeled with both de-identification and concept information (see Section 4.1).

we can also investigate the difference between gold and predicted de-identification labels.


Semantic matching operations between two mentions (and their contexts) are performed only at the output layer and are relatively superficial.

it is hard for their models to capture all the lexical, semantic and syntactic cues in the context.


Moreover, there can be multiple dialogue acts mentioned in a single dialogue turn, which requires the model to attend to different acts for different sub-sequences.

a global vector is unable to capture the inter-relationships among acts, nor is it flexible for response generation especially when more than one act is mentioned.


Being a span enumeration type model, DYGIE++ only works on paragraph level texts and extracts relations between mentions in the same sentence only.

we subdivide SCIREX documents into sections and formulate each section as a single training example.


The target of our framework is to conduct the formats controlled text generation.

the indicating symbols for format and rhyme as well as the sentence integrity are designed based on the target output sequence.


But the Rhyme accuracy and the sentence integrity will drop simultaneously.

in the experiments we let k = 32 to obtain a trade-off between the diversity and the general quality.


Since the gradients from optimizing the functional task are sent all the way into the base captioning model, this causes catastrophic forgetting of the core knowledge of language, leading to language drift.

we use a language regularizer term in the form of Kullback-Leibler divergence between pre-trained and fine-tuned language modeling distributions (Havrylov and Titov, 2017).


We can observe that the larger the validation loss, the lower the BLEU score.

the validation loss can be a good performance proxy.


Page's argument is that the original student is not going to be much worse off with a com-puter than with an (average) human reader, because originality is a subjective construct.

once research uncovers objective and measurable aspects of "original" writing, relevant features can be added into an AWE system; finding such aspects, as well as measuring them, is still work in progress.


In particular, a story entails a highly structured network of relations (timelines, causality, etc.).

stories do exercise abilities beyond simple factoid extraction.


The ones canonicalized by the guidelines and by annotators following them may not always be the most useful.

it may prove beneficial to appeal directly to human intuition about what understanding entails.


We expect that introducing such importance information for the words in the deep learning models might lead to improved performance for RE.

in this work, we propose to obtain an importance score for each word in the sentences from the dependency trees (called the syntax-based importance scores).


This dataset does not provide training, development, or test splits due to the small number of samples.

we run 5-fold cross validations and report the average scores.


First, we notice that the class name generation goal is similar to the hypernymy detection task which aims to find a general hypernym (e.g., "mammal") for a given specific hyponym (e.g., "panda").

we leverage the six Hearst patterns (Hearst, 1992), widely used for hypernymy detection, to construct the class-probing query.


Most state-of-the-art diacritic restoration models are built on character level information which helps generalize the model to unseen data, but presumably lose useful information at the word level.

to compensate for this loss, we investigate the use of multi-task learning to jointly optimize diacritic restoration with related NLP problems namely word segmentation, part-of-speech tagging, and syntactic diacritization.


Despite leveraging tasks focused on syntax (SYN/POS) or morpheme boundaries (SEG), the improvements extend to lexical diacritics as well.

the proposed joint diacritic restoration model is also helpful in settings beyond word final syntactic related diacritics.


In an average development document with 201 candidates per mention, the number of pairwise queries needed to fully label a document is 15, 050, while the maximum number of discrete queries is only 201 (i.e., asking for the antecedent of every mention).

the average document can be fully annotated via discrete annotation in only 2.6% of the time it takes to fully label it with pairwise annotation, suggesting that our framework is also a viable exhaustive annotation scheme.


However, structured attributes in product catalogs are often sparse, leading to unsatisfactory search results and various kinds of defects.

it is invaluable if such structured information can be extracted from product profiles such as product titles and descriptions.


Our main idea is that by encouraging TXtract to predict the product categories using only the product profile, the model will learn token embeddings that are discriminative of the product categories.

we introduce an inductive bias for more effective category-specific attribute value extraction.


The insight behind our loss function is that a product assigned underÃ„â€° could also be assigned under any of the ancestors ofÃ„â€°.

we consider hierarchical multi-label classification and encourage TXtract to assign a product to all nodes in the path fromÃ„â€° to the root, denoted by (Ã„â€° K ,Ã„â€° KÃ¢Ë†â€™1 , .


It might be hard to say that the superior performance of TCattn is due to the neural architecture and attention scores rather than the richer training resources.

a comparison between TCattn and a model that uses both student essays and the source article is needed.


As creation is expensive, most annotated clinical datasets are small, such as for our task.

we look to alternative data sources for pre-training our model.


However, training on synonyms will allow for a greater variety of terms to be seen by our model than otherwise possible.

using all synonyms taken from the annotated subset of the UMLS, we pre-train our linker before training on the annotated clinical notes.


Because of this, the common practice is to test new methods on a small number of languages or domains, often semi-arbitrarily chosen based on previous work or the experimenters' intuition.

this practice impedes the NLP community from gaining a comprehensive understanding of newly-proposed models.


Another limitation is that generated data is of limited use for training models, since it contains simple regularities that supervised classifiers may learn to exploit.

we create IMP-PRES solely for the purpose of evaluating NLI models trained on standard datasets like MultiNLI.


Thus, we anticipate two possible rational behaviors for a MultiNLI-trained model tested on an implicature: (a) be pragmatic, and compute the implicature, concluding that the premise and hypothesis are in an 'entailment' relation, (b) be logical, i.e., consider only the literal content, and not compute the implicature, concluding they are in a 'neutral' relation.

we measure both possible conclusions, by tagging sentence pairs for scalar implicature with two sets of NLI labels to reflect the behavior expected under "logical" and "pragmatic" modes of inference, as shown in Table 2.


Overall, more than 20% of the treebanks in the UD 2.2 collection have flat structures in more than 20% of their training-set sentences.

a parsing approach taking into account the special status of headless structural representations can potentially benefit models for a large number of languages and treebanks.


CRF models the conditional probability of the whole label sequence given the whole input sequence.

instead of using the label distribution over individual token, we could use the probability distribution for the whole label sequence, to compute KL divergence.


But as explained in Sec.1, the adversarial loss of conventional VAT cannot be calculated on top of CRF.

vAT in the second set of Table.2 only applies CRF for label loss.


And the meaning of each character changes dramatically when the context changes.

a CSC system needs to recognize the semantics and aggregate the surrounding information for necessary modifications.


By contrast, the similar characters are semantically distinct in CSC.

we deeply investigate the effect of our SpellGCN and propose several essential techniques.


Since the pronunciation similarity is more fine-grained compared with the shape similarity category, we combine the pronunciation similarities into one graph.

we construct two graphs corresponding to pronunciation and shape similarities.


However, more than half of the databases in Spider contain 5 tables or less.

we also report the coverage of attributes only considering the databases which have more than 5 tables, where Spider only covers 49.6% of attributes.


Table 2 shows that our corpus contains a much higher lexical complexity of the questions than Spider (0.67 instead of 0.52).

our approach seems to avoid trivial or monotonous questions, which also matches with our impression from manual inspection.


Roy and Roth (2017) shown significantly lowered performance if highly similar MWPs are removed.

dataset diversity is more critical than the dataset size for accurately judging the true capability of an MWP solver.


Since MWPs are usually clearly specified (with a sure answer), there is no ambiguous interpretation once the answer is given.

as opposed to other corpora in which annotations (mostly linguistic attributes) are mainly based on human subjective judgment, the MWP answer/equation annotation is more objective and must be consistent.


Therefore, as opposed to other corpora in which annotations (mostly linguistic attributes) are mainly based on human subjective judgment, the MWP answer/equation annotation is more objective and must be consistent.

human carefulness, instead of human agreement, is a more critical issue in this task.


In grades 5 and 6, improved math skills enable students to solve difficult MWPs that require more aggregative operations and additional domain knowledge.

the grade level is a useful indicator of difficulty and can be employed to evaluate the capability of MWP solving systems.


The "words" in abstract syntax can be seen as word senses, which the concrete syntax can realize by different word forms.

the abstract function Love in Figure 7 corresponds to words in different languages used for expressing a certain sense of the English word love.


Segment analysis is also used when forming compound words in languages like Finnish, German, and Swedish.

english summer time translates compositionally to Swedish sommar+tid, which is rendered as sommartid.


Because of this information loss, one cannot expect that such details will be correctly deduced or guessed, especially without a wider context.

the default linearization by the GF generator (and the guessed linearization by the JAMR generator) of the AMR concept person in Figure 20 is in the singular form.


Recall that our method first collects the top M candidate chains, ordered by retrieval score (Section 3.2).

a simple baseline is to use that retrieval score itself as a measure of chain validity.


For output structure generalization, we formatted the answers as JSON to enable more complex zero-shot relation extraction tasks.

the models output answers as both text and JSON, in a seq-to-seq fashion, depending on the question type.


"Chris", supposedly tagged with "Person" in this example sentence, is tagged as other labels in most cases.

in the predicting process, it is difficult to label "Chris" correctly.


The advantage of stance over TS is indirect stances.

we also investigate how well various methods perform on indirect stance.


Custom methods for other targets also behave in a similar manner (c.f Figure 4), with certain targets like Putin outperforming the best OTS method, STB in this case, with fewer than 195 labeled tweets.

instead of having different training sizes for different targets, we use the same amount and find that the LR custom methods outperform OTS methods for all targets except Macron.


Although researchers have designed a lot of meaning representations, recent work focuses on only a few of them.

the impact of meaning representation on semantic parsing is less understood.


A semantically equivalent program may have many syntactically different forms.

if the training and testing data have a difference in their syntactic distributions of logic forms, a naive maximum likelihood estimation can suffer from this difference because it fails to capture the semantic equivalence (Bunel et al., 2018).


We regard Prolog, Lambda Calculus, and FunQL as domainspecific MRs, since the predicates defined in them are specific for a given domain.

the execution engines of domain-specific MRs need to be significantly customized for different domains, requiring plenty of manual efforts.


With these resources, we crossvalidate the correctness of annotations and execution engines by comparing the execution results of logical forms.

we found nearly 30 Prolog logical forms with annotation mistakes and two bugs in the execution engines of Prolog and FunQL.


We use two strategies for this purpose: (1) we explicitly introduce distinct divergence categories for unrelated sentences and sentences that overlap in meaning; and (2) we ask for annotation rationales (Zaidan et al., 2007) by requiring annotators to highlight tokens indicative of meaning differences in each sentence-pair.

our approach strikes a balance between coarsely annotating sentences with binary distinctions that are fully based on annotators' intuitions (Vyas et al., 2018), and exhaustively annotating all spans of a sentence-pair with fine-grained labels of translation processes (Zhai et al., 2018).


Without this, the model can learn the goal task (such as translation) with reasonable accuracy, but the learned semantic embeddings are of poor quality until batch sizes approximately reach 25,000 tokens.

we use a maximum batch size of 50,000 tokens in our ENGLISHTRANS, BILIN-GUALTRANS, and BGT W/O PRIOR, experiments and 25,000 tokens in our BGT W/O LANGVARS and BGT experiments.


We hypothesize that these datasets contain many examples where their gold scores are easy to predict by either having similar structure and word choice and a high score or dissimilar structure and word choice and a low score.

we split the data using symmetric word error rate (SWER), 7 finding sentence pairs with low SWER and low gold scores as well as sentence pairs with high SWER and high gold scores.


Most existing KGs are built separately by different organizations, using different data sources and languages.

kGs are heterogeneous that the same entity may exist in different kGs in different surface forms.


Equivalent entities in two KGs are usually neighbored by some other equivalent entities.

structure information in KGs are very important for discovering entity alignments.


However, such a hard cutoff of the search space makes these approaches insufficient in the exploration of the (already scarce) labeled data and limited by the ranker since most sentences are discarded, 2 albeit the discarded sentences are important and could have been favored.

although these studies perform better than directly applying their base SDS models (See et al., 2017;Tan et al., 2017) to MDS, they do not outperform state-of-the-art MDS methods (Gillick and Favre, 2009;Kulesza and Taskar, 2012).


This variant solves L2 but may re-expose the RL agent to L1 since its MMR module and neural module are loosely coupled and there is a learnable layer in their combination.

we design a second variant, RL-MMR SOFT-ATTN , which addresses both L1 and L2 by tightly incorporating MMR into neural representation learning via soft attention.


The number of epochs is set to 10,000 and we adopt early stopping -the training process terminates if RL-MMR cannot achieve better results on the validation set after 30 continuous evaluations.

the runs often terminate before 5,000 epochs, and the overall training time ranges from 40 to 90 minutes.


Note that since labels are transmitted using a model, the model has to be transmitted as well (directly or indirectly).

the overall codelength is a combination of the quality of fit of the model (compressed data length) with the cost of transmitting the model itself.


However, despite multiple recent efforts on this newly proposed dataset, published work so far in multimodal KPE has either omitted available features (Sun et al., 2020), or has adopted a brute force approach to feature encoding (direct concatenation of raw features) (Xiong et al., 2019).

in this work we strive for a more nuanced approach to leveraging available features for multimodal KPE and offer a uniquely comprehensive approach.


Second, the behavior and characteristics of visual and text modality features are different from one another.

the first-step self-attention should be modeled in separate networks for text and visual features.


First, unlike previous continual learning works on image classification (Kirkpatrick et al., 2017;Zenke et al., 2017), VisCOLL requires predicting, for example, a noun with a verb or an adjectivewhich results in a significantly large search space.

of this increased search space, memory based continual methods (Robins, 1995;Aljundi et al., 2019a) cannot expect to store prototypes of each visited compositions.


Second, the increased search space makes it infeasible to view all possible combinations of atomic words at train time.

to succeed on VisCOLL, models should generalize to novel compositions at test time (also called composition generalization) (Lake and Baroni, 2017; Keysers et al., 2020).


In contrast, task boundaries in VisCOLL are unknown and "smooth" (i.e., with gradual transitions between tasks)-a setting that is closer to real-world situations.

visCOLL rules out many continual learning algorithms which require explicit task identity and boundary (Kirkpatrick et al., 2017;Rusu et al., 2016).


Finally, the algorithm greedily tries to put the proposed number of instances into each time interval to construct the stream.

the constructed data stream has a gradually shifting task distribution without strict boundaries.