To utilize contextual information of an operand token, researchers built hand-crafted features that capture the semantic content of a word, such as the unit of a given number (Roy and Roth, 2015; Koncel-Kedziorski et al., 2015; Zhou et al., 2015; Upadhyay et al., 2016; Roy and Roth, 2017) or dependency relationship between numbers (Kushman et al., 2014; Zhou et al., 2015; Upadhyay et al., 2016).

in this paper, we propose a pure neural model called Expression-Pointer Transformer (EPT) to address the two issues above.


To obtain a solution to the generated equations, we use SymPy (Meurer et al., 2017) at the end of the training phase.

recent neural models have only utilized ‘Op (operator/operand)’ tokens (Wang et al., 2017; Amini et al., 2019; Chiang and Chen, 2019; Huang et al., 2018; Wang et al., 2019), resulting in two issues: (1) the expression fragmentation issue and (2) the operandcontext separation issue.


The paraphraser can generate phrases like "find people from Stanford", which is trivial to parse correctly.

we use the standard train/test split and following previous work, use 20% of the human paraphrases from the original training set for validation, so that validation and test sets are from the same distribution.


For instance, in this work, if a propaganda class c is predicted by the multiple binary classifiers (indicates the sentence contains this propaganda technique), then the token-level predictions belonging to the propaganda class c should also exist.

neural networks are less interpretable and need to be trained with a large amount of data to make it possible to learn such implicit logic.


In this paper we propose the use of information from word graphs to enhance intent classification, more specifically, for the detection of out-of-scope examples.

the difference in ISER of BERT+ against BERT is of 5%, where the former achieved 37% and the latter 32%.


Eq. (7) again asserts something trivial: low contextual uncertainty implies in an informative context.

in these Figures we see that, especially for highly ambiguous words, contextual uncertainty tends to be very small.


She was allegedly gang-raped on Sunday when she went outside her house in Kosi Kalan, in Uttar Pradesh's Mathura district, to relieve herself.

there has also been analysis work augmenting ROUGE with human evaluation (Narayan et al., 2018;Liu and Lapata, 2019).


We aim to address the above issues by quantifying the primary sources of errors over representative models.

for example, a news article describes an event as happening "on Wednesday" in a summary although the original document has "on April 1".


The footage was shot in September last year after the girl's body was found in woodland, enraging locals.

the Point-Generator model reduces the error count to 14, demonstrating the effectiveness of the copy mechanism in faithfully reproducing details.


It also reduces duplication on the word level but tends to cause redundancy to a certain degree.

a human annotator is presented the original text and an output summary in juxtaposition, and is asked to select segments that are deemed incorrect after reading.


Coverage solves repetition errors by a large margin, but shows limits in faithful content generation.

joined by a groom on another of her fell ponies, the queen cut a relaxed figure as she enjoyed her ride but, as is her wont, eschewed a helmet in favor of one of her silk scarves.


For instance, the F-score of 73.3% for plain UKB with SOFTCONSTRAINT (shown in Table 2) drops to 72.1% with only Japanese translations, to 64.2% with only Italian translations, and to 58.0% with no translations.

for fair comparison, when applying SOfTCONSTRAINT to a system variant without sense frequency information, we set γ to 0 to turn off the p freq component.


An example is illustrated in Figure 1, where Allan Donald and The 48-year-old former Test paceman are enriched with the same special tokens.

existing summarizers cannot make effective use of these expressions to establish correspondence between sentences, often leading to ungrammatical and nonsensical outputs.


Recent innovations in Transformer-based ranking models have advanced the state-ofthe-art in information retrieval.

total, Online, Online, 1 Query-Document Pair 1 Query-Document Pair N doc Documents typical transformer Ranker reuse strategies with different time vs. space trade-offs: 1) a document representation reuse strategy that stores the Document Representation Module's output, and 2) a projected document representation reuse strategy that stores the Interaction Module's intermediate transformed document representations.


These strategies have the same overall math, produce the same ranking results, and only differ in time/space efficiency.

the performance gains come at the computational cost of inferring the many token-level interaction signals at the evaluation time, which scales quadratically to the input length.


Moreover, latent-variable models can represent multimodal distributions.

in Section 3, we show that encoders learn to partially memorize the first few words and the document lengths, as was first discovered by Kim et al.


However, information related to the second word in the latent variable can help the decoder predict the first word.

moreover, using reconstructions, we observe that they decrease memorization: the first word and the sentence length are not recovered as accurately than with the baselines, consequently yielding more diverse reconstructions.


We use the resized image patches as additional training data to to emphasize these patches' importance.

we focus on eliciting fewer but more informative explanations to reduce expert involvement.


Note, however, that the models fit the training data worse and do not generalize as well as the original model.

as training data we use the English Easy-to-Read version of the Parallel Bible Corpus (Mayer and Cysouw, 2014) that contains the New Testament.


The Pearson correlation between Kendall's tau metric and the XNLI classification accuracy in a zero-shot scenario (mBERT only finetuned on English and tested on all other languages) is 46% when disregarding English and 64% when including English.

iD 0 shows high multilinguality with 0-shot accuracies .57 and .45.


This shows that, the proposed method can effectively utilize the added language-specific adapters to improve generalization of shared parameters across languages.

we show that, contrary to previous belief, negative interference also impacts low-resource languages.


Advances in pretraining language models (Devlin et al., 2018; Liu et al., 2019; Yang et al., 2019) as general-purpose representations have pushed the state of the art on a variety of natural language tasks.

To study negative interference, we compare multilingual models with monolingual baselines.


A more negative value indicates that the parameter is more likely to be pruned for that language and vice versa.

we use the Universal Dependencies treebanks (Nivre et al., 2018).


For ablation studies, we also report results for JointPair trained with adapters shared between two languages, denoted as share adpt.

we study whether gradient conflicts exist between languages in multilingual models.


Not only is beam search usually more accurate than greedy search, but it also outputs a diverse set of decodings, enabling reranking approaches to further improve accuracy (Yee et al., 2019; Ng et al., 2019; Charniak and Johnson, 2005; Ge and Mooney, 2006).

this lightweight model is no longer state of the art, but its decoding is representative of more recent architectures (Suhr et al., 2018; Yin and Neubig, 2018; Lin et al., 2019).


For languages with rare scripts (e.g., CJK and Thai), CLUSTER strongly outperforms JOINT in all tasks.

as an extreme example, a character-based model would have to learn how to reconstruct each word, while a word-based model is exempt from this task.


We tune the number of fine-tuning epochs and batch-size using the development data of Scitail and Amazon Electronics tasks following (Bansal et al., 2019).

Fig. 5 and Fig. 6 show the CCA similarity on the two datasets: CoNLL and Scitail.


As Wiki itself is a collaborative knowledge repository, editors are likely to attack others due to disputes on specific domain knowledge.

unlike the cloze procedure, text normalization involves more diversified and challenging tasks.


It randomly masks a certain portion of tokens from the input and then learns to predict these masked words.

tNt randomly manipulates tokens from the input text.


We show the minimum and maximum accuracy for each task across the 10 experiments.

published results for mBERT zero-shot accuracy vary as much as 17 points on the MLDoc classification task across four papers.


This shares the same underlying sequence-consistency concept as BERT's MLM.

this is prohibitively expensive due to the computational costs.


The decoupled biLSTM extended with ELMo inputs is able to outperform the transformer model initialised with RoBERTa pretraining.

for an extended version of this model, we also try incorporating contextualized word vectors, by augmenting the input with ELMo embeddings (Peters et al., 2018).


MASKAUGMENT through DAL and MTL objectives becomes larger as the number of labeled dialogues in the source domain gets smaller.

table 5: Micro-F1 scores for each dialog act (DA) on the test split of target (GRes) domain.


However, existing adversarial text generation approaches that try to perturb in the input text space might lead to generations lacking diversity or fluency.

we argue that our model creates more natural and meaningful attacks to real-world tasks by demonstrating our attacks are more robust against model re-training and across model architectures.


After this initially automated training, we scheduled a 1hour long phone call with them to discuss our annotation instructions and annotation interface.

small-scale studies on manually annotated datasets have also been conducted (Morris and Picard, 2012; Lord et al., 2015).


This intuition is confirmed by an analysis of the discourse connective "though" among all posts we collected, which revealed a clear tendency towards the end of a reply, as illustrated in Figure 1.The lexical discourse marker "though" was found by splitting a large collection of posts and replies from r/AskParents into Elementary Discourse Units (Mann and Thompson, 1988), using a neural discourse segmenter (Wang et al., 2018).

as (1) shows, the entirety of a response to a question rarely constitutes advice.


For instance, BEESL identifies the +REG-ULATION event anchored at "activated" in the following sentence: "Tax [...] maximally activated HTLV-I-LTR-CAT and kappa B-fos-CA" albeit the gold standard does not contain the event in this instance.

this shows that BEESL's performance is clearly affected, but that the system is relatively robust to noisy, nongold silver entities.


Without multi-task learning and multi-label decoding, the F1 score drops to 61.44 (independent classifiers) and 61.13 (ST setup, BEESL ST in Table 3).

the full power of BEESL is only achieved by using both the multi-task and the multi-label approach, which leads to the novel state of the art.


In our experiments, we aim at evaluating the multilingual and the cross-lingual question answering capabilities of different models.

One reason is that many of the subjects in these two groups such as Business & Economics, Geography, and History can be answered using knowledge that is easily accessible in sources such as Wikipedia (e.g., “Who was the first prime minister of Poland after 1990?”), i.e., without the need for complex reasoning or calculations, which are often needed in order to answer questions in subjects such as Physics and Chemistry.


In the Public Figures data, the lexicons that overlap the most with the high-bias posts are "Beautiful", "Arrogant", and "Sexual", which suggests that bias  in these comments focuses on appearance and sexualization.

we then generate masked posts: for every word w in the post, we generate a version of the post that omits w. we run these masked posts through our gender-prediction model and compare the prediction scores where w is omitted and where w is not omitted, averaging across all occurrences of w in the 500 posts.


If women ask “Do I look ok?” more frequently than men, this naive classifier would identify “UR hot!”

when it is addressed to someone who said "Do I look ok?


If the classifier predicts the gender of the addressee with high confidence based only on the text directed to them, we hypothesize that the text is likely to contain bias.

bias in comments directed towards politicians are less focused, and differences between the high-confidence prediction posts and the random sample are smaller.


Pixel+Local, which uses both image encoding, g i , and the structural representation computed only based on the properties of the element, offers further improvement on the accuracy.

the most frequent caption is "go back" that amounts to 4.0% of the distribution.


Finally, the Predicate + Object captions were identified as the ones that contain at least one of the 22 verbs and one of the 194 nouns and appear at least twice in the corpus.

the view hierarchy is a structural tree representation of the UI where each node has a set of properties such as content description, class information, visibility, and bounding boxes.


We elaborate the encoders in section 3.3.

it embeds visual knowledge even for plain text inference.


The fact that SeqMix can improve over simple regularization methods (such as WordDrop) even without GECA indicates that despite its crudity, SeqMix is somewhat effective at biasing models to learn the appropriate compositional structure.

sCAN sCAN is a command execution dataset designed to test for systematic compositionality of data-driven models.


For both, we use Reddit comments as our base data.

our results disagree with prior observations on empathy and we find that, while the compassion-like empathy of Buechel et al.


We empirically investigate three different methods for inducing semantic dependencies, including attention (Vaswani et al., 2017), sparse attention (Correia et al., 2019) and hard Kuma discrete structures (Bastings et al., 2019).

Sun et al. (2019b) also proposed a GCN model based on dependency trees for aspect sentiment analysis similar to depGCN of .


In 1995, Lyla Novacek (Keri Russell) is a cellist studying at the Juilliard School and living under strict rule of her father (William Sadler).

they both seem to be "just in it for the money"I really hope this will be the last Rush Hour film, the first and second was both really really enjoyable with fun comments, good comedy and good action scenes this third installment has nothing of it.For some good Jackie action stick with his older films and those made in Hong Kong, for Chris tucker films watch any other film he's in besides this one..the goofy antics of Chris tucker mixed with the amazing martial arts moves of Jackie Chan made both of the previous Rush Hour films funny and entertaining.


Results in Table 4 show that, our system can indeed predict tags that are very relevant to the new types of stories.

this paper considers the problem of characterizing stories by inferring properties such as theme and style using written synopses and reviews of movies.


The speed at which movies get reviews fluctuates a lot.

com'on, ye people of sense and sensibility, join me and suffer this movie and be merry you found the strength and then give it the only rating it deserves....PS: I am not going to elaborate on the "oh-you're-so-mean!"


I want to think that most of Rush Hour 3's problems are attributed to the fact that it was just a rushed film.

i just finished watching August Rush and i am in no way exaggerating when i say that it is by far the best movie i have ever seen.


Given this, can we recover phylogenetic similarities across gender systems using our methods?

several variants of MI are preferred in community detection.


Because different random restarts yield different accuracy results, we report ranges.

using our own data, we estimate their character corpus at ∼30,000 tokens.


We run the same initial procedure from Section 5, giving us a vector space inhabited by both written words and pinyin words.

while EM only considers the top M pinyin triples, final decoding works on entire sentences and is free to create previously-unseen pinyin trigrams.


We ran their expectation-maximization (EM) algorithm for 170 iterations on a character corpus of 300,000 tokens, then applied their decoding algorithm to our 6059-token test, obtaining a token pronunciation accuracy of 8.6%.

we modify the nearestneighbor search that produces word/pronunciation mappings.


The results show that our method can consistently outperform the baselines, particularly when the given gold training data are less.

We follow similar experimental settings as NER and POS tasks, except that the same sequence tagging model released by Li et al. (2019a) is used for evaluation


For example, "[ORG U.N.] official [PER Ekeus] heads for [LOC Baghdad] ".

for random deletion, we also report the result when gold and synthetic data are equally sampled, denoted by rd * .


However, because their model again conditions on the target data, they have to train the autoencoder jointly with the target corpus, defeating the purpose of large-scale pretraining.

in this study, we apply it to supervised and unsupervised text style transfer.


For the sentiment classification task, T3 can perform the targeted attack to make an originally positive review be classified as the most negative one, and vice versa.

In most countries, the dispensary is subject to pharmacy legislation; with requirements for storage conditions, compulsory texts, equipment, etc., specified in legislation.


Contextual embeddings are proved to be overwhelmingly effective to the task of Word Sense Disambiguation (WSD) compared with other sense representation techniques.

this has revealed the potential of such a fine-grained WordNet gloss utilization, and the employment of more valuable resources such as Wikipedia rather than web mentions for further investigation.


"It always bothered me that we confused an enduring mission with a temporal goal. "'

the terrorists destroyed "Iraq newspapers dating to the early 20th century, maps and books from the Ottoman Empire, and book collections contributed by about 100 of Mosul's establishment families. "


In this paper, we instead use a NER model trained on the source domain to learn token-level representations that minimizes the supervised cross-entropy loss.

we use the OntoNotes train/development/test splits released for the CoNLL 2012 shared task.


By using regex-based extractors and a list of comprehensive dictionaries that capture crucial domain vocabularies, LUSTRE can generate rules that achieve SoTA results.

since CRF-AW is fully supervised, we give it the sets of labels we iteratively accumulated during the active learning of our model.


The difference between the token and componentlevel evaluation is that the former accumulates the credits over all tokens regardless the actual classes they belong to, whereas the latter evaluation accumulates the credits with respect to the actual classes.

we developed a system named PARTNER (Qian et al., 2020) that implements this framework.


Length of the context plays an important role.

in unconstrained human descriptions of changes, ∼40% of the referred-to entities were unmentioned in the text (e.g., the knife and cutting board in several cooking recipes) (§4.4).


For the sake of completeness, we provide Table A2 to help further explore the depths of this unique dataset.

we thus face the challenge of how best to optimize our usage of the limited input space, which is made more difficult by the many different types of input context (e.g., entries, characters, challenges) within STORIUM.


While GPT-2 has successfully been used as a state-of-the-art model for story generation (Mao et al., 2019;Guan et al., 2020), one crucial challenge is the length of the contexts: each entry in a story can condition on any narrative element that comes before it (e.g., previous entries, scenes, challenges).

We then only consider substrings with at least one nonstopword as matches matches (careful scrutiny of Figure 1 reveals an unmatched stopword it).


Our template based approach bears similarities to sentence fusion (Barzilay and McKeown, 2005), and prototype based text editing (Hossain et al., 2020; Cao et al., 2018; Guu et al., 2018; Wu et al., 2019).

in this work, we investigate two methods for Natural Language Generation (NLG) using a single domain-independent model across a large number of APis.


In this work, we focus on tackling the first problem only.

experiments show that our approach achieves the state-of-the-art on supervised event extraction and discovers a set of high-quality unseen types.


More specifically, 1) SANA achieves 84.35% in SST2 dataset which is higher than the distillation only model, but lower than task-level supervised model.

instead of treating attention as a by-product of model training, the following work explored how machine/human can consume attention for model improvement or explanation, respectively.


Tab. 2 shows the classification accuracy for three classification datasets.

our work is of combining the strength of the two works: we automatically improve attention supervision via self-supervision signals, but we build it with free task-level resources.


For each task, we train binary logistic regression probes with a range of hidden sizes and select the smallest yielding at least 97% of the best model's performance.

our results point to a paradigm of probing with latent variables, for which LSL is one potential technique.


Descriptive methods produce highspecificity tests of what structure is present in the model, and facilitate discovery of new patterns that were not hypothesized prior to testing.

for all LSL probes, we use N = 32 latent classes.


Since nPMI is information-theoretic and chance-corrected, it is a reliable indicator of the degree of information about gold labels contained in a set of predicted clusters.

that is the question we investigate in this work.


Morevoer, cycle-GANs only synthesize the input and verify whether the input is synthesized (e.g. the utterance looks like a user request).

we proposed GAZP to adapt an existing semantic parser to new environments by synthesizing cycle-consistent data.


Results in Table 8 show that SentiLARE-EF-LS already outperforms RoBERTa remarkably on SST-Less by simply augmenting input features with linguistic knowledge.

the statistics of these datasets are reported in table 4.


In the neural CRF model, the score can be obtained from the neural network encoders such as biLSTM networks.

peer review and rebuttal, containing rich arguments, are a worth-studying domain for argument mining.


information about reviews and author responses, such as review passages, rebuttal passages, review scores, reviewer confidence, etc.

we further evaluate the performance of different difficulty levels.


For example, the idiom "Time is money" is a metaphor in which "time" is the noumenon and "money" is the metaphoric object.

as mentioned earlier, these categories are not disjoint.


First, we conduct a statistical analysis in an attempt to answer various questions involving exaggeration, such as: (1) are there strong lexical indicators of hyperbole; (2) how lexically diverse are the non-hyperbolic versions of a given hyperbolic sentence; and (3) how lexically diverse are the hyperbolic versions of a given nonhyperbolic sentence?

first, given that hyperbolic sentences may be more descriptive (e.g., compare the first sentence in each class in Table 1), are hyperbolic sentences longer than non-hyperbolic sentences on average?


Ferre (2014) shows that at the textual level, a hyperbole can be present in a word or in the interpretation of a certain context.

the more word overlaps there are between two sentences, the higher their similarity is.


Figures 17 to 22 show additional examples of the extracted alignments from the different layers of the NMT system.

in Sections 4 and 5, we provide insights into the working of Transformers using norm-based analysis.


To verify whether the results obtained in the Section 5 are reproducible in different settings, we conducted an additional experiment using the model with a different number of attention heads.

note that we analyzed a model with 12 heads in each layer.


We fine-tune a pretrained BERT base model on the dataset of the CoNLL'03 NER shared task (Tjong Kim Sang and De Meulder, 2003) and use it to extract entities from candidate paragraphs.

hotpotQA is a widely used large-scale multi-hop QA dataset.


Nevertheless, when pre-trained models are used as a fine-tuning approach, which is a common practice, graph structure does not contribute to the final results.

in feature-based setting, all models are trained for 30 epochs with a batch size of 24.


Selected paragraphs are concatenated as context C. Encoding Layer.

we suppose this is another attention pattern that entities attend to the whole sentence.


Hallucinations Other errors stem from the fact that the model can only rely on the knowledge about possible definienda that it is able to store in the parameters during the pre-training and training stages.

In this paper we overcome these limitations by proposing a unified approach to computational lexical semantics that has as its central focus Definition Modeling (DM), i.e. the task of generating a gloss1 from static or contextual embeddings (Noraset et al., 2017


We also note that none of the results with L=n configurations from Table 1 can match best performing AVG(L≤n) configurations with layer-wise averaging.

all scores are Mean Reciprocal Rank (MRR) scores (in the standard scoring interval, 0.0-1.0).


Another observation that holds across all configurations concerns the usefulness of providing contexts drawn from external corpora, and corroborates findings from prior work (Liu et al., 2019b): ISO configurations cannot match configurations that average subword embeddings from multiple contexts (AOC-10 and AOC-100).

micro-averaged F 1 scores reported , obtained as averages over 5 experimental runs for each configuration; standard deviation is also reported.


In other words, we can intervene on the location entity to set it to another different location entity without destroying the sentence correctness at the grammatical level.

firstly, we review the definition of the structural causal model (SCM) (See figure 2(a)): e := f E (g) c := f C (g) x := f X (e, c) where G is a confounding variable that influences the generation of both entity E and context C, X is the input example that is generated by E and C, and Y is the evaluation result (the f 1 score) of the NER model.


This phenomenon suggests that the non-spurious correlations are more located in entity representation rather than context representation.

data acquisition is a challenging task for some special domains.


And it proves that the state-of-the-art neural networks can easily exploit such name regularity, mention coverage and context diversity knowledge, and therefore achieve state-of-the-art performance in these benchmarks.

despite the success of recent models, there are specific advantages in current NER benchmarks which significantly facilitate supervised neural networks.


Because currently no suitable dataset is available for verifying our conclusions, this paper constructs a new dataset from Wikipedia.

in this way, all mentions will share identical frequency in the vanilla and CR dataset.


In the following, we will illustrate the empirical findings through the test, with one subsection for one kind of information.

this group of experiments can be used to verify the Conclusion 3 we proposed before.


Table 1 shows all our randomization test with examples.

our investigation leads to three valuable conclusions, which shows the necessity of de-cent name regularity to identify unseen mentions, the hazard of high mention coverage to model generalization, and the redundancy of enormous data to capture context patterns.


For example, the political relationship between two countries might intensify because of trade fights.

we report the best results on each dataset in Table 1 in the main body.


Since then, hyperbolic analogs of several other approaches have been developed (De Sa et al., 2018; Tifrea et al., 2018).

an important analogy to vector spaces (vector addition and scalar multiplication) in non-Euclidean geometry is the notion of gyrovector spaces (Ungar, 2008).


Also, the degree of each entity is accumulated over all timestamps.

existing embedding approaches for temporal KGs typically learn entity representations and their dynamic evolution in the Euclidean space, which might not capture such intrinsic structures very well.


To retain high-confidence predictions for self-training, ensemble methods like tri-training (Saito et al., 2017), mutual learning (Ge et al., 2020) and dual information maximization (Ye et al., 2019) have been introduced.

in the training process, we predict pseudo labels on all the target samples in T .


Without this, systems run the risk of merely paraphrasing the biases in text.

in the second example, "Ana strutted" implies that she is more active and decisive, compared to "Ana wandered" which portrays her as aimless and passive.


Probabilities p b are assigned to each training instance to indicate how likely that it contains biases.

for each setting, we report the average results across 5 runs.


2019, and report test results for the last epoch.

recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data.


The second is the use of antonyms or adversarial object-words to substitute critical words.

the first operator is negation for yes-no questions, which  is achieved by a template based procedure that negates the question by adding a "no" or "not" before a verb, preposition or noun phrase.


Identifying that the distribution of answers in the VQA dataset led models to learn superficial correlations, Agrawal et al. (2018a) proposed the VQA-CP dataset by re-organizing the train and test splits such that the the distribution of answers per question-type was significantly different for each split.

over-reliance on datasets can cause models to learn spurious correlations such as linguistic priors (Agrawal et al., 2018a) that are specific to certain datasets and do not generalize to "Out-of-Distribution" (OOD) samples, as shown in Figure 1.


Note that these 100 examples are expected to be enriched with positive examples, yet in a biased manner, since by construction, all examples match the query we started with.

we focus on practical scenarios of binary text classification, where the annotation budget is very small, and the data is often skewed.


Information about the performance on the test set of the queries used in the Imbalanced-practical scenario is given in Table 5.

the use of AL with deep pre-trained models for text classification -and BERT in particular -has so far received surprisingly little consideration.


We further extended our VVMAs beyond NLP, to image classification.

we also used VVMAs for the matrices in the attention mechanism.


Stehwien and Vu (2017) and Stehwien et al. (2018) (henceforth, SVS18) showed that neural methods can perform comparably to traditional methods using a relatively small amount of speech context— just a single word on either side of the target word.

In future, we could follow Tran et al. (2018) by giving an explicit feature for the duration of a given token normalized by the average duration of that token in the corpus


For a detailed description of how we divided data into train, development, and test sets for cross-validation, see Appendix A.2.

since pitch accents are deviations from a speaker's average pitch, intensity, and duration, we hypothesize that, as in some non-neural models (e.g.


Following Shah et al. (2018); Rastogi et al. (2019), every grammar production in the simulator is paired with a template whose slots are synchronously expanded.

this shows the impact of error propagation as the model predicts the target dialog state based on past representations.


We use a two-phase coordinate ascent framework as described in Algorithm 1.

as expected, by inducing our model to predict the question attributes for the target utterance, our model achieves the highest inquisitiveness (58% question rate).


For probabilistic linking, we use a 6-layer encoder-decoder Transformer model (Vaswani et al., 2017).

we aim to train a linking model such that conditioning on linked documents has a positive effect on dialog modeling performance.


In this task, each example consists of a pair of short texts: a premise and a hypothesis.

we propose four alternative protocols, each aimed at improving either the ease with which annotators can produce sound training examples or the quality and diversity of those examples.


The research on TE dates back more than two decades and has made significant progress.

we often solve each task separately by first gathering task-specific training data and then tuning a machine learning system to learn the patterns in the data.


Creation of CONJNLI involves four stages, as shown in Figure 1.

once the conjuncts are identified, we perform three operations by removing, adding or replacing one of the two conjuncts to obtain another sentence such that the original sentence and the modified sentence form a plausible NLI pair.


As an example, we apply it on the boolean subset of the dataset by Richardson et al. (2020) containing samples with “boolean and” and find that our model achieves a near perfect accuracy on their test set.

6.3 Predicate-Aware RoBERTa with Adversarial Fine-Tuning Table 8 consolidates our final results on both the datasets.


You and your friends are not welcome here, said Severn.

following their suggestion, we conduct an instability analysis of CONJNLI by training RoBERTa on MNLI with 10 different seeds (1 to 10) and find that the results on CONJNLI are quite robust to such variations.


From the systems perspective, natural language processing pipelines that include knowledge graphs can rely on calibrated confidence scores to determine which KGE predictions to trust.

to arrive at your answer, you must use English-language Wikidata and/or Wikipedia, even if you know the answer ahead of time.


Note that there have been recent (concurrent) efforts to construct appropriate datasets for evaluating KGE calibration (Pezeshkpour et al., 2020; Safavi and Koutra, 2020).

now we need a label for every triple considered, whereas with the CWA we only needed labels for a small group of positives.


It is worth mentioning that our expansion here is adaptive and every label may have a different number of seeds.

mETA does not restrict to single metadata types and goes beyond by employing motif patterns to capture the metadata information.


Table 4 shows the performance of SHERLOCK and the baselines for both the explicit negation and the implied negation cases.

the main idea is to find the subject associated with the governor node, and extract that as the wide scope ("Next week").


The current state of the art learned method uses a Neural BiLSTM-CRF model (Fancellu et al., 2016).

Finally, for some cases, we also use domain specific cues that imply a non-availability (For example, in “Dr. John out of office on Monday.”, “out of office” implies an unavailability to meet.)


2 experts with experience in recruiting applicants for CRC positions of all levels design the annotation guidelines in 5 rounds by labeling each resume with either one of the four CRC levels, CRC1-4, or Not Qualified (NQ), indicating that the applicant is not qualified for any CRC level.

given the maximum number of tokens, N , that the transformer encoder (TE) allows, any section S i Ã¢Ë†Ë† R that contains more than N -number of tokens is pruned by applying the following procedure: Then, the pruned section S i is created for every S i , where S i Ã¢Å â€  S i and |S i | Ã¢â€°Â¤ N .


Several previous studies have identified such flaws and reported that the corpus is noisy (Vinyals and Le, 2015 ;Li et al., 2016; Baheti et al., 2018), where noisy refers to unacceptable utterance pairs in this context.

of our preliminary experiment, we discover that, out of all scores given for utterance pairs, 25% was unacceptable (scored as 1: Strongly disagree or 2: Disagree) and almost half was acceptable (scored as 5: Strongly agree or 4: Agree).


In our experiments, the DQN model is trained 100 epochs where each epoch contains 1,000 transitions.

we parameterize an approximate value function Q(s, a; θ) using a three-layer deep neural network.


Decoders A and B are specialized to generate only sentences and paths (respectively) from these embeddings.

if BLEU2 metric is used for the selection, then BERT-GRU achieves higher scores.


As can be seen from the table, KGPT can achieve better results than the mentioned baseline models.

though template-GPt-2 is getting slightly better score with 500 training samples, the overall performance on three datasets are remarkably lower than KGPt, especially under more extreme cases.


As the input RDF triples were modified from the original triples in DBPedia, we first need to check whether there are seen triples in pre-training dataset KGTEXT.

the existing pre-trained text generation models (Radford et al., 2019; Keskar et al., 2019; Raffel et al., 2019) are initially designed to condition on text input, thus lacking the ability to encode structured inputs.


We use documents of English Wikipedia and BookCorpus (Zhu et al., 2015) as our pre-training corpus, and perform WordPiece tokenization as BERT (Devlin et al., 2018).

it differs from existing generative pre-training methods in that PALM goes beyond the solely autoencoding/autoregressive methods and combines the merits of autoencoding and autoregression in a single framework.


PALM gives surprisingly good empirical results on a variety of context-aware generation tasks, including pushing the state-of-the-art Rouge-L on the MARCO Natural Language Generation benchmark to 0.498 (Rank 1 on the leaderboard 1 ) and on Gigaword summarization to 36.75, as well as establishing the state-of-the-art ROUGE-1 (44.30) and ROUGE-L (41.41) on CNN/Daily Mail.

bERT does not make predictions autoregressively, so it is not effective for generation tasks.


Latent state s ∈ S contains the current game information (e.g. locations of the player and items, the player’s inventory), which is only partially reflected in o.

we train this model for 10 epochs until the validation loss converges, unlike previous models which we train for 3 epochs.


Natural Language Generation (NLG) is a challenging problem in Natural Language Processing (NLP)-the complex nature of NLG tasks arise particularly in the output space.

this suggests that CIDEr might be unstable and sensitive to the selection of references.


In addition, a handful of models for the TwitterStance dataset have been designed for cross-target stance detection (Augenstein et al., 2016; Xu et al., 2018), including a number of weakly supervised methods using unlabeled data related to the test topic (Zarrella and Marsh, 2016; Wei et al., 2016; Dias and Becker, 2016).

to test model susceptibility to sentiment polarity, we generate swapped examples.


.. we need(-) to get those GOP members out of the House & Senate, since they only support(+)!patronize(-) billionaire tax breaks, evidently(+)!obviously(-).

we collect annotations on both topic and stance, using the ARC data as a starting point.


We define an optimization criteria to partition texts into coherent time windows, and provide an efficient dynamic programming algorithm, which reduces the average absolute prediction error by over an hour against baselines.

in the end, we use the winning BERT model to label our unlabeled data for training.


With these models, we show our book-length prediction algorithms and their corresponding metrics as well (Section 5).

a baseline model with random guessing would have an expected error of 6 hours.


With this, we now have a probability distribution over 24 hours for each window using our model.

to evaluate our methods, we construct ground truth for the books in our dataset.


Recognizing a story's flow through time is essential to understanding the text.

we focused on times that pinpointed hours within a day such as "two o'clock" or "noon".


However, since the semantic representation used by SCAN only covers a small subset of English grammar, SCAN does not enable testing various systematic linguistic abstractions that humans are known to make (e.g., verb argument structure alternation).

we include in our training set a single example to generalize from ("primitive exposure example") per generalization case that requires it.


It had 2 encoder and decoder layers, 4 attention heads, and a feedforward dimension of 512.

an error analysis revealed that the errors made by LSTMs were more systematic than those of Transformers.


Their PP modifier depth generalization accuracy was much lower (LSTM: 2%; BiLSTM and Transformer: near 0%).


Note that there is no token masked when producing sentence embeddings, which is different from pretraining.

through empirical probing over the embeddings, we further observe that the BERt sentence embedding space is semantically non-smoothing and poorly defined in some areas, which makes it hard to be used directly through simple similarity metrics such as dot product or cosine similarity.


If embeddings are distributed in different regions according to frequency statistics, the induced similarity is not useful any more.

(2) we first finetune both BERT and BERT-flow models on the SNLI+MNLI textual entailment classification task in a siamese fashion (Reimers and Gurevych, 2019).


On the STS-B dataset, BERT sentence embeddings are even less competitive to averaged GloVe (Pennington et al., 2014) embeddings, which is a simple and non-contextualized baseline proposed several years ago.

as discussed by Gao et al. (2019), anisotropy is highly relevant to the imbalance of word frequency


In this section, we report the scores computed from fully-trained models on the two benchmarks, Wikitext-103 and Melo-Lyrics, compared against baselines.

we found that the advantages of training balanced data distributions can be fully leveraged by sequentially performing tasks of frequency class prediction and token generation from the selected class.


It indicates that a decomposition of the softmax function without consideration of the data distribution (i.e., frequency distribution) aggravates both the likelihood and token diversity performances, regardless of the number of classes.

models tend to overproduce words frequently appearing in the data, while hardly utilizing informative words (Dinan et al., 2020).


As a simple yet effective remedy, we propose two novel methods, F2-Softmax and MefMax, for a balanced training even with the skewed frequency distribution.

often monotonous or dull, texts generated from existing methods do not fully reflect the rich diversity and expression in human language (Welleck et al., 2020).


The dataset was collected by asking workers to chat with each other naturally with a given persona.

bERTScore (Zhang et al., 2020) performs soft-overlap between candidate and reference sentences by using bERT embeddings directly without fine-tuning, and has been shown to correlate with human judgment robustly.


To overcome this issue, some learning-based metrics were proposed to train a coherence scoring model by considering the utterance-level semantics, such as ADEM (Lowe et al., 2017), RUBER (Tao et al., 2018), and BERT-RUBER (Ghazarian et al., 2019).

one branch deploys BERT (Devlin et al., 2019) to learn the coarsegrained utterance-level contextualized representations, while another learns the fine-grained topiclevel graph representations by constructing topiclevel dialogue graphs and applying a graph neural network on the graphs to model the topic transition dynamics.


First, the "sentence-encoder" or token-level RNN is a bi-directional LSTM (Hochreiter and Schmidhuber, 1997) encoding each sentence.

sentence Classifier As with the pointer network, we use a hierarchical LsTM to encode the document and produce a sequence of sentence representations d 1 , ..., d N where N is the number of sentences in the document.


In order to get an unconditional language model to do abstractive summarization, we can use the fact that LMs are trained by factorizing the joint distribution over words autoregressively.

we demonstrate the utility of language models by demonstrating that a variety of tasks can be modeled using language models.


In particular, more than 10% of the 20-grams from the abstracts generated by the pointing model are also found in the article, showing that it tends to copy long sequences of words.

the rest of the article is provided as additional in-domain training data for the LM.


We find the case where they are not statistically different, de-cs, to be particularly interesting: de-cs was the only language pair in WMT19 where the systems were unsupervised (i.e., did not use parallel training data).

we release our model, metrics toolkit, and preprocessed training data.


Our method achieves state-of-the-art performance on the most recent WMT shared metrics and QE tasks, without training on prior human judgements.

the model trained on ParaBank 2 prefers its own beam search output to a copy of the input.


BERT-era question answering systems have recently achieved impressive performance on several question-answering (QA) tasks.

an important difference between SQUaD and NQ is that NQ answers are often significantly longer, which is a prominent bias in the definition of what an answer is, which may be difficult to overcome with just unsupervised adaptation.


"Leaves" is a series of time intervals: The bus leaves at 10 am every day, so we will go to the bus stop at 9 am today.

in copular constructions, we choose to label the verb as the event, instead of an adjective or preposition.


While BLEU has higher correlation with crowd raters on whether the rewrite is sensical or grammatical, most correlation coefficients are less than .5, and many do not imply a positive correlation at all.

while the simplified rewrites in the wikiSplit dataset are not guaranteed to be meaning preserving and cannot be used in a benchmark, the original complex sentences are semantically and syntactically diverse, with adequate complexity.


We consider the state-of-theart seq2seq model trained on WikiSplit (Botha et al., 2018) and our rule-based model.

if you manage to appropriately split a sentence which many other workers have skipped, you will receive a bonus.


Our implementation is based on the model of Zhu et al.

making alignments can be formalized as a matching problem, which aims to find the most relevant AmR graph node for each target word.


Both settings give large improvements over the baseline.

we first force our model to generate a gold sentence in order to calculate the accuracies for node prediction and edge prediction.


One such widely studied task is image captioning (Hossain et al., 2019; Liu et al., 2019) which provides a textual description T given an image I.

place the dough in the oven at 180 degrees celsius circulating air for about 20 minutes .


In the problem defined above, our goal is to forward transfer the past knowledge to improve the new task learning.

a typical sentiment analysis (Sa) or social media company that provides sentiment analysis services has to work for a large number of clients (Liu, 2012).


Unlike traditional gates that regulate the feature information flow through the sequence chain, the goal of the proposed p-gates is to select useful parameters (which represent the learned knowledge from previous tasks) to be transferred to the new task to make it learn better.

for our lifelong learning setting, we use 5 random task sequences to compute the accuracy as different task sequences may give different results.


As we can see, continual learning models LWF-T and UCL (the latest algorithm) that only deals with catastrophic forgetting also achieve better results than I-CNN as the tasks are similar and share a great deal of knowledge (HAT is markedly worse).

for example, on Amazon-10, L2PG's average accuracy is 2.20% higher than LWf-T, 6.48% higher than HAT and 2.66% higher than UCL.


Automated radiology report generation has the potential to reduce the time clinicians spend manually reviewing radiographs and streamline clinical care.

we develop an end-to-end report generation framework that consists of two stages.


We extract the findings section from the radiology reports and then utilize spaCy for tokenization.

current clinical practice requires a radiologist with specialized training to manually evaluate x-rays and note their findings in a radiology report.


Pointer networks have been previously proposed as a way to copy parts of the input in hybrid seq2seq models.

the previously pointed to source token is tagged with insert.


We can use various architectures as the Encoder, which converts a sequence of tokens into a single vector.

we searched the size of the vocabulary among 8K, 16K, 24K, and 32K, and we selected 16K for Twitter(Ja) and Twitter(En), and 32K for weibo(Zh), SNLI, and Genre&Rating.


The goal of this study is to improve the performance of downstream tasks by optimizing the tokenization.

we describe the details of these datasets in the following.


freeze after k epochs achieves the balance of the stability and high F1.

each classifier is built with one linear full-connected layer and Softmax layer.


More and more applications like Twitter allow users to post multi-modal messages.

rule-based methods strongly rely on the collected patterns, and it is challenging to identify the sarcasm caused by uncollected patterns.


In this section, we first define the multi-modal sarcasm detection task.

we find that it doesn't bring much improvement even it contains more parameters.


in practical application not all the users can participate the training due to different reasons.


Motivated by the LSTUR model proposed by An et al.

for better privacy protection, we apply local differential privacy (LDP) technique to the local model gradients.


(Detailed results are given in the Appendix).

this metric represents how well the generated dataset represents the train set.


Markov Chain (MC), which was the preferred algorithm in (Akkaradamrongrat et al., 2019) showed worse performance than sample-copy (the baseline over-sampling approach) for most B low thresholds.

to feature-based over-sampling techniques, data augmentation generates additional samples through transformations applied directly to the data.


In this work we present BalaGen, a balancing-viageneration framework.

both of these approaches require additional domain data which is not always available.


To improve the performance of extractive summarization, non-neural approaches explore various linguistic and statistical features such as lexical characteristics (Kupiec et al., 1995), latent topic information (Ying-Lang Chang and Chien, 2009), discourse analysis (Hirao et al., 2015; Liu and Chen, 2019), and graphbased modeling (Erkan and Radev, 2004; Mihalcea and Tarau, 2004) .

first, we conduct a quantitative analysis on the CNN/Daily Mail corpus, based on the assumption that the writing style variability of summaries can be characterized through different combinations of sub-aspects (Lin and Bilmes, 2012).


As a straightforward and effective method, extractive summarization creates a summary by selecting and subsequently concatenating the most salient semantic units in a document.

in the news domain, neural models easily overfit by leveraging position-related features due to the prevalence of the inverted pyramid writing style.


In the following, we assume access to a generaldomain PTLM, as described in Section 2.1, and a corpus of unlabeled target-domain text.

one such model is BioBERT (Lee et al., 2020), which was initialized from general-domain BERT and then pretrained on biomedical scientific publications.


In this section, we use the proposed method to create GreenBioBERT, an inexpensive and environmentally friendly alternative to BioBERT.

the correct answer may appear several times, often with slightly different wordings.


at turn 4, the turn-level state predictor easily predicts the correct value unknown and TEN keeps the state correct.


In this paper, we proposed span attention to integrate n-gram into span representations to enhance chart-based neural constituency parsing.

the left side is the backbone chart-based parser.


Moreover, we run our models on the test set of each dataset and compare the results with previous studies, as well as the ones from prevailing parsers, i.e., Stanford CoreNLP Toolkits (SCT)14 (Manning et al., 2014) and Berkeley Neural Parser (BNP)15 (Kitaev and Klein, 2018).

recent chart-based parers (Stern et al., 2017; Kitaev and Klein, 2018; Gaddy et al., 2018; Kitaev et al., 2019; Zhou and Zhao, 2019) make rare effort to leverage such n-gram information


The other results are obtained from the best checkpoint of the best agent.

even though they still learn from both hard and soft targets, they only learn their poorly predicted tokens from other agents.


We first experiment with N = 10, L = 5, and D = 10K, but all methods can reach a nearperfect sequence accuracy (see Figure 3).

being able to control the aspects of the datasets allows us to compare the characteristics of the three inference methods more thoroughly and analyze the appropriate situations to apply each method.


Throughout all the experiments, three inference methods share the same model structure with dmodel = 512, dembedding = 512, nlayers = 1, r learning = 10 -5 , r teacher forcing = 0.5, and r dropout = 0.5 (Srivastava et al., 2014).

tagging performs worse than Recurrence in all other cases.


But, the decoder of Tagging produces a sequence of editing operations, such as deletion and insertion, that is later applied to the source text to yield the edited text via a realization step (Malmi et al., 2019).

we conclude that Recurrence is more data-efficient and overall better performs than End2end and Tagging in most situations, especially in AES (long-to-short).


The experimental results show that TED outperforms all unsupervised abstractive baselines on all datasets.

like most machine learning algorithms, summarization models can also be divided into supervised and unsupervised categories.


The pretraining employs automatic filtering mechanism and does require any human-labeled data.

tED innovatively encodes the similarity by a transformer encoder with much more modeling capability.


However, one should carefully examine and clean the source data to take advantage of lead bias, as the top three sentences may not always form a good summary.

agents confiscated electronic devices and computer files from roberts.


This causes NMT models to experience a limited set of subword candidates which are frequently observed in the training data.

we used the English-French language pair.


Our method significantly outperforms both the BASE and the SR.

our method does not replace the word but replaces its subword composition.


Our baselines are NMT models trained with deterministic segmentations (BASE) and models trained with the subword regularization method (SR) (Kudo, 2018).

the diversification of subword segmentations mostly relies on the pre-trained subword language models from which erroneous segmentations of unseen words are less likely to be sampled.


Proposition 1 shows that the objective is maximized when the mutual information is maximized to I max .

all model and training details are in appendix B.


We ablated in-domain pretraining, mixup training, confidence penalty loss, and self-training progressively, the results of which are shown in Table 3.

there exist 10 groups of the relationship types, and five of these (CPR:3, CPR:4, CPR:5, CPR:6, and CPR:9) were used in the evaluation.


Biomedical text data, including PubMed abstracts, usually contain information about biomedi-cal entities and their relationships with each other.

the max, min and std precision of our proposed model are 80.08, 75.64 and 1.56 while those of BioBERt are 78.64, 76.18 and 0.83.


To show the significant improvement in terms of classification performance, we report the mean and standard deviation values of every experiment in the ablation study.

several approaches, called "calibration" techniques, have been applied to several domains that require high reliability, such as autonomous driving and medical diagnosis (Guo et al., 2017;Jiang et al., 2012).


It also explains the relatively high ROUGE scores in the evaluation.

the initial learning rate is set to 1e - 9 and linearly increased to 0.001 with 16000 warmup steps.


We thank the anonymous reviewers for their valuable comments.

we leverage a hierarchical structure to reduce the burden of computing.


To improve the clinical correctness of the generated reports, Liu et al. (2019a) and Irvin et al. (2019) adopted clinically coherent rewards for RL with CheXpert Labeler (Irvin et al., 2019), a rule-based finding mention annotator

although the F-scores of the no-mention labels are high, the F-scores of the positive, negative, and uncertain finding labels are relatively low.


The underlined part represents the correct description corresponding to the italic part.

our proposed RL-DA generated a correct description: "there is no pleural indentation".


Although the above-mentioned distributional hypothesis is proposed for language models, if we look at the knowledge graph from the perspective of this hypothesis, we can find that similar hypothesis exists in knowledge graphs (KGs).

knowledge graph G can be redefined as G = (V, E), where V represents the nodes in G, involving entities in E and relations in R, and E denotes the directed edges among the nodes in V .


where, W f , W V and Wh are the trainable weight matrices.

also, the mention of different images in the contextual information confuses the model in selecting the correct images.


The appropriate responses to the user queries are highly dependent on the visual information pertaining to the different aspects of the various images in the conversation.

experimental results show that our proposed methodology outperforms the baseline models in the case of both automatic and human evaluation metrics.


Nevertheless, existing GCN-based ED methods do not consider dependency labels, which may serve as significant indicators to reveal whether a word is a trigger or not.

in addition, our statistical results on the benchmark ACE2005 dataset show that "nsubj", "dobj" and "nmod" take up 32.2% of triggerrelated dependency labels (2.5% for each relation on average among all 40 dependency relations), which means that simultaneously modeling syntactic structure and dependency labels can be crucial to make full use of the dependency trees to further improve the performance of ED.


Event Detection (ED) is an important information extraction task that seeks to recognize events of specific types from given text.

a meanpooling operation is applied to compress features since it covers information from all channels.


In this paper, we propose a novel architecture named Edge-Enhanced Graph Convolution Networks (EE-GCN), which simultaneously exploits syntactic structure and typed dependency label information to perform ED.

the entity type and dependency label embeddings are randomly initialized.


A generated sentence is verifiable if it can be corroborated or disproved by Wikipedia, and we find that the verifiability of generated text strongly depends on the decoding strategy.

intuitively, random choices might hamper verifiability when sampling a token in specific positions of the sentence, for instance, in a named entity, potentially making the overall sentence non factual.


At test time, the model is presented the conversation history and 20 candidate responses and the model has to pick the correct response.

this chatbot requests users to provide natural language feedback when the users are dissatisfied with its response


It is reasonable to assume that the larger position embeddings have similar first 512 values with the small one since they all express the corresponding relationship between tokens when the input length is less than 512.

for a particular dataset, when we set the ERLength of the BERT, letting it exceed more data's DLength can always bring more improvements.


Among all the models, only Seq2UMTree can be applied for all sentences in both datasets 2 and the space complexity is O(2l + r).

previous work uses pIpELINE to extract triplets from text (Nadeau and Sekine, 2007;Chan and Roth, 2011).


β is a hyper-parameter set to 1.0 by default.

the NumAsTok baseline has a finite numeral vocabulary and treats all the OOV numerals as UNK num .


This experimental methodology, inspired by Caglayan et al. (2019), creates a systematic gap in the speech signal that can be resolved by leveraging the visual context; for example, when the audio drops during online distance-based learning or video calls with family and friends.

for purple and sign, the model attends to the correct proposals, but fails to recover the words pink and ship, respectively.


We present BLEU score for our method and compare it with SVD, GroupReduce (Chen et al., 2018), Structured Emedding (Shi and Yu, 2018), Tensor Train (Khrulkov et al., 2019) and a smaller transformer network with the same number of parameters.


During the ES, a population of P trained models are kept throughout the search phase, where initially, the population is initialized with random architectures.

this process is computationally very expensive, making it infeasible to run on a single GPU in a reasonable amount of time.


We first compare our Transformer encoder with the previous models in the single-criterion scenario.

we merge the wikipedia corpora used in "8Trad" and "8Simp" to form a mixed corpus, which contains both the simplified and traditional Chinese characters.


In this work, we only adopt the vanilla Transformer encoder since we just want to utilize its selfattention mechanism to model the criterion-aware context representation for each character neatly.

we do not balance the datasets and randomly pick 10% examples from the training set as the development set for all datasets.


"Char Types" is the number of unique characters.

it is unnecessary to use a specific private layer for each criterion.


Since our DIPAIRTSF model is end to end trained, the model should learn to push the information of the full input sequence to arbitrarily selected (N + M) token embeddings.

techniques that model the texts independently such as the dual-encoder models (Das et al., 2016;Johnson et al.; Chidambaram et al., 2019; Cer et al., 2018; Henderson et al., 2017; Reimers and Gurevych, 2019) are able to run efficient inference on large-scale text pairs.


Training Details For fair comparison, we use the same hyper-parameter settings and the training strategy as Ahmad et al. (2019) to train the parsing models.

this method may suffer from imperfect word alignment between two languages.


We give details and results regarding our column pruning approach in Section D. Full results for SQA are displayed in Section E. Section F shows the accuracy on the pre-training tasks held-out sets.

the model does not use special embeddings to encode the table structure but relies on a template approach to format the table as natural language.


The sequence labelling model is optimized by Adam optimizer (Kingma and Ba, 2015) with batch size 64, learning rate 0.0006 and decay rate 0.992.

suppose s i,j is the similarity score between i-th and j-th words, the post score post i,j is: Then, we normalize the scores by a softmax and get the probability distribution of replacing words.


MAT applies word masks or substitutions (details in Method section) when computing loss from adversarial examples, which forces the model to predict the right labels with no word information or wrong information.

despite the success of adversarial training (AT) on text tasks such as text classification (Miyato et al., 2017) and part-of-speech (POS) tagging (Yasunaga et al., 2018), its gains on other sequence labelling tasks, such as named entity recognition and chunking, are not significant (Yasunaga et al., 2018).


Masked language model (Devlin et al., 2018) smooths this inconsistency by applying replacement of tokens for some data while masking the rest (equivalent to word dropout).

early stopping is applied based on model performance on the development set.


For all the experiments, we use finetune the pretrained model for 3 epoches with learning rate 2e-5 and batch size 32.

these results suggest that our method is more suitable for fine-tuning with smaller amounts of data, and that our approach to injecting the label information is at least not detrimental to the original pretrained model.


This is reflected by the clearly opposite trends observed in Figure 3 vs. those observed in Figure 1a.

First, we trained a model for the North-Germanic family (NG) with three languages – Danish (DA), Swedish (SV), and Norwegian (NB).


In this work, the graph edges are labelled dependency relations, which are predicted as part of the actions of a transition-based dependency parser.

after proposing two novel Transformer models of transition-based dependency parsing as strong baselines, we show that adding the proposed mechanisms for conditioning on and predicting graphs of Graph2Graph Transformer results in significant improvements, both with and without BERT pre-training.


As shown in the table, models that used only a single attention component, be it Supervised Self-Attention based (SSs) or Task-specific attention based sentence representation (TSS) achieved the same F1-score for the entity recognition task.

to avoid over-fitting, we apply a dropout strategy (Ma and Hovy, 2016; Srivastava et al., 2014) of 0.5 for our model.


The size of the learned character-level embedding are 100 dimensional vectors.

single attention is insufficient in exploring this multi-aspect information and consequently risks losing important cues.


Several studies have explored (Guan et al., 2019) SRL with deep learning techniques.

there are several SRL annotation conventions, such as PropBank (Palmer et al., 2005) and FrameNet (Baker et al., 2007).


If the task is simple and easy to solve, why not just build a rule-based system rather than a usersimulator that is then used with RL techniques to train the dialogue system, where more uncontrollable factors are involved?

for each evaluation task, we will first present an Mturk worker with a randomly sampled user goal, which contains the constraints about specific domain slots and some slot information that the user is looking for.


How sensitive are adversarial learning to pretrained dialogue policy?

since the output of the dialogue policy is a set of discrete dialogue actions, it is difficult to pass the gradient update from the discriminator to the policy model.


1) datasets, a particularly interesting QA task that provides a flexible space of candidate answers along with a simple evaluation.

the views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.


However this sophistication not only requires adhoc heuristics that are tailored to a specific formalism (AMR in this case) but also relies on alignment information with the source words.

we then follow the binary relations to expand the graph.


This also differs from Karamanolakis et al. (2019), which updates the weights of the initial seed words but does not provide pseudo-labels for documents with no seed words.

we expect such correct translations to lead to further improvements over automatic translations.


Google Translate erroneously returns "medical" as a Uyghur translation of the word "medical."

in Sinhalese, we use 5,000 unlabeled documents for training the student and 36 labeled documents for evaluation.


This is explained by the fact that CLTS prioritizes the most indicative seed words for translation.

we now evaluate CLTS for several cross-lingual text classification tasks in various languages.


5a, even compared with L2L ptr which equipped with pointer network and tend to copy from source document.

we claim that a better understanding of dataset biases can help us interpret models' discrepancies.


We use cross entropy as loss function to train LSTM non and Trans auto .

we ask two questions as follows: Q1: How do different neural architectures of summarizers influence the cross-dataset generalization performances?


In particular, Voita et al. (2019) studies the evolution of representations from the bottom to top layers and finds that, for MLM, the token identity tends to be recreated at the top layer.

Figures 10a, 10d, 10g, 10j, 10m show left and right cws for sentences belonging to short length category (l ≤ 25).


Similarly, grid-based features Vg are fused with question embedding q by concatenation.

vQA settings: There are two main vQA settings, namely multiple choice and open-ended following (Antol et al., 2015).


However, most existing domain adaptation methods focus on single-modal tasks such as image classification or sentiment classification, and thus may not be directly applicable to multi-modal settings.

question encoding: The question q of length T is first tokenized and encoded using word embedding based on pre-trained GloVe (Pennington et al., 2014) as S = {x0, x1, ..., xT }.


It reduces the difference in distributions by transforming the feature representation of the data in the target domain.

we validate our proposed method on two popular VQA benchmark datasets, VQA 2.0 and Vizwiz, in both directions of adaptation.


We study the problem of visual question answering (VQA) in images by exploiting supervised domain adaptation, where there is a large amount of labeled data in the source domain but only limited labeled data in the target domain, with the goal to train a good target model.

most existing domain adaptation methods focus on single-modal tasks such as image classification or sentiment classification, and thus may not be directly applicable to multi-modal settings.


Gilmer et al. (2018) laid out a set of potential constraints for the attack space when generating adversarial examples, which are each useful in different real-world scenarios.

additionally, synonym substitution methods, including TEXTFOOLER and GENETICaTTaCK, often require that words be substituted only with neighbors in the counter-fitted embedding space, which is designed to push synonyms together and antonyms apart (Mrksic et al., 2016).


The lack of a consistent definition and standardized evaluation has hindered the use of adversarial examples to understand and improve NLP models.

Our framework enables fair comparison between attacks, by separating effects of search methods from effects of loosened constraints.


Perturbations generated by TEXTFOOLER were rated an average of 3.28, while perturbations generated by GENETICATTACK were rated on average 2.70.

our inspection of the perturbations revealed that many violated these constraints.


To enforce grammaticality, we added Language-Tool.

due to the rule-based nature of grammar, automatic evaluation is preferred.


The Stage 1 workers are equally distributed across the city sub-sections, so the dataset splits are not biased toward specific workers.

assembly occurs in a smaller environment, requiring agents to focus less on understanding paths than in navigation and more on understanding the 3D spatial relations of objects from the limited egocentric viewpoint.


Object manipulation and configuration is another subject that has been studied along with language and vision grounding (Bisk et al., 2016;Wang et al., 2016;Li et al., 2016;Bisk et al., 2018).

c: Objects will always be a book, hourglass, mug, bucket, ball, tv, or bowl, but may vary in color and texture.


For word and action embedding sizes, we use 300 and 64, respectively.

most studies focus on addressing the problem in relatively simple environments from a third-person view.


Table 1: Lengths of the instructions (in words), paths, and action sequences for both turns across all subsections in the city.

in the 2nd turn, our model fails to find the "striped red mug" by missing the left turn around the "yellow and white banner".


We incorporate the naturalistic variation mentioned below in Section 3 to these datasets as they are used extensively in goal-oriented dialog research and use them as benchmarks for our experimental evaluation 2 .

we use the best performing hyper-parameters reported by both models -BossNet and GLMP for each dataset.


Starting from early work by Agrawal et al.

results of the experiments are reported in Table 4.


We consider some of the major US healthcare providers, with which almost everyone has interacted at different levels (insurers, pharmacy chains, ...): thus, not only finance experts (example (a) in Table 3) and local sources (b), but also politicians (c), physicians (d), policymakers (e) and the general public are interested in their outcome, resulting in a dataset which collects different registers.

sD on complex and articulated texts, such as news articles, has been considerably less studied, mainly due to the scarcity of published datasets (Pomerleau and Rao, 2017;Hanselowski et al., 2019).


We collected 665,667 tweets posted between February 28, 2020 and May 8, 2020, with a maximum of 10,000 samples for each day using Crimson Hexagon 1 .

we can produce a simple white-box method for identifying who-needs-what sentences.


In the generic summarization b(w) = 2, for comparative summarization b(w) = 3, as used by Gillick et al. (2009).

most extractive summarizers have two components: sentence scoring and selection.


Note that as shown in Figure 1, because non-word misspellings are preprocessed already, the detection of these non-word misspellings can be trivially accomplished, which results in all models having non-word recall of 1.000.

since the transformer-encoder (Vaswani et al., 2017) computes contextualized token representations, we take h char , the [CLs] token representation of each character sequence as the local character-level representation of s * .


Both of the baseline models (row 1 and 2) perform poorly, because they perform spelling corrections upon character sequences, which disregards the semantics of the context, as their poor real-word performance in Table 2 row 1 and 2 suggests.

we use the transformer-encoder (Vaswani et al., 2017) to encode the input sequences and denote it as Encoder.


Our work focused on a specific instantiation of channels as languages.

at inference (generation) time, we can generate unconditionally by seeding the canvas with the [SEP] token and predicting the first actual token or provide as much, or as little, partial/complete sequence in each channel.


We experiment with the Multi30K dataset containing English, French, Czech, and German.

KERMIT is able to model the joint p(x, y), conditionals p(x | y), p(y | x), as well as the marginals p(x), p(y).


This observation indicates that incorporate different embeddings as input is more effective than directly using pre-trained models.

next, we use two matrices to map them to their embeddings, with e kc i,j referring to the embedding of k c i,j and e vc i,j for v c i,j , respectively.


Moreover, in most cases, one would like to incorporate more than one types of extra features.

we try different values for hyper-parameters for our model, presented in Table 9.


First, each word embedding is weighted by a/(a + fr ), where fr stands for the underlying word's relative frequency, and a is the weight parameter.

dCT is a way to generate document-level representations in an order-preserving manner, adapted from image compression to NLP by Almarwani et al. (2019).


For ST models, the task model and simulator are of the same architecture, but we do not evaluate MT conditions since RoBERTa is not generative.

multi-agent game results appear in Table 3, though we note that RL results should be cautiously interpreted as we observe unstable training behavior from this method.


Early work on this topic proposes to generate explanations for images that are descriptive as captions and discriminative as labels (Hendricks et al., 2016).

the differences between them could result from their pretraining procedures, architectural differences, finetuning sample efficiency, or another cause.


Our aim is for explanations to better communicate the task model's reasoning process, without adopting the trivial solution, i.e., directly stating its output.

we show that our LAS scores provide a deeper understanding of explanation effectiveness than metrics like BLEU and discuss their relationship with our expert simulation analysis and crowdsourced human quality ratings.


While using gold linguistics task data (PTB, CoNLL-2005, andCoNLL-2009) with 10% probability, we only take one sentence as input that [CLS] and [SEP] are first and last tokens respectively.

for task-specific layers including syntactic and semantic scorers and decoders, we set the same hyperparameters settings as (Zhou et al., 2020).


Consequently, we propose to inject the knowledge from these syntaxbased importance scores into the graph-based models for ABSA via the consistency with the modelbased importance scores.

Model performance on the MAMS development set when the diversity term Ldiv is directly computed from the gate vectors.


For instance, an ABSA system should be able to return the negative sentiment for input sentence “The staff were very polite, but the quality of the food was terrible.” assuming “food” as the aspect term.

using the representation vectors from the graph-based models, we compute a second score for each word in the sentences to reflect the model's perspective on the importance of the word for the sentiment of the aspect terms.


The preceding results show a decrease in number of primed instances as contextual constraint increases.

we use the Semantic Priming Project (SPP) (Hutchison et al., 2013) as our source of human priming experiment data.


The PREMISE sentence describes a real-world situation and is always assumed to be true.

sUMMARY For this task, given a GENERALIZATION about a socially normative behavior or judgement: Write an UNDERMINING CONTEXT in one sentence.


These statements represent generic commonsense hypotheses about social behaviors and their acceptability that are held as norms in a society.

an UNDERMINING context provides a situation that weakens the generalization; it makes the generalization less relevant or effective.


Recent work has proposed hyperbolic neural components, such as word embeddings (Tifrea et al., 2019), recurrent neural networks (Ganea et al., 2018) and attention layers (Gulcehre et al., 2019).

Figure 6: Norm of text vectors for the Euclidean and hyperbolic model.


The types are highly correlated given that they often co-occur in similar contexts.

researchers have incorporated these isolated components into neural models, whereas the rest of the layers and algorithms operate under Euclidean assumptions.


In this manner, we make consistent use of the same analytical model of hyperbolic space across all components, which eases their integration.

we hypothesize that our proposed hyperbolic model will benefit from this representation.


We first validate the advantage of our architecture on two tasks in dialogue history object retrieval described in Section 2.

referring expressions are a ubiquitous part of human communication (Krahmer and Van Deemter, 2012) that must be studied in order to create machines that work smoothly with humans.


However, our architecture is difficult to directly apply to referring expression generation because it outputs modulated feature maps.

this task is repeated until the size becomes 1/32 of the original image.


Our experiments show an improvement on the Augmented-Flipped testing set after correcting the conflicted labels in the Original training set, and the combination of the two methods produce a model that gives the best performance across all testing sets.

in addition to the original QQP dataset, we derived three additional datasets using the data augmentation and label correcting methods introduced in Section 3.


The precision parameter of the path given the cluster helps in penalizing the 'BORN IN' path.

we report test results for the best hyper-parameter values that we got on this validation set.


The process will generate about O(n) candidate regions.

to HiReAWR , the absolute F1 measure improvement of HiRe HRR is 0.6%.


Note that high translation quality of Transformer models entails a large number of parameters.

hence, we set pN R = 2000 for retraining of quantization baselines, and for experiments where we quantize and retrain each block in Transformer at a time, we set pN R = 1000.


Interestingly, our (2.5, 1.8, FP) model with the average of 11.3-bit outperforms the 2-bit baseline in terms of inference speed.

in addition, as we illustrate later, even assigning different quantization bits for each row of an embedding block can further reduce the overall number of quantization bits of the entire Transformer model.


For Transformer, we suggest the following two techniques to decide the number of quantization bits for each block: 1) in the case of embedding block, frequency of each word is taken into account and 2) for encoder and decoder blocks, we find the minimum number of quantization bits for each type of sub-layers that allows reasonable degradation in BLEU score after quantization.

as for inference speed up, addressing memory wall problems may be of higher priority rather than attaining a low number of quantization bits.


In the meantime, proper knowledge utilizing also gets rid of repetition to some degree and makes the dialog more coherent, which has been proved during the previous evaluation phase.

the forget gate controls how much knowledge to be discarded during the multi-source fusion phase.


For manual evaluation, we first select metrics Fluency and Coherence clearly defined in (Wu et al., 2019) to measure whether the dialog agent can express fluently and logically.

This enhances the system’s expressive ability especially in terms of the out-of-vocabulary words. Wu et al. (2019) applied KnowlegePost (Lian et al., 2019) to minimize the divergence between knowledge prior and knowledge posterior distribution to learn a better knowledge representation and concatenated the knowledge context vector with decoder feed.


Table 3: Examples of coreference errors made by Chirps and corrected by Barhom et al. (2019): 1) false positive: wrong man / two men alignment (disregarding location modifiers).

it extracts binary predicate-argument tuples from each tweet and aligns pairs of predicate mentions whose arguments match, by some lexical matching criteria.


Following this, we explore the multi-domain case of the Overnight dataset wherein there is no gold-standard training data in either language.

prior study broadly found this technique to yield improved parsing accuracy (Iyer et al., 2017;Dong and Lapata, 2016;Finegan-Dollak et al., 2018), a crosslingual implementation requires crafting multiple language-specific trans-lation tables for entity recognition.


This work is closely related to relation reasoning in knowledge bases and few-shot learning.

to summarize, our main contributions are: (1) we study the problem of few-shot multi-hop relation reasoning over KB, which is new and important; (2) we propose a novel model called FIRE to solve the problem by exploring several beneficial components; (3) we conduct experiments on two datasets and the evaluation results demonstrate the superior performance of FIRE over state-of-theart methods.


PBoS-n showed good improvement for the Polyglot RW case (25 to 32), matching the performance of the other two.

we found that PBoS is insensitive to subword length constraints and decide to keep the setting simple.


Note this has nothing to do with embeddings, so no training is involved in this experiment.

the model can be trained by fitting a set of pre-trained word embeddings.


Note that we do not assume a vocabulary.

in summary, we find 1) that PBoS surpasses KVQ-FH for English and German and is comparable to KVQ-FH for italian; 2) that PBoS and KVQ-FH surpasses BoS for English, German and italian; and 3) no definitive trend among the three models for Russian.


We omit the prediction time for KVQ-FH, as we found it hard to separate the actual inference time from time used for other processes such as batching and data transfer between CPU and GPU.

for each segmentation, we compose a word vector as the sum of all subwords that appear in the segmentation.


Note that, our methods and the other lexicon-based baselines are only applied to words being predicted as class K class by SourceBERT.

in this section, we detail the construction of the partially labeled dataset D t for the target task.


With experiments conducted on two well-known public datasets, i.e. Yelp and Amazon, we demonstrate that the proposed methodology outperforms existing unsupervised methods very consistently in fluency, and presents competitive results in terms of sentiment conversion and content preservation.

Its actually worse than a little better than the filter that came with the unit.


Main results on the MS MARCO passage retrieval task are shown in Table 1, comparing BERTlarge (Nogueira et al., 2019a) to T5 models of different sizes.

we map a relevant document to "hot" and a non-relevant document to "cold".


Robust04 (Voorhees, 2004) is the test collection from the TREC 2004 Robust Track.

in addition, we would like to thank Google Cloud for credits to support this work.


We believe these two issues are closely related.

the main advantage of our approach is that by "connecting" fine-tuned latent representations of relevance to output target tokens, we can exploit the model's latent knowledge (e.g., of semantics, linguistic relations, etc.)


Unlike the fullyconnected layer in the encoder-only approach, T5 can exploit the part of the network used for generating output tokens.

we apply inference directly using our model trained on the MS MARCO passage data; Robust04, Core17, and Core18 relevance judgments are only used as a test set, which makes our results zero-shot.


RnnOIE-Supervised (BERT) even perform worse than ClausIE and OpenIE4 on WEB and NYT.

two training instances are automatically generated using dependency pattern for predicates "operates" and "has".


There is a growing amount of research literature on various debiasing methods to improve the robustness of models against individual biases in the training data (Clark et al., 2019;Mahabadi et al., 2020;Utama et al., 2020;He et al., 2019;Schuster et al., 2019b).

all our models are trained for 2 epochs.


This requires significant modifications to the model and may make it harder for the decoder to generate fluent and diverse text as found in the targets.

given a source and a target, how can one know if a target token wyt is unsupported by the source?


models pick up on the noise and may hallucinategenerate fluent but unsupported text.


Furthermore, we break down the performance of overall SQL queries into each $VALUE slot, results presented in Table 3.

for Acc ex , we execute both the generated query and the ground truth and compare whether the retrieved results match each other.


To alleviate the above problem, we propose AirConcierge, an end-to-end trainable text-to-SQL guided framework to learn a neural agent that interacts with KBs using the generated SQL queries.


In the transformer, the representation of each query token gets updated by self-attending to the representations of all the query tokens and graph nodes in the previous layer.

to mitigate these problems, we manually filter the data by removing graphs with disconnected components, lowquality instances, or excessively long descriptions


Parsing text into structured semantics representation is one of the most long-standing and active research problems in NLP.

we explore the possibility of augmenting the user-generated data with synthetic data in order to train a better model.


Because of the nature of motion and its importance in natural language, we selected it as the first semantic prime to explore.

if there are multiple iNANiMATE entities in motion please highlight all of them.


Since the vocabulary of the connection phrases is limited, we automatically generate text stitch training data by dropping certain words in free texts with simple rules.

we require the removed tokens to have the aforementioned POS tags, plus adj, adv, det, intj, pron which are also found in connection phrases sometimes but with lower probability.


More training details can be found in the appendix.

it is less sensitive to expression variations and content orders than other automatic metrics.


Since it is a corpus of spoken language, it has some characteristics that differ significantly from written corpora: there is no punctuation, nor any sentence or paragraph mark, there are speech turns and disfluencies, like word repetitions (e.g.

several metrics have been developed to evaluate automatic coreference resolution systems (see (Poesio et al., 2016) for an overview).


Table 6 presents the number of spurious and missing mentions, compared to the total number of mentions in the gold standard corpus (that is, the mentions that should be detected) and the predicted corpus (mentions that the system has actually found).

coreference is easier to solve in ANCOR (38.95% of coreference only errors).


However, we are planning to release the primary annotations in their original form as part of our additional material.

we generally follow this idea: reported is more summarizing and less precise, while indirect ST&wR can usually be read as a transformation of direct ST&wR that allows us to reconstruct the 'original' quote in more detail.


One reason for this is that the newspapers and magazines that were used for our corpus tend to contain quite complex texts (political commentary and reports in historical German).

It would be orthogonal to our categories and is thus not included. In addition, many aspects of mind representation that are more removed from the idea of thought as speech and were pointed out by literary scholars (e.g. Palmer, 2004) are excluded in our annotation.


A direct predecessor of our corpus is Brunner (2015), a corpus of 13 German narratives from the 18th and early 19th century (approx. 57,000 tokens)

an author of the 'Digitale Bibliothek' could only be selected again after all other authors available in this decade had already been drawn.


By this very primitive matching, we naturally neglected polysemy, and, due to the noise in the Treq table, some equivalents were admittedly suboptimal.

even a noun in the position of the indirect object is classified as an accusative noun, although it corresponds to dative in most languages and some grammars call the two morphologically distinct pronominal cases in these virtually non-inflective languages subject/object case.


Figure 1 illustrates an example of Chinese discourse parsing tree.

as mentioned by Rhetorical Structure Theory (Mann and Thompson, 1988), a discourse is composed of elementary discourse units (EDUs), which can be formed into a hierarchical structure by relating each other with discourse relations.


For example, we observed that the most frequent collaboration labels for claims are B-extension (43.9%), B-new (37.0%) and B-challenge (11.1%).

during discussions with teachers where we visualized the collaboration annotations in the corpus that came from their particular classrooms, we found that teachers were very curious about whether students were introducing new information into the discussion or building off of what was previously said.


Explanation: Infinitival complements of verbs are not segmented as separate EDUs.

explanation: Infinitival complements of verbs are not segmented as separate eDUs.


the model achieves a state-of-the-art F-score of 96.7 for the RST-DT corpus (Carlson et al., 2003) improving on the previous best model by 7.2%.


By using a separate connective model first, the window for the neural model is already determined.

this data collection consists of two parts, bbc-news and bbcsport, both from the years 2004 to 2005.


In Table 6, we study differences in the position of Arg1 compared to Arg2.

because the predicted annotations are not as reliable as the gold annotations, the model could easily make false assumptions about the data when using a bad balance of gold labels and proxy labels.


(3) To exploit implicit connectives given by treebank annotators, we add a task named implicit connective prediction at the fine-tuning step.

they singled out this task while we perform it jointly with MaskedLM and next sentence prediction.


This selection of the second-echo images renders the two sets of functional images as comparable as possible.

these include bigram lexical surprisal, trigram lexical surprisal, bigram POS surprisal, trigram POS surprisal, CFG surprisal, CFG bottomup node count, CFG top-down node count, MG bottom-up node count, and MG top-down node count.


It shares an underlying book with the cross-linguistic project, Alice in the Language Localizer Wonderland (Fedorenko, 2016) and an emphasis on naturalistic text with the Narrative Brain Dataset (Lopopolo et al., 2018) (hereafter NBD).

sixteen participants were scanned with a three-echo EPI sequence where the field of view was 240 x 240 mm resulting in 33 slices with an inplane resolution of 3.75 mm 2 and thickness 3.8mm.


Table 1 shows details about the GoodWriting dataset.

we use optuna 9 to optimize hyperparameters in feature-based models.


In this study, we created an automated essay scoring (AES) system for nonnative Japanese learners using an essay dataset with annotations for a holistic score and multiple trait scores, including content, organization, and language scores.

by contrast, neural-network-based methods do not need to create features and have produced state-of-the-art results in various datasets (Ke and Ng, 2019).


We follow the feature set described by Zesch et al. (2015).

we developed AES systems using two different approaches: a feature-based approach and a neural-network-based approach.


While developing a computational model for humour recognition in Portuguese, we decided to tackle two styles of humour and gathered some texts for validation purposes  (Clemêncio et al., 2019).

four sets of negative examples were gathered, namely: news titles from Reuters; proverbs on the Web; sentences from the British National Corpus (BNC); and sentences from the Open Mind Common Sense project.


Section 4. presents the experiments we carried out on our data.

when reporting sexist content, the speaker is still conveying lack of commitment, and a general sense of disapproval or dismissal may emerge.


The null hypothesis is that all models have equivalent results and so the rankings must be equal.

the remaining 165 features were computed for each text in the corpus.


All 19 newly proposed features are computed on the linked documents.

the drop in performances is higher for the combination of surface and shallow features (-.09) than for the graph-based deep semantic features (-.04) on the five-level classification task.


We use all 1,000 articles (200 titles on five complexity levels) as our dataset, and apply ten-fold cross-validation setup with ten repetitions (in Weka Experimenter) without controlling for which titles and text versions end up in which fold.

this can be explained by the fact that in the binary classification task the system needs to decide which of the two versions of the same text is simpler.


To the best of our knowledge, there were no other studies that attempted at five-level text classification in the scope of conceptual text complexity.

we behave as we had 1,000 independent articles on five different complexity levels, but choose to have 200 titles on five complexity levels, in order to: (1) allow for learning subtle differences between different complexity levels of texts treating the same topic; and (2) avoid bias that might arise from topic differences if we used corpora with five complexity levels, in which each level has different topics (as in the case of typical language learners corpora).


The task of automatically assessing conceptual complexity has only been proposed recently.

on the five-level classification task, they were outperformed by all other feature sets.


For ZuCo 1.0 the normal reading and task-specific reading paradigms were recorded in different sessions on different days.

on the basis of the GECo and ZuCo 1.0 corpora, we extracted the following features: (i) gaze duration (GD), the sum of all fixations on the current word in the first-pass reading before the eye moves out of the word; (ii) total reading time (TRT), the sum of all fixation durations on the current word, including regressions; (iii) first fixation duration (FFD), the duration of the first fixation on the prevailing word; (iv) single fixation duration (SFD), the duration of the first and only fixation on the current word; and (v) go-past time (GPT), the sum of all fixations prior to progressing to the right of the current word, including regressions to previous words that originated from the current word.


For the document (a) (slides) in Figure 1, only Str classifier correctly predicts the class label with a probability of 0.74.

as an example, consider a scholarly works repository, which contains publications that are typical for an institutional repository such as research articles, white papers, slide decks from presentations, and other scholarly publications.


All Train-1, Dev, and Test follow a similar distribution as the original datasets.

co-training methods are semi-supervised and assume that all views are "sufficient" for learning.


Hence, it depends on the mean frame size of the aspect flows generated from a dataset, so that the model can avoid the unwanted effect of generating various sub-frames in a short frame.

frequencydomain features are extracted from the sound spectrum, a representation of the distribution of the frequency content of sounds (Giannakopoulos and Pikrakis, 2014d).