A new compound, Ag 16 B 4 O 10 , has been obtained as a coarse crystalline material via solid state synthesis.

The shiny black crystallites are insensitive to humid air and start decomposing thermally at $623 K with silver metal and amorphous B 2 O 3 resulting in the nal solid residues, see Fig. S1.† The title compound can be prepared from various starting materials in the required molar ratios, as there are boron(III) oxide, boronic acid, silver oxide and nally elemental silver as an essential component, while adding varying amounts of water as a mineralizing agent.

At rst glance it appears unintuitive that for the synthesis of such a considerably reduced material applying moderately elevated oxygen pressures of 2 Â 10 7 to 5 Â 10 7 Pa is an indispensable requirement.

However, running the experiments at ambient pressure results in the decomposition of silver oxide to the metal, while a too high oxygen pressure would end up in the formation of silver(I) borates.

So, the synthesis of the title compound is a delicate balancing act, and even for the optimized synthesis conditions, given above in detail, one or the other synthesis run may fail in yielding single phase products.

Fig. 1 displays a Rietveld prole t of an X-ray powder diffractogram of a sample obtained using the optimized synthesis procedure; a three-phase renement has revealed that only traces of Ag (2.1%) and Ag 2 O (2.3%) are present.

The atomic parameters obtained are given in Table S4.†


The constitution has been conrmed unambiguously by single crystal X-ray structure determination.

As illustrated in Fig. 2, the general structural organization of Ag 16 B 4 O 10 derives straight forwardly from a ccp pattern of elemental silver: out of every twenty silver atoms four adjacent ones are replaced by boron.

Since boron atoms are of considerably smaller size compared to silver, the substitution generates some empty space tolerating the insertion of oxygen atoms to form the polyoxoanion B 4 O 10 8À .


From this picture it is immediately obvious that the title compound ideally matches the notion of classifying crystal structures of extended inorganic solids rather based on the packing of the cations 31 than on that of anions.

Even beyond, it lends strong support to the further reaching concept according to which certain oxides may be regarded as alloys being stuffed with oxygen.

32,33 This particular structural interrelation is underpinned by Fig. 3 and 4, showing the crystal structure of the title compound along the view directions [001] and [121], respectively.


This article is licensed under a Creative Commons Attribution 3.0 Unported Licence.


There are no conicts to declare.


As occurs with the chemical bond, the terms valence and oxidation state enclose elusive concepts that are very useful in everyday chemistry but it is difficult to establish a unique agreed definition for them [1].

These concepts lack both the possibility of being measured in the laboratory and a quantum-mechanical operator allowing their calculation.

In this context, sub-valent compounds tend to exhibit anomalous compositions and they are compounds where the bonding of one of the atoms involves fewer electrons than expected from the electron counting of its valence shell.

In the solid state, sub-valence is associated with oxidation numbers lower than nominal values for any of the atomic constituents of the compound [1].


According to a regular composition, one would expect silver atoms to participate in the electron counting of a chemical bonding network with one electron, that in its 5s valence shell.

In the Ag16B4O10 title compound of the edge article commented here [2], however, only 8 electrons are needed to account for the charge of the [B4O10]8− electron-precise supertetrahedral unit, which forces silver atoms in this compound to play a sub-valent role (only half electron from each Ag atom is required) to fulfill the observed chemical formula.


The sub-valent behavior of silver is not exclusive to this compound (see for example earlier discussions in [3-5]) and has been extensively investigated by the corresponding author of the article commented here who identified the existence of local clustering of Ag atoms hosting electron pairs as a general “new bonding pattern” (see [2] and [6] and references therein).

In this comment, we provide a consistent interpretation of the chemical bonding network of Ag16B4O10 that contains analogies but also qualitative and quantitative differences with respect to the discussion presented in [2].

These differences are essential for the understanding of the structure and bonding in Ag16B4O10 and are far-reaching since they can be extended to other silver oxide crystals such as Ag7Pt2O7 [6].


The idea that metallic behavior is partially maintained in Ag16B4O10 is in part suggested in the discussion of [2], where it is pointed out that silver sub-valent oxides present Ag(I) sub-arrays resembling the structure of the pure fcc-silver metal.

However, in our view, it was not explicitly applied to interpret the sub-valent nature of the title compound.

To investigate the residual metallic bonding properties of Ag16B4O10, we propose to follow the electron counting associated with the metallic subarray from the pure silver metal to the borate compound.


Bearing this idea in mind, we provide an alternative interpretation of the bonding network of Ag16B4O10 supported by a topological analysis of the Electron Localization Function (ELF) in (i) the Ag metal fcc structure, (ii) the Ag deficient fcc sub-array after removal of the [B4O10] units, and (iii) the Ag16B4O10 crystal structure.

By performing these topological analyses, we will be able to check whether metallic silver features are maintained or not in the title compound, and to what extent.


Our results clearly reveal that [B4O10]8− behaves as an independent anion in agreement with the analysis in [2].

The equivalence of [B4O10]8− with the well-known P4O10 molecule was also justified resorting to the extended Zintl–Klemm concept in [7].

However, we found differences with respect to the interpretation of [2] in the peculiarities of the silver subarray.

To solve the electron-excess, electron pairs confined in four approximate tetrahedral voids of the Ag16 silver fcc-distorted unit cell are proposed in [2].

Although this proposal successfully explains the observed semiconducting behavior and the diamagnetic ordering of this compound, the limited analysis of ELF values at the attractor positions carried out in [2] is not enough to suggest that one electron pair is localized in each of these voids.


By combining VASP electronic structure calculations [8] within the DFT-GGA-PBE approach [9] with critic2 topological searches of critical points and basin electron density integrations [10], we perform an extensive ELF analysis in the Ag-related structures detailed in the three subsections below (see computational details in the ESI†).

Although other methods such as ELI-D better describe the dynamic electronic correlation and could provide further insight into electron poor metal systems [11], ELF analysis reliably accounts for the valence description of inorganic materials and metals [12-14], and is simple and illustrative enough to analyze the bonding properties of the title compound.


There are three main results in the ELF analysis of pure silver metal that are relevant to understand the structure of Ag16B4O10.

The first one concerns the localization of the ELF attractors associated with the valence electrons of fcc-silver at its room conditions equilibrium geometry (a = 4.146 Å).


Two sets of valence attractors become apparent.

The first set corresponds to d-electrons located not further away than 1.0 Å from the Ag nuclear positions (small black balls in Fig. 1 (left)) with an ELF value of h = 0.27.

The large number of them is a typical consequence of the crystal environment.


More interesting are s-valence attractors easily identified at the eight tetrahedral interstices (8c positions) of the fcc unit cell (yellow balls in Fig. 1 (left)).

They appear around 1.84 Å away from the four neighbour Ag nuclei with h = 0.20.

Electron density integrations inside each of these tetrahedral voids result in 0.53e.


This yields a total of 4.24 electrons in the whole unit cell, a value that is consistent with the valence oxidation state (Nv_ox) of +1 for metallic silver, where Nv_ox is defined in this context considering the number of valence electrons hosted in the lattice interstices.

This definition is different from the standard Nox definition which obviously yields a zero value for a pure metal.


The second ELF feature is that the topology is qualitatively the same when we expand the fcc silver unit cell to the volume corresponding to that of the fcc distorted silver sub-lattice hosting the B4O10 cluster (a = 4.251 Å).

The relevant eight ELF attractors still appear at the tetrahedral positions.

However, and in concordance with the generalized stress–redox correlation [15], the electron population inside the basins of the tetrahedral voids decreases at this expanded sub-lattice.

This is a consequence of the lower overlap between the electron density clouds of neighbouring atoms when their internuclear distance increases.

In fact, the electron density integration within the ELF tetrahedral-attractor basins estimates now 0.38e in each void.

Thus, using these valence electrons as before to evaluate the valence oxidation state of silver, we obtain Nv_ox(Ag) = +0.76 for the expanded fcc-Ag lattice at the same volume of the Ag16B4O10 compound.

Under this perspective, the expanded fcc-Ag already behaves as sub-valent.


Finally, we find that all the ELF attractors at the tetrahedral voids of the fcc-Ag structure (regardless of the unit cell parameter) belong to the same ELF domain.

This means that the ELF values at the maxima are linked through ELF first-order saddle points or basin interconnecting points (bips) forming a three dimensional (3D) chemical entity usually called a superbasin.


This is the typical ELF picture of metals [12].

The small difference between the ELF value at the attractor (att) and at the bip reflects a highly delocalized nature of the electrons in the 3D-superbasins.

Specifically, hatt = 0.20 and 0.19, and hbip = 0.17 and 0.16 in the room conditions and in the expanded fcc-Ag structures, respectively.


The concept of ELF superbasins is not new and has been employed many times to describe the merging of ELF basins belonging to the same atom (see for example [13] and [14]) or belonging to different atoms [16, 17].

Being a key concept in our alternative interpretation, it is useful to realize that these superbasins constitute 3D channels in the real space that account for the electrical conductivity of metals, as first noted by Silvi and Gatti [12].


In the fcc-Ag structure, the electrical circuit originating from the connection of the tetrahedral ELF attractors belonging to the valence superbasin is easily visualized in Fig. S1.†


Let us now consider the Ag16,4 host structure with the same lattice parameters as in Ag16B4O10.

Ag–Ag nearest neighbour distances range from 2.76 Å to 3.23 Å, with an average value of 3.05 Å, pretty close to the Ag–Ag shortest distance of 3.01 Å in the expanded fcc-lattice.

Breaking the symmetry of the fcc structure by replacing with a vacancy 4 out of the 20 tetrahedral Ag4 units of the unit cell (Z = 4) makes tetrahedral attractors not equivalent any more.

Up to five different types of ELF attractors identified with different colors in Fig. 2 (left) are found.

Among then, four have a multiplicity of 16, whereas the other (cyan) has a multiplicity of 4.


Despite the symmetry breaking, there are particular ELF features in the defective silver-vacant structure that resemble the description obtained for the expanded fcc-lattice.

For example, the ELF at the tetrahedral maxima keeps similar values around hatt = 0.20 and the average basin population (0.40e) is only a little bit higher that the 0.38e value found in the expanded pure metal.

However, basin electron populations oscillate now between 0.50e and 0.30e (see Table S1†).

This range of values illustrates how relevant is the position of the attractor relative to the vacancy.

Vacancies not only break the symmetry of the structure but also interrupt the connectivity between attractors.


The existence of vacancies reduces the number of available tetrahedral voids which are the only positions where ELF attractors tend to be located in these silver compounds.

We observe that this reduction is not balanced by an overall increase of the basin electron populations (see Table S1†).

Thus, a lower valence oxidation state of the Ag atoms is produced.

Using the basin electron populations, the average valence oxidation state in the defective Ag-vacant structure was found to be Nv_ox(Ag) = +0.42.

These values evidence an increase of the sub-valent characteristics of the defective silver-vacant structure.


In the Ag16B4O10 title compound, ELF analysis allows us to differentiate an ELF domain for the [B4O10]8− unit with equivalent results to those in [2].

Attractors between B atoms and the bridge or terminal oxygens are detected, as well as lone pair superbasins containing up to three ELF maxima (see Fig. S2†), reinforcing the structural similarity to the (valence) isoelectronic P4O10 molecule previously emphasized in [2] and [7].


When the ELF map of the host silver subarray is examined, the same attractors at the tetrahedral voids as in [2] are found with ELF values around 0.25 (black balls of Fig. 2).

We found analogies only up to this point with the results of [2].

We understand that to be conclusive with respect to the chemical bonding and the structure of the new silver oxide compound, its bonding network should be compared with that of pure silver and the deficient silver structure.


As observed in Fig. 2 (right), only two types of attractors at tetrahedral voids (instead of one as in fcc Ag and five in Ag16,4) are identified.

The presence of borate clusters destroys and empties some of the basins at the tetrahedral voids close to the oxygen positions.

Oxygen atoms act as electron sinks of the otherwise populated basins in the defective silver-vacant structure.

Such an electron transfer to the oxygen atoms reflects an ionic interaction between the two fragments in the Ag16B4O10 compound.

In particular, pink, orange, and green attractors with distances shorter than 2.75 Å to oxygen atoms are not seen in the silver borate compound, whereas only black and cyan attractors beyond 3 Å of any oxygen atom still are present in this compound.


Electron density integrations within each of the basins of these two attractors lead to 0.71e and 0.36e, respectively.

We notice that it is the Ag4 tetrahedra associated with the basins that are more populated where some Ag–Ag distances are especially shorter than those in pure silver metal.

Nevertheless, these values show that the number of electrons hosted in these tetrahedral voids is definitely lower than 2, questioning the presence of pairwise electrons.

In this way, the sub-valent nature of Ag16B4O10 might be considered to be inherited from the Ag16,4 host lattice since we obtain Nv_ox(Ag) = +0.40, almost coincident with the value found in the defective silver-vacant structure.


Besides the quantitative picture provided by the electron density integration, the most interesting result of this third ELF analysis comes from the identification of different independent valence superbasins associated with the silver subarray of Ag16B4O10.

These superbasins are equivalent and result from the bips connecting black–black (hbip ≈ 0.16) and black–cyan (hbip ≈ 0.19) attractors of values hatt ≈ 0.25 (black) and ≈ 0.23 (cyan).

The ELF profile displaying all the possible connections between black and cyan silver attractors is shown in Fig. S3†. Focusing just on one of these equivalent superbasins, we observe that all black attractors are connected with one cyan and two black attractors forming a 1D infinite coil-like chain of helical symmetry with ramifications ending at the cyan attractors.

There are four of these electrical circuits per unit cell hosting the electrons delocalized among the tetrahedral voids and running along the c axis of the crystal as shown in Fig. 3.

The existence of these unconnected superbasins is used below to explain the different behavior of pure silver metal and the borate compound with respect to metallicity.


We have seen that, in the fcc-silver metal, s-type ELF attractors only appear inside the tetrahedral voids.

This fact is noted both in the defective and silver borate compounds, and points towards a general behaviour that could be used to explain other silver sub-valent compounds.

Tetrahedral attractors allow the formation of chemical entities called superbasins which identify regions where the metallic bonding properties are maintained.

Interestingly, in the pure fcc-silver metal a 3D superbasin is spread out along the whole crystal structure, whereas in the borate compound, it splits up into four independent 1D superbasins per unit cell resembling infinite circuits of helical symmetry along the c axis.

Since electron delocalization is an inherent property of superbasins reflected by the close ELF-values at the maxima and the bips, this striking result provides an alternative explanation of why the silver borate compound is a small band gap semiconductor whereas the silver metal is an electrical conductor.

Macroscopically, we can understand that borate clusters play the role of crystal defects interrupting the electric conductivity along the three directions of the crystalline space.


The chemical entities associated with these superbasins allow us to discuss the sub-valent characteristics of silver-based compounds too.

For the fcc unit cell of silver under room conditions, our ELF analysis and electron density integration lead to a single Ag4+ superbasin.

In contrast, four equivalent Ag16,6.4+ entities appear in the unit cell of Ag16B4O10.


From these compositions, it is evident that the sub-valent characteristics of the silver (sub)array originated during the transition from the fcc-metal to the silver borate compound.

If we ideally understand the transformation as a three step continuous process involving (i) the expansion of the fcc-structure, (ii) vacancy creation and (iii) borate cluster insertion, we can identify two mechanisms that are able to reduce Ag atoms.


The first one is associated with an increase of the lattice parameter.

Crystal expansions decrease the electron density inside the tetrahedral voids (and therefore in the superbasins) allowing transfer of the valence electrons to the atomic cores.

Firstly, we obtained Nv_ox(Ag) = +1 and Nv_ox(Ag) = +0.76 when a = 4.146 Å and a = 4.251 Å, respectively.


Secondly, vacancy creation or cluster (borate) insertion disrupts the 3D superbasin connectivity, reducing the number of tetrahedral positions where silver electrons prefer to be accumulated.

As there is no other mechanism to reduce Ag atoms, an almost identical Nv_ox(Ag) value is obtained for the defective (+0.42) and the borate (+0.40) silver compounds.


In conclusion, we believe that these findings may throw light on the interesting experimental observations reported in [2] and might help to interpret the new chemistry exhibited by the silver oxides highlighted in [2] and references therein.

Our results emphasize the decisive role that residual metallicity plays in the existence of sub-valent compounds and could represent a promising strategy to be applied to other sub-valent silver oxides showing singular behaviour like the recently published Ag7Pt2O7 compound [6].

Claimed ‘synergistic interactions among the filled 4d10-shells’ strengthening silver inter-cluster bonding [2] may be accounted for by means of the proposed metallic superbasins rather than localized electron pairs.

Suggested silver-oxide dispersion interactions need other scalar fields such as reduced gradient density [18] or chemical pressure formalism [19] to reveal so-called non-covalent interactions.

We are currently working in that direction.


Ag16B4O10 features an exotic scheme of chemical bonding and extends the growing family of subvalent silver oxides.

These findings constitute a new general and intrinsic facet of the chemistry of silver, which has not been fully understood, yet, and definitely deserves to be analysed from different perspectives.

Against this background, we distinctly appreciate the efforts made by A. Lobato, Miguel Á. Salvadó, and J. Manuel Recio (LSR) in studying these phenomena at the example of the title compound [1].

While the computational results presented in the Comment article well comply with those published in our original paper [2], the interpretations follow different routes.

Whereas LSR focus on the analogy of pattern of the Electron Localization Function (ELF) in position space in the title compound with those found in elemental silver, we interpreted the electronic structure of Ag16B4O10, both in position and reciprocal space, also considering the interactions between cationic and anionic partial structures.


In structuring our discussion, we qualitatively rank the contributions of the different types of chemical bonding to the total cohesion energy of the title compound.

The markedly largest share results from conventional covalent and ionic bonding, followed by much weaker unconventional d10–d10 interactions [3].

The latter contribution is a crucial ingredient in the formation of the agglomerates of equally charged Ag+, which represent excisions of the ccp structure of elemental silver, even with respect to interatomic distances.

These unique structural characteristics are evolving in silver-rich oxides already without presence of residual 5s electrons, i.e. in electron precise Ag+ compounds.

Still, in an early review [3], the expectation was expressed that these features may enable existence of subvalent species: “The substructures thereby formed have empty s and p conduction bands, which can easily accommodate further electrons on reduction”.

This assumption has become true, as meanwhile several such subvalent silver oxides have been discovered [4–14], in addition to previously confirmed Ag2F and Ag3O [15,16].


The most puristic manifestations of bonding interactions between presumably closed shell d10 species are found in the distorted hcp structures of elemental zinc and cadmium.

Here, the set of commonly equal distances to the 12 nearest neighbours of an atom in an ideal hcp structure is conspicuously split into subsets of six substantially shorter in-plane and six longer out-of-plane separations.

Density functional (DFT) calculations do not give an unequivocal picture of this anisotropy as the results strongly depend on the functionals used [17].

By applying wavefunction based methods in the framework of the method of increments [18,19], the potential energy surface (PES) with respect to the lattice parameters was analysed [20,21].

While the Hartree–Fock PES is overall repulsive, a structured PES consistent with the observed structures can only be obtained if the filled d-shells are included in the treatment of the dynamical electron correlation.


Such calculations, which are computationally highly demanding, have not yet been carried out for subvalent silver compounds, and a quantitative explanation for the d10–d10 bonding in silver-rich compounds, which on its turn would be a crucial prerequisite for rationalizing existence of the subvalent silver oxides under discussion, is still elusive.

However, this issue is not the central subject of the comment by LSR, which focuses on the localisation of the residual 5s electrons.


In their analysis, LSR follow a particular line of reasoning, spotting the evolution of the electron localisation function (ELF) starting from the fcc elemental silver, moving via the silver sub-array as excised from Ag16B4O10 to the integral compound.

The results on the two structures consisting of silver alone are illustrative as such, and underline the merits of this approach for visualising and interpreting the bonding in metals and intermetallics in real space [22].


However, except for confirming that the excess electrons preferably accumulate in the tetrahedral voids (the sites of highest positive potential), the worth of these findings in understanding the bonding in Ag16B4O10 is but limited.

At least from the Ag–O bond lengths, which indicate presence of strong, regular bonds between the silver and borate fragment structures, it is obvious that it is not possible to separate these structural parts without causing severe perturbations.

Thus, the fragment silver structure cannot serve as a reliable reference for the same unit containing the embedded borate anion.

For this reason, we focus in our reply on the electronic properties of the entire configuration of Ag16B4O10.


The computational results obtained in both studies comply satisfactorily.

LSR do not state if they encountered a gapped band structure.

In our PBEsol-GGA-calculations [23] the gap is closed, although showing a minimum of the density of states (DOS) at the Fermi level.

Only by applying a hybrid functional [24], a gap is opened.

According to our experience, however, this issue is not crucial for the evaluation of the ELF.

Importantly, in both studies [1,2] the most populated ELF basins within the silver substructure are found at the same sites.

Yet, the exegeses of the computational results are appreciably divergent.


One must keep in mind that the ELF is not an “observable” in terms of quantum chemistry, since there is no Hamiltonian which operating at an appropriate wave function would reproduce such features, and correspondingly there is no experimental tool available that would allow to directly validate such results.

ELF does not prove the existence of electron pairs, bonds, or lone pairs. But there is a strong and appealing analogy between ELF attractors and basins, and classical Lewis structures, enabling to interpret the 3D-ELF in terms of a conceptual view of bonding.

The analogy gets weaker in intermetallic compounds or in compounds including transition elements.

Especially with late transition elements (like Ag) the values of the valence-(s,p)-attractors are much lower than the values of the d-attractors, see analysis by Kohout, Wagner and Grin [25].

Clearly, interpretation of ELF features falls within the scope of chemical concepts [26].


In our attempt to rationalise the at first glance puzzling experimental observations of an electron imprecise extended oxide to show semiconducting and diamagnetic responses, we assume that the eight excess electrons per formula unit will localise pairwise with opposite spin orientation.

In consent with LSR, we regard the (B4O10)8− anion as electron precise, consequently the excess electrons would be hosted by the silver partial structure classifying this compound as subvalent with respect to silver.


For identifying possible localisations of such electron pairs, we inspected the silver part of the structure for short Ag–Ag separations, which might indicate presence of e.g. 2c–2e bonds, and performed band structure calculations along with an ELF analysis.

As a result, we identified contracted tetrahedral subunits where 3 out of 6 Ag–Ag contacts along the edges of the tetrahedron are significantly shortened and were the ELF shows the highest value not associated to conventional bonds or lone pairs.

Quite satisfactorily, the number of such building blocks per unit cell exactly corresponds to the number of electron pairs to be accommodated.


Being aware that a partition of space, be it for structuring the electron density distribution or the ELF of a chemical entity, always suffers from arbitrariness, even if performed by applying a formal algorithm, we did not claim that the ELF contour drawn within the tetrahedra would comprise a complete pair of electrons with anti-parallel spins.

With this respect LSR misinterpreted our statement that the excess electron pairs were related to the contracted Ag4 units.


LSR go beyond our interpretation by considering regions with lower ELF values, which indicate that the maxima are linked through ELF first-order saddle points or basin interconnecting points (bips) bringing about extended chemical entities, i.e. superbasins.

At this point, we are discussing the spatial extension of the localized electron pair.

Unfortunately, there is no means of reliably validating which of the two views, assuming extended, metallic super basins or more localised electron pairs, would rather be appropriate in interpreting the bonding situation encountered.


In fact, we do not consider the extension of the localized electron pair as being pivotal to the physical properties and the stability of Ag16B4O10.

The analysis of the electronic structure in reciprocal space shows that bands with notable Ag-s-character are found far apart from the Fermi level at −6 eV. Just below the Fermi level, the band structure as given in Fig 7 of ref.

2 shows a low DOS in a range of 1.4 eV. This is due to some bands with high dispersion.

The electron density of these bands is mapped in Fig. 7b of ref.

2, illustrating the linear combination of Ag-d-orbitals and oxygen lone-pairs.

This can be understood as a continuation of d10–d10 interactions in the silver partial structure to the lone-pairs of the B4O10 8− anion via dispersion interaction, stabilizing the whole structure and leading to the semiconducting behaviour of the compound.


From the chemistry perspective of trying to define a generalizable “concept” for this particular type of bonding, one would compare with analogous compounds.

In Fig. 1 we present the silver sub-structure of Ag5GeO4 (ref.

10 and 11) as an example, giving appreciable support to a situation of local bonding.

Here again, the number of contracted polyhedra (octahedral Ag6 units) corresponds to the number of electron pairs to be accommodated, further the silver clusters and are not aggregated which rules out significant delocalisation.


Manifestations of subvalent silver in extended solids can no longer be regarded as exotic singularities since in recent years a sizeable number of oxides displaying such characteristics have been reported [4–16].

From the empirical point of view there are two particularly noteworthy issues: (1) subvalent silver has been encountered in combination with quite diverse cations, e.g. B, Si, Ge, Pb, Ni, Mn, Os or Pt, apparently without any systematic showing up, which nourishes the expectation that many more such candidates will be accessible; (2) opposite to common presumption, these oxides are strikingly stable in humid air and in particular against oxidising conditions, they even form by solid–state reactions applying elevated oxygen pressure.

In a heuristic approach, one may separate the total of specific bonding interactions responsible for the phenomena featured in conventional covalent and ionic contributions, and dispersive d10–d10 forces, superimposed by additional bonding provided by the excess electrons populating bonding 5s bands or local 5s/5p skeleton orbitals.

While the first two contributions appear to be sufficient for forming the extended subarrays of Ag+, providing low lying, empty 5s states [3], suited to accommodate excess electrons, the third component is subordinate and just gently modulates the silver substructures as indicated by global or local contractions.

The low bonding energy of the latter explains the wide phenomenological spread with respect to structure modulations and properties encountered.

Thus, one finds extended silver subarrays e.g. of Ag0.5+ in Ag2NiO2 [6,13,14] or Ag2F [15], or locally shrank tetrahedral or octahedral units embedded in the silver substructure [2,9–12].

Correspondingly, metallic and semi-conducting conduction properties, respectively, have been found.


The novel bonding motif is reminiscent of charge density waves in solid materials as all intermediate stages between delocalized to localized excess electrons would be covered.

However, for the localized scenario there appears to be a closer analogy to the “Polyhedral Skeletal Electron Pair Theory” describing the bonding in the so-called Wade–Mingos molecular clusters [27,28].

In this sense the excess electron pair would occupy the lowest bonding skeleton MO of the embedded silver clusters.


Mac Giolla and Ly begin by "warning against an over-reliance on cut-offs" (p. 2).

A cut-off is the level of evidence that a researcher considers sufficient to conclude that their data supports either hypothesis, or that the data is inconclusive.


Although Mac Giolla and Ly suggest that cut-offs "encourage categorical rather than continuous thinking" (p. 2), we believe it is possible to interpret Bayes factors as a continuous measure while simultaneously acknowledging whether the Bayes factor meets a pre-specified threshold of evidence to enable a categorical interpretation; for example, to state whether an intervention was successful or not, or that the evidence was inconclusive.

This judgment will influence subsequent behaviours, such as whether they encourage or discourage uptake of the intervention when disseminating their results.

Thus, although the use of cut-offs should not prevent the interpretation of the Bayes factor as a continuous measure, cut-offs can nonetheless guide the researcher's decision-making process.


Our purpose is not to discuss where the minimum cut-off should be set, which is a matter of ongoing debate in psychology (e.g. (Aczel et al., 2018; Benjamin et al., 2018; for an argument against cut-offs, see Morey, 2015).

Instead, we urge researchers to clearly specify when they are using cut-offs, and what those cut-offs are.

We believe that Warmelink et al. ( 2019) provide a reasonable and not over-reliant model of how cut-offs can be used: (I) introduce the Bayes factor as a continuous measure, (II) specify their interpretation of the Bayes factor cut-offs, (III) use labels that indicate the increasing strength of evidence, and (IV) always report the exact Bayes factor.

Violating these guidelines might suggest an over- reliance on cut-offs.


The term "nominal support" in Mac Giolla and Ly's commentary is not part of the standard Bayesian vernacular (see Box 1 for a short glossary), but appears to be equivalent to interpreting posterior odds.

We would like to suggest three things that deception researchers should consider before reporting posterior odds.


1) If posterior odds are reported, they should be reported alongside Bayes factors.

Van Ravenzwaaij and Wagenmakers (2019) recently argued that, as the posterior odds are the product of the Bayes factor and the prior odds, both should be reported and justified alongside posterior odds.


2) Mac Giolla and Ly suggest that the use of cut-offs is problematic for Bayes factors, and they then offer posterior odds as an alternative.

Readers might assume that posterior odds therefore avoid the use of cut-offs in a manner that Bayes factors do not.

However, this will again not be the case for researchers who either desire to or are required to categorically interpret their results.

Again, the use of such thresholds to interpret either Bayes factors or posterior odds does not diminish their value as continuous measures.

Researchers can correctly interpret a Bayes factor as a continuous measure and acknowledge whether it met a pre-specified minimum threshold of evidence in order to draw certain categorical conclusions.

The use of such thresholds can help with identifying the required sample size and limit researcher flexibility when interpreting results.


3) Further guidance is required on how deception researchers can specify scientifically- informed prior odds.

Mac Giolla and Ly specify 8% "for the sake of argument", but not all experimental hypotheses are equally (im)plausible, and the degree of scepticism will differ between researchers.

Meanwhile, reporting the Bayes factor allows readers to calculate their own posterior odds based on their prior odds.


Mac Giolla & Ly encourage deception researchers to make their studies more informative by reporting the posterior distribution.

A distinction exists in statistical inference between estimation (e.g. using posterior distributions) and hypothesis testing (e.g. using Bayes factors) (see Dienes, 2020).

To calculate posterior distributions, researchers use vague "default" prior models, which allow the posterior distribution to be primarily determined by the data.

However, vague prior models are not very useful for testing theories, as it is unlikely that the vague prior model represents a scientific theory of interest.

In contrast, Warmelink et al. ( 2019) used prior models that represented their scientific theory (i.e., the prior model is a 'model of H1', see Dienes, 2019, box 4) to calculate informed Bayes factors.


Researchers should exercise caution before automatically calculating posterior distributions using the model of H1 used to calculate informed Bayes factors.

Consider the Bayes factor reported on p. 7 of Warmelink et al (2019): BH(0, 7.05) = 3.64.

The model of H1 was specified using a half-normal distribution (indicated by the subscript H in the notation and the solid black line in Figure 1A and 1B).

The half-normal distribution represents several aspects of the theory: (I) only effects in one direction are considered plausible, these are represented as positive in the analysis, (II) smaller effect sizes are considered more plausible than larger effect sizes, and (III) effect sizes up to twice the approximate scale-of-effect are considered plausible (the approximate scale-of-effect was 7.05, based on the results of Warmelink, 2012).


To see why caution should be exercised before calculating a posterior distribution, consider Figure 1.

The red dotted line in Panel A shows the actual posterior distribution for the data in this example.

There is nothing particularly disconcerting about Panel A - the most plausible effect sizes according to the posterior distribution lie midway between the most plausible effect sizes according to the model of H1 (i.e., those close to zero), and the most plausible effect sizes according to the data (i.e., those close to the observed mean difference of 9.41).


Now consider Panel B. The red dotted line in Panel B shows the posterior distribution assuming that Warmelink et al. ( 2019) had observed the same magnitude of effect, but in the opposite direction predicted by theory.

The posterior distribution is clearly not an accurate estimation of effect sizes in light of the data.

The model of H1 assigned zero plausibility to results in the opposite direction to the prediction and consequently so does the posterior distribution.


In sum, posterior distributions can be informative for estimation purposes when a vague prior model has been used, but researchers should exercise caution before automatically calculating posterior distributions when reporting informed Bayes factors.


In this section we provide two further suggestions that we believe will improve the reporting of Bayesian analyses.


In line with Jeffrey's platitude that "the answer should depend on the question", the result of the Bayes factor depends on the specified models of H1.

APA guidelines state that one should therefore report how sensitive the inference is to the specification of the model of Figure 1.


Both panels present the model of H1 used by Warmelink et al (solid black line), the model of the data (dotted black line) and the posterior distribution (dashed red line).

Panel A shows the actual results.

Panel B shows the posterior distribution for counterfactual data with the same effect size, but in the opposite direction to what was predicted.


H1 whenever reporting Bayes factors (Appelbaum et al., 2018, p. 20) 1 .

This can be done by reporting robustness regions alongside Bayes factors (for examples see Lakens et al., 2020; Lin, et al., in press).

Robustness regions communicate the extent to which the inference drawn from a Bayes factor is dependent on the model of H1 used to calculate the Bayes factor.

If all plausible effect sizes that could be used to model H1 fall within the robustness region, the inference is robust; if plausible effect sizes that could be used to model H1 fall outside the robustness region, the inference is not robust.


Robustness regions are reported as RR[LL, UL] where LL corresponds to the smallest scale-of-effect and UL corresponds to the largest scale-of-effect that draw the same conclusion as the model of H1 used to calculate the Bayes factor assuming the specified minimum cut-off for evidence is constant.

Consider again the Bayes factor discussed above: BH(0, 7.05) = 3.64.

Warmelink et al. ( 2019) concluded that they had obtained moderate evidence for an effect, when specifying their model of H1 with a scale-of-effect of 7.05 and a minimum cut-off of B = 3.


To create robustness regions, we calculated a series of Bayes factors, each time manipulating the scale-of-effect to identify the smallest and largest scale-of-effect that still produces B > 3 (see DeBruine and Dienes, 2020, for R code that calculates Robustness Regions non-iteratively).


In this case, the robustness region is RR[4.44, 15.74].

Whether this can be considered robust or not depends on the scientific context.

If effects outside the robustness region are scientifically plausible, the inference is not robust.

If the robustness regions are judged to cover all theoretically plausible effect sizes, the inference is robust.

In this example, we conclude that the results are not robust.


Many deception researchers may argue that an effect smaller than 4.44 details would be too small to be of interest.

However, an effect larger than 15.74 can be considered plausible and interesting (e.g. Vrij et al., 2018, report a mean difference of 29 details between two interviewing conditions) 2 .


As Mac Giolla and Ly highlight, 19 of the 22 Bayes factors reported by Warmelink et al. ( 2019) were inconclusive.

Here we provide a simple method (with corresponding R script available at https://osf.io/34nb6/, based on original by Palfi & Dienes, 2019) that researchers can adopt to estimate the number of participants required in order to obtain a pre-specified level of evidence (originally outlined by Dienes, 2015).


As sample size increases, and the standard error reduces towards zero, Bayes factors are better able to distinguish between evidence for two competing hypotheses.

The process requires one to (i) estimate an approximate scale-of-effect if H1 were true (for a tutorial on how to estimate what your theory predicts, see Dienes, 2019), (ii) estimate the standard error for a given sample size (e.g., the standard error from a past study), and (iii) specify a minimum level of evidence one wishes to obtain.

The R script then simulates the standard error in light of increasing sample size, and calculates Bayes factors for each sample size.

It does this assuming the evidence supports H1 and H0 separately.

The script also outputs a graph to show the Bayes factor for H0 and H1 at each tested sample size.


Figure 2 shows the graph created assuming a researcher wanted to replicate the result from Warmelink et al. ( 2019), BH(0, 7.05) = 3.64, with a minimum cut-off of 3.

In this case, a replication would require 130 participants to obtain a Bayes factor of 3 in favour of H1 assuming the study supported H1, or 328 participants to obtain a Bayes factor of 3 in favour of H0 assuming the study supported H0 3 .

It is common that more participants are required to obtain evidence for the 2 To interpret robustness regions, it can be useful to consider the smallest effect of interest.

Dienes (2020) provides guidelines on how to do so.

If one is uncertain whether a finding is robust, however, one can continue to collect data until the robustness region extends below any plausible smallest effect size of interest.

3 Assumptions include that the effect size and sample error obtained by Warmelink et al (2019) are accurate.

point-null hypothesis than for the alternative hypothesis.

In this example, to ensure enough participants are recruited that the study could obtain sufficient evidence for either hypothesis, a replication study should look to recruit at least 328 participants.


We whole-heartedly agree with Mac Giolla and Ly when they encourage researchers to design more informative studies and we are grateful for their commentary that sought to educate deception researchers on matters of reporting and interpreting Bayesian analyses.

Here, we have sought to add a number of clarifications, and also specified two additional ways that the Bayesian analyses in Warmelink et al. ( 2019) could have been more informative.

If researchers adopt these recommendations, we believe they will be better able to assess the evidence provided by their research.


There is potential for confusion between the different uses of the word prior and posterior in Bayesian analyses.

Here we provide an overview.


Prior odds specify the relative degree of conviction a researcher has in two theories.

If a person believes each hypothesis is equally plausible, that person has prior odds of P(H1) / P(H0) = .50/.50 = 1.

Prior odds are updated by the level of evidence in the data (i.e., the Bayes factor) to derive the posterior odds: Prior odds x Bayes factor = Posterior odds.

The posterior odds specify the relative degree of conviction a researcher has in two hypotheses following data collection.


A prior model is a probability distribution that shows the plausibility of different population effect sizes (x-axis = possible effect sizes, y-axis = plausibility, see Figure 1).

Warmelink et al (2019) used informed Bayes factors, meaning that the prior model was specified to represent the theory being tested (Dienes, 2019, Box 4).

Their prior models were created by specifying the approximate effect size (i.e., the "scale-of-effect") that is plausible if the theory is correct.

Prior models can be updated with the model of the data to create the posterior distribution.

The posterior distribution demonstrates the plausibility of population effect sizes following data collection assuming the hypothesis is true.

Note that "distribution" and "model" are used interchangeably.


In order to avoid confusion, we adopt Dienes's (2020) vernacular, referring to the prior model as the "model of H1", to reflect the fact that the prior model represents predictions made by the theory.


Bayes factors quantify the evidence in support of the null (absence of an effect) or the alternative hypothesis (presence of an effect).

Based on commonly used cut-offs, Bayes factors between 1/3 and 3 are interpreted as evidentially weak, and one typically concludes there is an absence of evidence.

In this commentary on Warmelink, Subramanian, Tkacheva, and McLatchie (2019) [1], we discuss how a Bayesian report can be made more informative.

Firstly, this implies a departure from the labels provided by commonly used cut-offs when reporting Bayes factors.

Instead, we encourage researchers to report the value of the Bayes factors, or to convert these values into nominal support for the hypotheses.

Secondly, researchers can provide recommendations to design follow-up studies by examining the posterior distribution of the magnitude of the effect size.

Lastly, we show how individual Bayes factors can be evaluated in the context of large-scale meta-analyses.


A crucial distinction when interpreting the results of a study is the difference between the absence of an effect from the absence of evidence.

This distinction is typically ignored within frequentist statistics, where ‘non-significant’ p values (p > .05) are incorrectly interpreted as an absence of an effect [2].

For this reason alone, Warmelink, Subramanian, Tkacheva, and McLatchie (2019) [1] should be commended for reporting Bayes factors, which can quantify the evidence for the alternative (presence of an effect) over the null hypothesis (absence of an effect), and vice versa.

For instance, a Bayes factor BF10 = 8 implies that the observations are eight times more likely under the hypothesis that the effect is present than under the hypothesis that the effect is absent.

Similarly, BF10 = 0.125 (equivalently, BF01 = 1/BF10 = 8) implies that the data are 8 times more likely under the null compared to the alternative.


To aid interpretation, cut-offs (e.g., Jeffreys, 1961) [3] for Bayes factors have been proposed.

These cut-offs suggest that Bayes factors between 1 and 3 (1/3 and 1) provide only weak evidence for the alternative hypothesis relative to the null hypothesis (or for the null hypothesis relative to the alternative; e.g., Jarosz & Wiley, 2014) [4].

Hence, when confronted with evidentially weak Bayes factors, the common conclusion is that there is an absence of evidence [5], and the results are perceived as inconclusive.


The reporting of Bayes factors is still quite novel within deception research, though it appears to be gaining popularity [6][7].

We believe this is a good thing, but it does bring with it new challenges for reviewers and editors.

Using Warmelink, Subramanian, Tkacheva, and McLatchie (2019) [1] as a concrete example, we elaborate on the continuous nature of Bayes factors and how to interpret them.

We begin by warning against an over-reliance on cut-offs.

Next, we give some suggestions on how to make a Bayesian report more informative.


Risk of cut-offs and reporting alternatives
As noted above, cut-offs have been suggested to aid the interpretation of Bayes factors.

These cut-offs, however, should be used with caution.

Let us take the results of Warmelink et al. ( 2019) [1] as a concrete example.

In total, they report 22 Bayes factors.

Nineteen of these are between 1/3 and 3, and, therefore, evidentially weak according to the cut-offs they themselves adopt.


The remaining three analyses resulted in Bayes factors of BF10 = 3.58, BF10 = 3.64, and in the opposite direction BF01 = 4.76, which were interpreted to provide ‘moderate’ evidence.

This interpretation highlights the main risk of cut-offs: They encourage categorical rather than continuous thinking.

Yes, a Bayes factor of 3.58 provides more evidence than a Bayes factor of 2.8, but to say that one provides moderate evidence, whereas the other provides weak or inconclusive evidence, seems somewhat misguided [8].


The boundaries between the categories further fade away, when we convert a Bayes factor into posterior model probabilities.

A Bayes factor only measures the strength of the relative evidence provided by the data, whereas the posterior model probabilities quantify the nominal support for the hypotheses after data observation.

To calculate the posterior model probabilities, we combine the prior plausibility of the hypotheses with the Bayes factor using the following formula:


P(H1|data) = (BF10 * P(H1)) / (BF10 * P(H1) + P(H0))
P(H0|data) = 1 - P(H1|data)


For instance, if the plausibility of the presence and the absence of an effect are equal before data observation, then we set P(H1) = 0.5 and P(H0) = 0.5.

Observations leading to a Bayes factor of BF10 = 3.58 then yield a nominal support of 78%, P(H1|data) = 3.58 * 0.5/(3.58 * 0.5 + 0.5), for the hypothesis that there is an effect, leaving a posterior probability of 22%, P(H0|data) = 1 - 0.78, for the hypothesis that the effect is absent.

These probabilities can in turn be visualized using a simple pie chart known as a pizza plot [8].


Now look at Figure 1.

The dark portion of the pie chart corresponds to the 78% in support of H1.

Imagine that this chart was on the table in front of you.

You close your eyes, spin the chart, and at random place your finger down.

Now open your eyes.

How surprised would you be to find your finger on the white portion of the pie chart? Your degree of surprise is an intuitive measure of the strength of the nominal support for H1 [8].


Our guess is you would not be that surprised – there was after all a 22% probability of your finger landing on the white portion.

Similar computations show that the ‘evidentially weak’ Bayes factor of 2.8 would lead to a posterior probability of 74% in support of H1 and 26% for H0.

In terms of nominal support for the hypothesis, the distinction between ‘evidentially weak’ and ‘moderate’ Bayes factors in this example is negligible.


Hence, the categorizations are not that informative, and we strongly encourage researchers to discuss their results using the value of the Bayes factors or the posterior probabilities for each hypothesis instead.


Again, we can use Warmelink et al. ( 2019) [1] as a concrete example.

In their discussion, they conclude that Hypothesis 2 was partly supported and that Hypothesis 3 was not supported.

We worry that such language exaggerates the strength of evidence of the data.

Ultimately, researchers should be fair to the uncertainty implied by the Bayes factors.

This uncertainty is lost when we focus on labels such as ‘weak’ and ‘moderate’, and when the relative evidence is not converted to nominal support.


However, simply reporting that results are uncertain will likely leave many reviewers and readers wanting.

We see at least two options to help researchers provide more informative Bayesian reports: first, by focusing on the posterior distribution of the effect size and second, by relating Bayes factors to the broader research field by discussing prior model probabilities.

We will discuss each approach in turn.


Beyond Bayes factors: Examining the posterior distribution of the effect sizes
A Bayesian report can be made more informative by studying the posterior distribution of the effect size parameter.

This can provide us with an estimate of the magnitude of the effect, given that it exists, or provide us with information on how to design a follow-up study.

For instance, say we conduct a large sampled between-group study (N = 1,000) and find a standardized between-groups difference (Cohen’s d) = 0.1.

With a default Bayesian t-test, this would produce a Bayes factor around 1.5 in support of the null hypothesis, that there is no difference between the two groups.

Although this result helps little in determining whether the null or alternative is true, by examining the posterior distribution of the parameter, and the associated credible interval (the Bayesian equivalent to the frequentist confidence interval), one can still learn something from the data.

Specifically, that if there is an experimental effect, it is likely to be very small indeed (for a pedagogical online tool to visual this point see https://rpsychologist.com/d3/bayes/).

Such a conclusion is considerably more valuable than what one can glean by solely focusing on the Bayes factor.

Furthermore, with the estimate of the effect in hand, one can then efficiently plan follow-up studies.


The potential role of prior model probabilities
Another way to move beyond a simple reporting of Bayes factors is to use more informed prior model probabilities when calculating the nominal support for hypotheses.

In order to calculate the nominal support for hypotheses, some form of prior model probabilities must be specified.

Above, for lack of better choice, we had equal prior model probabilities: P(H0) = 0.5 and P(H1) = 0.5.

That is, before looking at the data, H0 and H1 were seen as equally likely.

More informed prior model probabilities can be derived from large-scale field-wide meta-analyses, or better still can be agreed upon by the research community.

In particular, if one has reason to be sceptical for the presence of an effect, the prior model probabilities can be weighted in favour of H0, thereby correcting for false positives.


In the context of deception detection, we have strong reason to be highly sceptical to most cues to deceit.

It seems that the research on deception cues, prior to 2003, is compatible with a world in which there are no reliable cues to deceit at all [9].

Hence, in the best case, Luke’s findings suggest that only a small fraction of deception studies lead to the detection of an actual effect – a genuine cue to deception.


For the sake of argument, let us say that only 8% of the published articles report an actual deception cue, leaving 92% of the published articles reporting a false-positive finding.

Thus, P(H1) = 0.08 and P(H0) = 0.92.

This interpretation of Luke’s finding can now be used as a context to evaluate the moderate BF10 = 3.58 as reported in Warmelink et al. ( 2019) [1] using the Equation (1).


A direct calculation shows that the nominal support for H1 of 8% is then increased to 24%, P(H1|data) = 3.58 * 0.08/(0.92 + 3.58 * 0.08), which leaves a posterior probability of 76% in support for H0.

Hence, there is indeed relative evidence for H1, because the Bayes factor is larger than 1, but based on the posterior model probabilities we cannot conclude that there is more nominal support for H1 than H0.

In other words, the relative evidence of BF10 = 3.58 is not enough to overcome the initial scepticism brought about by Luke’s findings.


By changing P(H0) = 0.5 and P(H1) = 0.5 to the more stringent P(H0) = 0.92 and P(H1) = 0.08, we now require BF10 to be larger than 11.5 (=0.92/0.08) before the posterior model probability for H1 exceeds that of H0.

In other words, we changed the threshold of preference – the strength of the evidence needs to be at least 11.5 before we begin to consider a found deception cue feasible.

Moreover, we now need a Bayes factor of 40.77, which can be verified using Equation (1), before we get nominal support of P(H1|data) = 0.78.


Our aim here was to demonstrate how prior model probabilities can let us draw important conclusions about the hypotheses and thereby provide more informative results than simply reporting Bayes factors.

We wish to stress however that our suggested base rate of 8% for H1 is only meant for illustrative purposes.

This matter should be discussed at the level of the research community.


Concluding remarks
We provided some suggestions on how to make a Bayesian report more informative.

Firstly, we warn against an over-reliance on the standard Bayes factor cut-offs.

Instead, we encourage researchers to report the value of the Bayes factors or convert it to the nominal support for the hypotheses.

Secondly, we encourage researchers to go beyond a simple reporting of Bayes factors.

This can be achieved, for example, by studying the posterior distribution of the effect size or by evaluating Bayes factors in the context of a large-scale meta-analysis using Equation (1).


Ultimately, however, there is no quick fix for making studies more informative after the fact.

Greater care must be taken before data collection to increase the chances of producing compelling results (for guidelines see Schönbrodt & Wagenmakers, 2018 [10] and see Ly et al., 2019 [11], on how to quantify replicability).

Luke (2019) [9] brought the consequences of ignoring this advice to light.


It seems that the extant research on deception cues, prior to 2003, is compatible with a world in which there are no reliable cues to deceit at all.

Individual small-sample studies, whose effects may well be nothing more than sampling variation, were over-interpreted, published, and, in worst case scenarios, used for policy recommendations.

It took some 60 years for this issue to come to light in DePaulo et al. ’s (2003) meta-analysis [12]. And almost another 20 for us to understand the true consequences of this practice in Luke’s re-analysis of the data.

One can rightly wonder how much quicker we would have come to this conclusion if studies from the outset had been designed to provide more informative and compelling results.


Since DePaulo et al. ’s meta-analysis, deception researchers’ focus on traditional cues to deceit has waned.

Instead, there is a new focus on interviewers actively eliciting cues to deceit, (Vrij & Granhag, 2012) [13], of which Warmelink et al. ( 2019) [1] is one example.

Rather than passively observing truth tellers and liars, these methods consist of asking questions in a strategic manner in order to increase differences in the statements or behaviours between these groups.

This new approach may be a genuine path to a viable method of deception detection, or it may be another dead end. If we do not start designing our studies to produce more informative results, we may need to wait another 80 years to find out.


Acknowledgements
We would like to thank Timothy J. Luke and Eric-Jan Wagenmakers for their helpful comments on earlier versions of this paper.


Conflicts of interest
All authors declare no conflict of interest.


Bayes factors provide a continuous measure of evidence for one hypothesis (e.g., the null, H0) relative to another (e.g., the alternative, H1) [1].

Warmelink et al. ( 2019) reported Bayes factors alongside p-values to draw inferences about whether the order of expected versus unexpected questions influenced the amount of details interviewees provided during an interview [2].

Mac Giolla & Ly (2019) provided several recommendations to improve the reporting of Bayesian analyses and used Warmelink et al. ( 2019) as a concrete example [3].

These included (I) not to over-rely on cut-offs when interpreting Bayes factors; (II) to rely less on Bayes factors, and switch to ‘nominal support’; and (III) to report the posterior distribution.

This paper elaborates on their recommendations and provides two further suggestions for improvement.

First, we recommend deception researchers report Robustness Regions to demonstrate the sensitivity of their conclusions to the model of H1 used.

Second, we demonstrate a method that deception researchers can use to estimate, a priori, the sample size likely to be required to provide conclusive evidence.

Mac Giolla and Ly’s (2019) commentary on Warmelink, Subramanian, Tkacheva, and McLatchie (2019) discussed how deception researchers could improve their use and reporting of Bayes factors, which quantify relative evidence for competing hypotheses [3].

Mac Giolla and Ly’s advice included: 1.

Not to over-rely on cut-offs when interpreting Bayes factors.

To report alternatives to Bayes factors, including ‘nominal support’.

3. To report the posterior distribution.

We broadly agree, but in the first part of our reply we add a number of clarifications that we believe deception researchers should consider before following these suggestions.

We also add two further suggestions: 1. Report Robustness Regions to demonstrate the sensitivity of conclusions to the different models of H1 that could be used to calculate Bayes factors.

2. Design informative studies by estimating the sample size required to obtain a minimum level of evidence.

In this section, we comment on the three suggestions made by Mac Giolla and Ly (2019) on the reporting of Bayesian analyses.


Do not over-rely on cut-offs Mac Giolla and Ly begin by ‘warning against an over-reliance on cut-offs’ (p. 2) [3].

A cut-off is the level of evidence that a researcher considers sufficient to conclude that their data support either hypothesis, or that the data are inconclusive.

Although Mac Giolla and Ly suggest that cut-offs ‘encourage categorical rather than continuous thinking’ (p. 2), we believe it is possible to interpret Bayes factors as a continuous measure while simultaneously acknowledging whether the Bayes factor meets a pre-specified threshold of evidence to enable a categorical interpretation; for example, to state whether an intervention was successful or not, or whether the evidence was inconclusive.

This judgement will influence subsequent behaviours, such as whether the researchers encourage or discourage uptake of the intervention when disseminating the results.

Thus, although the use of cut-offs should not prevent the interpretation of the Bayes factor as a continuous measure, cut-offs can nonetheless guide the researcher’s decision-making process.

Our purpose is not to discuss where the minimum cut-off should be set, which is a matter of ongoing debate in psychology (e.g., Aczel et al., 2018; Benjamin et al., 2018; for an argument against cut-offs, see Morey, 2015) [4][5][6].

Instead, we urge researchers to clearly specify when they are using cut-offs, and what those cut-offs are.

We believe that Warmelink et al. ( 2019) provide a reasonable and not over-reliant model of how cut-offs can be used: (I) introduce the Bayes factor as a continuous measure, (II) specify the interpretation of the Bayes factor cut-offs, (III) use labels that indicate the increasing strength of evidence, and (IV) always report the exact Bayes factor [2].

Violating these guidelines might suggest an over-reliance on cut-offs.


Researchers should report ‘nominal support’ (posterior odds) The term ‘nominal support’ in Mac Giolla and Ly’s commentary is not part of the standard Bayesian vernacular (see Box 1 for a short glossary), but appears to be equivalent to interpreting posterior odds [3].

We would like to suggest three things that deception researchers should consider before reporting posterior odds.

1. If posterior odds are reported, they should be reported alongside Bayes factors.

Van Ravenzwaaij and Wagenmakers (2019) recently argued that, as the posterior odds are the product of the Bayes factor and the prior odds, both should be reported and justified alongside posterior odds [7].

2. Mac Giolla and Ly suggest that the use of cut-offs is problematic for Bayes factors, and they then offer posterior odds as an alternative [3].

Therefore, readers might assume that posterior odds avoid the use of cut-offs in a manner that Bayes factors do not.

However, both posterior odds and Bayes factors can be interpreted using cut-offs by researchers who either want to or are required to categorically interpret their results.

Again, the use of such cut-offs to interpret either Bayes factors or posterior odds does not diminish their value as continuous measures.

Researchers can correctly interpret a Bayes factor or posterior odds as a continuous measure and acknowledge whether it met a pre-specified minimum threshold of evidence in order to draw certain categorical conclusions.

The use of such thresholds can help with identifying the required sample size and limit researcher flexibility when interpreting results.

Further guidance is required on how deception researchers can specify scientifically informed prior odds.

Mac Giolla and Ly specify 8% ‘for the sake of argument’, but not all experimental hypotheses are equally (im)plausible, and the degree of scepticism will differ between researchers [3].

Meanwhile, reporting the Bayes factor allows readers to calculate their own posterior odds based on their own prior odds.

Report the posterior distribution Mac Giolla & Ly encourage deception researchers to make their studies more informative by reporting the posterior distribution [3].

A distinction exists in statistical inference between estimation (e.g., using posterior distributions) and hypothesis testing (e.g., using Bayes factors) (see Dienes, 2020) [8].

To calculate posterior distributions, researchers use vague ‘default’ prior models, which allow the posterior distribution to be primarily determined by the data.

However, vague prior models are not very useful for testing theories, as the vague prior model might not represent the scientific theory of interest.

In contrast, Warmelink et al. ( 2019) used prior models that represented their scientific theory (i.e., the prior model is a ‘model of H1’, see Dienes, 2019) to calculate informed Bayes factors [2][9].

Researchers should exercise caution before automatically calculating posterior distributions using the model of H1 used to calculate informed Bayes factors.

Consider the Bayes factor reported on p. 7 of Warmelink et al. ( 2019): BH(0, 7.05) = 3.64 [2].

The model of H1 was specified using a half-normal distribution (indicated by the subscript H in the notation and depicted by the solid black line in Figure 1A,B).

The half-normal distribution represents several aspects of the theory: (I) only effects in one direction are considered plausible, and these are represented as positive in the analysis; (II) smaller effect sizes are considered more plausible than larger effect sizes; and (III) effect sizes up to twice the approximate scale-of-effect are considered plausible (the approximate scale-of-effect was 7.05, based on the results of Warmelink, 2012) [10].

To see why caution should be exercised before calculating a posterior distribution, consider Figure 1.

The red dotted line in Panel A shows the actual posterior distribution for the data in this example.

There is nothing particularly disconcerting about Panel A – the most plausible effect sizes according to the posterior distribution lie midway between the most plausible effect sizes according to the model of H1 (i.e., smaller effect sizes closer to zero), and the most plausible effect sizes according to the data (i.e., those close to the observed mean difference of 9.41).

Now consider Panel B. The red dotted line in Panel B shows the posterior distribution assuming that Warmelink et al. ( 2019) had observed the same magnitude of effect, but in the opposite direction predicted by theory [2].

The posterior distribution is clearly not an accurate estimation of effect sizes in light of the data.

The model of H1 assigned zero plausibility to results in the opposite direction to the prediction and consequently so does the posterior distribution.

In sum, posterior distributions can be informative for estimation purposes when a vague prior model has been used, but researchers should exercise caution before automatically calculating posterior distributions when reporting informed Bayes factors.


Additional suggestions In this section, we provide two further suggestions that we believe will improve the reporting of Bayes factors.

Report Robustness Regions to demonstrate the robustness of conclusions to models of H1 In line with Jeffrey’s platitude that ‘the answer should depend on the question’, the result of the Bayes factor depends on the specified models of H1.

APA guidelines state that one should therefore report how sensitive the inference is to the specification of the model of H1 whenever reporting Bayes factors (Appelbaum et al., 2020, p. 20) [4].

This can be done by reporting robustness regions alongside Bayes factors (e.g., Lin, McLatchie & Linkenauger (2020) and Palfi, Parris, McLatchie, Kekecs & Dienes (In Press)) [11][12].

Robustness regions are reported as RR[LL, UL] where LL corresponds to the smallest scale-of-effect and UL corresponds to the largest scale-of-effect that lead to the same conclusion as the model of H1 used to calculate the Bayes factor, assuming the minimum threshold of evidence is constant.

Consider again the Bayes factor discussed above: BH(0, 7.05) = 3.64 [2].

Warmelink et al. ( 2019) concluded that they had obtained moderate evidence for an effect, when specifying their model of H1 with a scale-of-effect of 7.05 and a minimum threshold of B = 3 [2].

To create robustness regions, we calculated a series of Bayes factors, each time manipulating the scale-of-effect to identify the smallest and largest scale-of-effect that still produced B > 3 (see DeBruine & Dienes, 2020, for R code that calculates Robustness Regions non-iteratively) [6].

In this case, the robustness region is RR[4.44, 15.74].

Whether this can be considered robust or not depends on the scientific context.

If effects outside the robustness region are scientifically plausible, the inference is not robust.

If the robustness regions are judged to cover all theoretically plausible effect sizes, the inference is robust.

In this example, we conclude that the result is not robust.

Many deception researchers may argue that an effect smaller than 4.44 details would be too small to be of interest.

However, an effect larger than 15.74 can be considered plausible and interesting (e.g., Vrij, Leal, Jupe, & Harvey, 2018, report a mean difference of 29 details between two interviewing conditions) [13].

Estimate the sample size required to obtain conclusive evidence As Mac Giolla and Ly highlight, 19 of the 22 Bayes factors reported by Warmelink et al. ( 2019) were inconclusive [3][2].

Here, we provide a simple method (with corresponding R script available at https://osf.io/34nb6/, based on original by Palfi & Dienes, 2019) that researchers can adopt to estimate the number of participants required in order to obtain a pre-specified level of evidence (originally outlined by Dienes, 2015) [14][15].

As sample size increases, and the standard error reduces towards zero, Bayes factors are better able to distinguish between evidence for two competing hypotheses.

The process requires one to (1) estimate an approximate scale-of-effect if H1 were true (for a tutorial on how to estimate what your theory predicts, see Dienes, 2019) [9], (2) estimate the standard error for a given sample size (e.g., the standard error from a past study), and (3) specify a minimum level of evidence one wishes to obtain.

The R script then simulates the standard error in light of increasing sample size and calculates Bayes factors for each sample size.

It does this assuming the evidence supports H1 and H0 separately.

The script also outputs a graph to show the Bayes factor for H0 and H1 at each tested sample size.

Figure 2 shows the graph created assuming a researcher wanted to replicate the result from Warmelink et al. ( 2019), BH(0, 7.05) = 3.64, with a minimum cut-off of 3 [2].

In this case, a replication would require 130 participants to obtain a Bayes factor of 3 in favour of H1 assuming the study supported H1, or 328 participants to obtain a Bayes factor of 3 in favour of H0 assuming the study supported H0.

It is common that more participants are required to obtain evidence for the point-null hypothesis than for the alternative hypothesis.

In this example, to ensure enough participants are recruited that the study could obtain sufficient evidence for either hypothesis, a replication study should look to recruit at least 328 participants.


Conclusion We whole-heartedly agree with Mac Giolla and Ly when they encourage researchers to design more informative studies, and we are grateful for their commentary that sought to educate deception researchers on matters of reporting and interpreting Bayesian analyses [3].

Here, we have sought to add a number of clarifications and also specified two additional ways that the Bayesian analyses in Warmelink et al. ( 2019) could have been more informative [2].

If researchers adopt these recommendations, we believe they will be better able to assess the evidence provided by their research.


Gastroenteropancreatic neuroendocrine neoplasms (NENs) are rare tumors defined by the expression of specific diagnostic biomarkers [1-3].

Cell differentiation is a major prognostic marker of neuroendocrine neoplasms [4, 5].

Indeed, regardless of the stage or the location of the primary tumor, it has been highlighted [3] that well-differentiated lesions have a better prognosis than poorly differentiated ones [4, 6-8].

In 2010, the World Health Organization (WHO) classification of neuroendocrine neoplasms was reviewed and validated the crucial role of the proliferative rate [9-14].

The WHO Classification 2010 defined three groups of tumors, according to the combination of the morphological characteristics and the mitotic index as well as the Ki-67 index [15]: grades 1 and 2 corresponded to well-differentiated neuroendocrine tumors (NETs), whereas grade 3 corresponded to poorly differentiated lesions called neuroendocrine carcinomas (NECs) (Table 1).


Initially, it was assumed that no NET with a mitotic or a Ki-67 index above 20% could exist.


The prognostic assessment of gastroenteropancreatic NENs has improved significantly since the introduction of the European Neuroendocrine Tumor Society (ENETS) and the WHO 2010 staging and grading systems [21].

The unifying grading scheme is based solely on the proliferative rate (mitoses and the Ki-67 index), permitting classification of all gastroenteropancreatic NENs as follows: low-grade, or G-1 (mitoses #2/10 HPF and Ki-67 index #2%); intermediate- grade, or G-2 (mitoses 2-20/10 HPF or Ki-67 index 3%-20%); and high-grade, or G-3 (mitoses .20/HPF or Ki-67 index .20%) (Table 1).

The WHO recommends that mitoses and the Ki-67 index should be counted in 50 high-power fields and 500-2,000 cells in the highest labeling regions, respectively.


According to this system, a distinction can be made between well-differentiated gastroenteropancreatic NETs (low-grade G-1 and intermediate-grade G-2), the most frequent form, and poorly differentiated gastroenteropancreatic NECs (high- grade G-3) that include small- and large-cell NECs.

Microscop- ically, well-differentiated NETs are composed of monotonous regular cells presenting round or oval nuclei with "salt and pepper" chromatin and eosinophilic granular cytoplasm.

The tumor nests are arranged in trabecular, insular, or sheet-like patterns (Fig. 1A, 1B).

In some cases, important nuclear pleo- morphisms may be encountered, but are not associated with tumor aggressiveness (Fig. 1B).

Poorly differentiated NECs show highly proliferative atypical cells that are organized in diffuse sheets with little nesting (Fig. 1C).

Apoptotic bodies and necrosis are usually observed, whereas the Ki-67 index is usually .50% (Fig. 1D).


However, it has now become clear that the G-3 group is heterogeneous, comprising, in addition to poorly differenti- ated NECs, tumors that retain the cytoarchitectural features of well-differentiated gastroenteropancreatic NETs, despite a mitoses/Ki-67 count above 20% (Fig. 1E, 1F).

NETs of the G-3 group seemed not to be as aggressive as classic poorly differentiated NECs, but showed a worse outcome than the G-2 group [18, 22].

The prognosis is considered as intermediate between NET G-2 and NEC.Thus, it recently appeared of clinical interest that pathologists distinguished them both from NECs by their well-differentiated pattern and their lower mitosis/ Ki-67 count, which usually ranges from 20% to 50%, but also from G-1/2 NET.

In some cases, the morphological distinction with NECs (i.e., the distinction between well and poorly dif- ferentiated NENs) remained problematic in case of important tumoral heterogeneity, with limited areas showing less dif- ferentiated patterns and/or necrosis.


Recently, it has been shown that sporadic NETs and NECs occurring in the pancreas are genetically different [23].

Whereas inactivating mutations in DAXX and ATRX were exclusively found in NETs, abnormal immunolabeling for p53 and the Rb/p16 pathways was, conversely, detected in virtually all NECs (Table 2).

Some of these molecular alterations may help pathologists to separate morphologically NEC from NET, in addition to morphological cellular characteristics.


An accurate pathological assessment of the proliferation index appears critical to rigorously identify highly proliferative NET G-3.

Determination of the Ki-67 index has become a critical element in NET disease management.

Discussions remain regarding the lack of uniformity and re- producibility between the different methods used, such as "eyeballing," manual counting, or digital image analysis.

De- spite good agreement for tumors showing a very low (,1%) or high (.70%) index, because of the marked interobserver and intraobserver variability, the eyeballing estimation did not appear to be reliable.

This was particularly observed around the cutoffs for grading a tumor as G-2 (2%-5% range) or G-3 (15% to .20%) [24].

Compared with the image analysis Well-Differentiated Grade 3 NET method, eye observation/counting tended to overestimate the Ki-67 index [24-26].

Thus, in practice, the "acceptable gold standard" for determining the Ki-67 index remained the manual counting of a sufficient number of cells (.500), through the microscope or more confidently on a printed version of a captured screenshot as demonstrated by some authors [27].

Besides the problems of quantification, other factors, such as the fixative process, the staining technique, or the antibody used, may potentially affect the reproducibility of Ki-67 values.

Underestimated grading has also been demonstrated when histological analysis is performed by endoscopic ultrasound- guided fine-needle aspiration [28, 29].


Most well-differentiated NETs (G-1 and G-2) are recognizable without difficulty by pathologists.

It is very unusual that an expert pathologist does not confirm an initial diagnosis.

Based on the 2010 WHO classification, morphology has, for the most part, unfortunately been disregarded.

The lackof NET G-3 in the current classification could lead to classifying a tumor as a poorly differentiated G-3, mainly because of a high prolifer- ative index, instead of a well-differentiated G-3 [30].

Over- diagnosis can be observed for tumors with a Ki-67 index .20%, showing a marked nuclear pleomorphism or a crushed artifact, particularly on biopsies.

However, it is clear that in rare cases, a gray zone persists for pathologists to distinguish NECs from highly proliferative NET G-3.

Some authors underline the need to revise criteria concerning G-3 tumors.

Revision should put emphasis on differentiation, modification of the Ki-67 cutoff value (50% instead of 20%), and integration of molecular characteristics that could allow a better distinction between these two types of proliferative tumors.

Mixed tumors and metastasis, from the lung in particular, also need to be dis- tinguished from gastroenteropancreatic NEC.

This remains a problem because these tumors are often diagnosed at a multimetastatic stage, and the diagnosis made sometimes requires the use of a microbiopsy performed under endoscopic ultrasound.

Furthermore, in contrast to NET, TTF1 and CDX2 are not reliable markers to establish the site of origin of a metastatic NEC.

Differential diagnosis with an acinar cell carcinoma is even more problematic and requires the use of a panel of immunohistochemical stains, including both neuro- endocrine and acinar markers (trypsin, chymotrypsin, and Bcl10) [31].


This distinction is particularly difficult for mixed acinar-neuroendocrine carcinomas [32].


The incidence of gastroenteropancreatic NENs is rising, prob- ably because of the increasing quality and quantity of com- puted tomography imaging [1, 33].

Epidemiologic data on NEN G-3 (NEC 1 NET G-3) are scarce.

Some studies have only reported NECs with a small-cell morphology [34, 35], whereas in contrast with lung NECs, large-cell NECs are more common than small-cell NECs in the gastroenteropancreatic area [36, 37].

Among gastroenteropancreatic NENs, 7%-21% are poorly differentiated (NEC) [1, 17, 35], but no data on G-3 tumors were available in these studies.


Some studies reported a frequency of 9%-16% of gastroenteropancreatic NEN G-3 [17, 38, 39].

Studies, which have reported both characteristics, have al- ways found more NET G-3 than NEC: 9.1% versus 6.7% in the Austrian study [17] and 13.5% versus 9% in the Pronet study [39].

Therefore, we can speculate that the NET G-3 incidence is probably underestimated.

For instance, we can expect some NET G-3 in the NORDIC NEC study [36], especially from the pancreas, because (a) no pathological review was performed in this population of 305 patients selected on Ki-67 .20% to confirm whether all tumors were poorly differenti- ated or not and (b) patients with pancreatic tumors had higher rates of positive somatostatin receptor scintigraphy (SRS) (46%), lower rates of Ki-67 (70% with Ki-67 ,55%), and longer overall survival.

These factors are more frequent in NET G-3 than in NEC.


Very few data comparing NET G-3 and NEC are available.

This was the goal of only four recent studies [16, 18, 37, 39].

Therefore, it is difficult to get an idea of the true distribution between NET G-3 and NEC, but NET G-3 would represent approximately one third of NEN G-3.The only prospective data come from the Pronet study, a 1-year prospective, epidemi- ologic study to assess the characteristics of newly diagnosed NEN, including a review of their diagnostic pathology.

This study reported 778 patients with gastroenteropancreatic NEN, including 104 (13.5%) NEN G-3.

From the 104 NEN G-3, the proportions of NEC, NET G-3, and mixed adenoneuroendocrine carcinoma were 69% (n 5 72), 20% (n 5 21), and 11% (n 5 11), respectively [39].

Another prospective national cohort, which focused on NEC (NET G-3 excluded) with a pathological review by the French national referent pathological network (TENpath), gathered interesting data on characteristics, prognosis, and treatments of NEC without comparisons with NET G-3 [40].


The paucity of available data on presentation of NET G-3 warrants particular caution regarding their distribution (Table 3)


In nonmetastatic tumors, the ENETS and U.S. guidelines recommend a surgical resection for localized NEN, irrespective of tumor grading [42-44].

Based on the treatment paradigm for limited-stage small-cell lung cancer, neoadjuvant chemo- therapy and radiation can be considered in many gastro- enteropancreatic NEC patients, particularly when surgical resection is difficult [43, 44].

No data exist on the efficacy of such an approach in NET G-3 patients.

As for well-differentiated NET G-1/2, surgery is recommended as the first option in nonmetastatic NET G-3 patients [43, 44].


In pancreatic tumors, the standard treatment is surgical resection with a 5% mortality-and a 50% morbidity-rate according to the type of surgery [45, 46].The ENETS guidelines suggest that a "wait-and-see" approach might be preferred for asymptomatic patients in case of G-1, ,2 cm tumors.

Con- sidering the lack of data, a similar approach cannot be rec- ommended in NET G-3.


Ingastrictumors, a cancersurgery iswarranted.

Antrectomy, indicated in selected type 1 gastric NET patients to avoid gastrin stimulation, is not recommended in NET G-3 [47, 48].

In colonic and rectal tumors, NET G-3 should be treated with appropriate cancer surgery combined with lymph node re- section [49].


Surgical management of metastatic NET is controversial.

The overall prognosis of these tumors is mainly based on growth of liver metastasis, cell differentiation, and Ki-67 index.

In NET G-1/2, liver surgery can be a valid option.

In pancreatic NET G-3, a similar overall survival of patients with NET G-2 and G-3 has been recently reported (67.8 and 54.1 months, respectively), significantly higher than in NEC (11 months) [18].

Therefore, from a surgical point of view, metastatic NET G-3 should be considered as NET G-2.


In well-differentiated NET G-1/2, no data favor the adminis- tration of adjuvant chemotherapy.

In NEC, there are no studies examining adjuvant postoperative treatments.

Nevertheless, their aggressive behavior and high rate of recurrence have led experts to consider adjuvant treatment, using platinum-based combinations as in small-cell lung tumors [44].

In NET G-3, no data suggest a benefit of such an adjuvant therapy.

In addition,

Based on experts considerations and from the NORDIC NEC study (Ki-67 with a cutoff for response found at 55%), we proposed a Ki-67 cutoff between 50% and 60% as the minimum level that should be considered satisfactory to propose plati- num based chemotherapy (Fig. 2).

Therefore, it appears reasonable to manage NET G-3 and NET G-1/2 patients similarly, without adjuvant therapy although there is a higher risk of recurrence [37].


In pancreatic well-differentiated NET, chemotherapy is the benchmark if the main treatment goal is the reduction of the tumor mass, particularly if it would allow secondary surgery.

A long time ago, streptozotocin plus doxorubicin was shown to be superior to streptozotocin plus 5-fluorouracil, both in terms of tumor regression rate (69% versus 45%; p 5 .005) and length of time to tumor progression (median: 20 months versus 6.9 months; p 5 .001) [52].

However, in view of renal toxicity of streptozotocin and cardiac toxicity of doxorubicin, different chemotherapy regimens have been investigated and may be recommended as a first-line option.


Altimari et al. identified a 50% efficacy of dacarbazine in patients with nonresectable intra-abdominal NET [53].

In line with this report, Bajetta et al. showed a 30% response rate with a doublet of dacarbazine and 5-fluorouracil (LV5FU2-dacarbazine) in a cohort of 30 patients [54].

Strosberg et al. identified a major response rate (radiologic objective response: 70%; stable disease: 20%) using capecitabine from days 1 to 14 in addition to temozolomide from day 10 to 14 [12].

In the BETTER trial, chemotherapy-naïve pancreatic NET G-1/2 patients were exposed to a bevacizumab-5-fluorouracil-streptozotocin asso- ciation and experienced a significant disease control rate (56% of partial responses and 44% of stabilizations) [55].

Irinotecan in addition to 5-fluorouracil (FOLFIRI) has also been considered as an option in second-line pancreatic NET.

This regimen was associated with an 80% disease control rate but without objective tumor response [56].

In pancreatic NEC, the association of cisplatin to etoposide appeared as the first chemotherapy option [7, 57].

Recently, Nakano et al. presented another platinum-based regimen with irinotecan and cisplatin and experienced a 50% overall response rate [58].

Similarly to pancreatic NET, FOLFIRI regimen has been evaluated as showing 37% overall response rate in second-line [59].

In a In nonpancreatic NET, no randomized study identi- fied a standard of care.


Interferon is mainly proposed in cases of persistent secretory syndrome despite the somato- statin analogs [61, 62].

Interferon did not provide any significant benefit in terms of progression-free survival versus 5-fluorouracil plus streptozotocin (14.1 versus 5.5 months; hazard ratio 5 0.75 [0.41-1.36]) [62].

Interferon-a (3 million IU, three times per week) was used, whereas other forms seem better-tolerated and possibly as effective [63].

Considering the lack of validated chemotherapy, some experts in line with well-differentiated NET recommended dacarbazine [53] with or without 5-fluorouracil (LV5FU2- dacarbazine) [64] or temozolomide [65].

An alternative could be the combination of capecitabine and temozolomide [12].

Cassier et al., using a platinum-based chemotherapy (gemcitabine and oxaliplatin regimen), suggested some efficacy in pretreated non-pancreatic-NET G-1/2 patients with an 84% overall response rate [66].

Nevertheless, physicians should keep in mind that the response rates to alkylant agents are very low in nonpancreatic NET, possi- bly due to a strong expression of O(6)-methylguanine DNA methyltransferase [67, 68].

The best results of a chemo- therapy regimen were obtained in a nonrandomized phase II study of naive patients with nonpancreatic NET [69].

In patients receiving a capecitabine-bevacizumab combina- tion, the response rate was reported as 18%, the overall disease control rate was 88%, and the progression-free survival was approximately 23 months [69].

These results require confirmation.

In nonpancreatic NET G-3, no chemo- therapy regimen should be considered as a standard of first-line care, considering the very small amount of data available.


In nonpancreatic NET G-3, no chemotherapy regimen should be considered as a standard of first-line care, considering the very small amount of data available.


In pancreatic NET G-1/2, two randomized phase III studies demonstrated the efficacy of a targeted therapy, either an mam- malian target of rapamycin (mTOR) inhibitor [9] (everolimus) or a tyrosine kinase inhibitor [14] (sunitinib).

The ENETS guidelines recommend targeted therapies in second-line and their use in first-line when chemotherapy is not appropriate [70].

In pancreatic NEC, there is no validation of a targeted therapy (except bevacizumab; see above).


In nonpancreatic NET G-1/2, an mTOR inhibitor (everolimus) has been evaluated.

In the Radiant 2 study, everolimus plus octreotide, compared with placebo plus octreotide, tends Well-Differentiated Grade 3 NET to demonstrate some efficacy [71].

Everolimus improved progression-free survival in patients with advanced NET associated with the carcinoid syndrome.

The median progression-free survival by central review was 16.4 months in the treated group versus 11.3 months in the placebo group (hazard ratio 0.77, 95% CI 0.59-1.00; one- sided log-rank test p 5 .026).

These data were significantly confirmed by the Radiant 4 trial, which reported that everolimus was associated with a 52% reduction in the estimated risk of progression or death in nonpancreatic NET G-1/2 (hazard ratio [HR] 0.48 [95% CI (0.35-0.67], p , .00001) [72].

Today, there are no data to propose a mTOR inhibitor or a tyrosine kinase inhibitor as first-line treatment in NET G-3.


The Clarinet study confirmed the efficacy of somatostatin analog, lanreotide in nonpancreatic NET G-1/2, and identified a benefit in pancreatic NET G-1/2 [74].

Lanreotide was associated with a significantly prolonged progression-free survival among pa- tients with metastatic NET G-1/2 (Ki-67 , 10%).

Patients with well-differentiated metastatic NET, Ki-67 proliferation index up to 5%, stable disease before treatment, and low-to-moderate hepatic tumor involvement (,25%) have been associated with tumor control during lanreotide treatment [75].

As more precise understanding of NET cell biology evolves, more accurate identification of individual tumors SST receptor (SSTR) profile will probably facilitate a more precise delinea- tion of SST analog treatments [74-76].

Nevertheless, because of the lack of data in NET G-3, the use of somatostatin analogs might be restricted to low hepatic tumor involve- ment and stable disease.

The use of somatostatin analogs in NET G-3 should be only considered with a close monitoring and may be after confirmation of SSTR expression on SST imaging.


Presence of liver metastases secondary to NET of pancreatic and nonpancreatic origin has been shown to be a major factor in the alteration of both quality of life and prognosis [77].

Hepatic artery embolization or chemoembolization, as well as radioembolization, have been shown to have good clinical, biochemical, and morphological responses when liver burden does not permit ablative therapies [78].

Similar efficacy of hepatic arterial embolization versus chemoembolization in the treatment of liver metastases from well-differentiated non- pancreatic NET in a prospective study has been identified [79].

These data do not favor the hypothesis of an additive efficacy of arterial chemotherapy versus embolization alone.

To date, no randomized study has sought to compare the efficacy of either embolization or chemoembolization in NET G-3.

The choice of management depends on liver burden, metastases pattern, origin of the primary tumor, tumor differentiation, and tumor proliferative activity.

In the presence of extrahe- patic metastases, especially bone, a systemic treatment is preferred.


Peptide receptor radionuclide therapy (PRRT) is an effective form of treatment for patients with metastatic NET.

However, delivering sufficient radiation dose to the tumor to result in a high percentage of long-term tumor remissions remains challenging because of the limits imposed on administered activity levels by radiation damage to normal tissues.

Sabet et al. identified a specific efficacy of peptide receptor radionuclide therapy with 177Lu-octreotate in advanced nonpancreatic NET [80].

PRRT was well tolerated and very effective in well-differentiated midgut NET.

Disease control rate was similar in patients with a ,2% Ki-67 and 3%-20% Ki-67 NET, 97% and 87%, respectively (p 5 .36) [80].

The treatment with 177Lu-DOTA0-Tyr3-Octreotate plus best supportive care (30 mg Octreotide long-acting repeatable [LAR]) was compared with high-dose (60 mg) Octreotide LAR in patients with inoperable, progressive somatostatin receptor-positive, well-differentiated NET G-1/2 of the small bowel and confirmed its efficacy (median overall survival: not reached, 8.4 months; hazard ratio [95% CI]: 0.21 [0.129- 0.338]; p , .0001) [81].

To date, no data exist that would justify PRRT in NET G-3.


The identification of NET G-3 is the cornerstone of an adapted therapy.

This entity does exist and represents up to 6% of NET.

Prognosis and response rate seem to be close to well- differentiated NET G-1/2 with a worse overall survival.

It seems that the cell differentiation should be first taken in consider- ation when discussing the therapeutic options.

To date, cell differentiation, mitoses, and Ki-67 index must be known before treatment choice.

The identification of a biological marker would guide the physicians to the best therapy.

Our recom- mendations for management are based on extrapolations of the literature, which has scant data on this particular subset of patients, warranting future studies further focusing on the clinical differences and molecular underpinnings in these patients.


We read with great interest the excellent article by Coriat et al. titled “Gastroenteropancreatic on Well-Differentiated Grade 3 Neuroendocrine Tumors: Review and Position Statement,” published in The Oncologist [1].

The authors reviewed in detail the epidemiology and made recommendations for the management of this new entity.

They concluded that Gastroenteropancreatic (GEP) Neuroendocrine Tumors (NETs), Grade 3 (NET G-3), exist and differ from Poorly Differentiated (PD) GEP Neuroendocrine Carcinomas (NECs) in terms of prognosis.


Almost simultaneously, we published a multicenter Italian retrospective analysis of a large series of GEP NECs that was not referenced in this article [2].

Our report identified an 18% prevalence of NET G3 (24/136) and confirmed that the category we named “NEC Type A”—defining a well-differentiated (WD) tumor with a Ki-67 index of 21%–55%—had a significantly different better survival than that seen with PD GEP NEC.

Furthermore, our study revealed that not only morphology but also Ki-67 index can affect survival.

In addition, patients with neoplasms falling into the PD GEP NEC category and with a Ki-67 index ≤55% had significantly longer survival than those with Ki-67 index >55%.


On the other hand, Coriat et al. suggested that a pancreatic primary may be more likely classified as a G3 NET, rather than as NEC, although the references for that statement did not provide information on the frequency of NET G3 at different sites.

In our study, we observed that the prevalence of the pancreatic primary site was remarkably represented in 21% (29/136) of the total cohort; of those patients, 34.5% (10/29) showed WD morphology [2].