Animal and human studies have suggested a strong causal rela- tionship between all levels of iron deficiency-even in the absence of anaemia-and the extent of physical activity.

29 Data from iron-deficient anaemic rats have shown a substantial improvement in walking ability within 15-18 h of iron administration despite unaltered Hb levels.

30,31 In heart failure, there is evidence that iron deficiency, and not anaemia, is an independent predictor of exercise intolerance in patients with heart failure.

In a prospec- tive study, peak oxygen consumption (VO 2 ) and the ventilatory response to exercise (VE-VCO 2 slope) were measured in 155 patients with stable heart failure (NYHA class I-IV) and mean LVEF of 26%.

26 Iron deficiency was defined as ferritin <100 ng/mL, or 100-300 ng/mL with TSAT <20%.

The results consistently indi- cated a significantly reduced exercise tolerance in patients with iron deficiency vs. patients with adequate iron, a difference that was maintained after adjustment for demographic and clinical variables.

26 These data confirmed findings from a previous prospec- tive study in 27 heart failure patients in whom iron deficiency was defined as TSAT <20%.


The negative influence of iron deficiency on exercise tolerance can be linked to the non-haematopoietic effects of iron, especially if one considers its role in mitochon- drial function in cells with high energy demand, such as cardiomy- ocytes and skeletal myocytes.

4,23 Mitochondria are the key sites of cellular iron processing, where synthesis of iron-sulfur (Fe/S) clusters and haem takes place, but are also the location for gen- eration of reactive oxygen species (ROS).

The activity of mito- chondrial Fe/S cluster proteins is reduced in failing hearts, an effect attributed to the lack of an Fe/S cluster centre.

33,34 On the other hand, clinical data and evidence from animal models with specific genetic defects have suggested that myocardial iron overload 35 or accumulation of iron in the mitochondria can induce or exacerbate cardiomyopathy, 34 presumably as a consequence of ROS-dependent mechanisms.

The question of whether iron accu- mulates in the mitochondria of failing hearts due to a shared aeti- ology, such as hypertension or myocardial infarction, has not been examined.


In a post-hoc analysis of a cohort of >500 chronic heart failure patients, iron deficiency was associated with lower quality of life than in those patients with normal iron levels, regardless of anaemia status.

In multivariate analysis, the effect on quality of life © 2015 The Authors.

European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology.

The pathophysiological impact of iron deficiency in heart failure patients with preserved ejection fraction (HFpEF) is not yet fully elucidated.

An observational study of 751 patients with heart failure which included a subpopulation of patients with HFpEF (defined as LVEF >50%) found that functional iron deficiency (TSAT <20%) occurred with a similar frequency in patients with or without HFpEF.

36 One well-designed study of 26 patients with HFpEF observed no significant association between iron deficiency (defined as ferritin <100 ng/mL or 100-299 ng/mL with TSAT <20%) and exercise capacity.

27 However, multivariate analysis of data from an epidemiological multicentre study of 1278 patients with heart failure showed an association between iron deficiency and health-related quality of life which was independent of whether LVEF was preserved or reduced.

28 Further studies with larger populations are needed to determine whether iron therapy can improve symptoms and cardiac function in HFpEF patients.


Oral iron therapy is relatively inexpensive and widely used, but the absorption of iron is low.

It also means that iron uptake can be inadequate to meet enhanced demand during treatment with an erythropoiesis-stimulating agent (ESA), limiting the haematopoietic response.

37 Oral iron is most frequently given in the form of ferrous (Fe[II]) salts, such as ferrous sulfate.

Oxidation of Fe(II) to Fe(III) can provoke oxidative damage at the mucosal boundary, resulting in local toxicity which leads to gastrointestinal side effects in up to 60% of patients.

38 These include complications such as constipation, dyspepsia, nausea, diarrhoea, and heartburn, which are particularly frequent with ferrous sulfate preparations and often lead to poor compliance.

Many constituents of food and some medications interact with ferrous salts, such that iron uptake is best when taken on an empty stomach, but gastrointestinal side effects are often improved by concomitant food, so there can be a difficult balance between maximizing uptake and minimizing gastrointestinal side effects.


Oxidative stress, with the potential for adverse consequences, has been observed in animal studies during treatment with fer- rous salts, especially at high doses.

Evidence suggests that this may include disruption of cell membranes, lipid peroxidation, DNA strand breakage, and immunological disturbances.

39 Moreover, fer- rous salts have been found to increase the severity of experimental colitis and affect the intestinal microbiomass due to redox reac- tions which lead to the production of ROS and mediate intesti- nal damage in rodents, 40 with evidence of increased prevalence of intestinal adenomas.


Long-term iron overload via dietary iron in rats induces necrosis in multiple organs, an effect attributed to cellular iron accumulation and consequent oxidative stress.

42 The effects of oxidative stress can be avoided by use of the oral prepara- tion iron[III]-polymaltose complex, which includes a polymaltose shell for stability so that iron is released in a controlled manner and non-transferrin-bound iron (NTBI) does not increase.

43 The structure of the iron[III]-polymaltose complex also minimizes the interactions with food or concomitant medications that can affect uptake of iron from ferrous salts.


Clinically, data from patients © 2015 The Authors. European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology.

with inflammatory bowel disease have demonstrated a significant increase in lipid peroxidation following ferrous sulfate treatment for 14 days compared with iron[III]-polymaltose complex.

44 As a consequence of the adverse events associated with oral iron therapy, it is considered unsuitable for patients who have defec- tive gastrointestinal absorption, those who cannot tolerate the required oral dose, or individuals who show poor compliance.

39 Oral iron is also unlikely to be adequate if a rapid effect is required, or if the rate of chronic iron loss exceeds the replenishment rate that is possible with oral administration.


Intravenous (i.v.) iron preparations bypass the absorption difficul- ties associated with oral iron, even in the presence of elevated hepcidin levels.

43,45,46 A series of i.v.

iron complexes has been devel- oped which enclose an iron-containing core within a carbohydrate shell, stabilizing the structure and controlling the release of iron.

43 The first complex, iron sucrose, was followed by low and high molecular weight iron dextran products and ferric gluconate.


More recently, ferric carboxymaltose, iron isomaltoside 1000, and feru- moxytol have become available.

These are more thermodynami- cally stable, with a low reduction potential, and permit administra- tion of a much higher dose of iron in a single administration without formation of large amounts of NTBI.

47 Among i.v. preparations, the less stable formulations, such as iron gluconate, are more prone to reduction, with greater risk of oxida- tive stress, so that the maximum single dose is smaller.


Experimen- tal evidence has shown significant differences between the effects of different i.v.

iron preparations.

Increased levels of markers for oxidative stress and inflammation have been observed in animals treated with sodium ferric gluconate, iron dextran, ferumoxytol, or iron isomaltoside 1000 compared with ferric carboxymaltose or iron sucrose, changes that were accompanied by hypotension and impaired liver and kidney function.

47 -50 However, it should be emphasized that most non-clinical in vivo studies have been con- ducted with iron doses that are significantly higher than those used in clinical practice.


Animal data showing that ferric carboxymaltose may have a favourable safety profile in terms of oxidative stress compared with iron dextran and ferric gluconate await confirma- tion.

Clinical data are limited, and are largely restricted to patients receiving haemodialysis.

Two recent studies which used standard doses of iron sucrose reported a transient (1-2 h) increase in NTBI and oxidative stress, accompanied by a short-lived increase in lipid peroxidation.

51,52 These results remain unconfirmed, and interpretation is complicated by the complex effects of chronic haemodialysis on oxidative stress, inflammation, and endothelial dysfunction.


It is challenging to compare accurately the risk of hypersensitivity reactions between different i.v.

iron preparations since events are very rare and reporting can be poor.

I.v.

iron complexes that contain dextran ligands can cause dextran-induced anaphylactic reactions.

These are rare, particularly when using low molecular weight dextran preparations, 53,54 but are serious events and have led to recommendations that i.v. iron dextran should be avoided.


Iron is essential for mounting an effective immune response, 56 but it is also biologically plausible that excess iron can impair normal neutrophil and T-cell functioning, with an associated risk of increased microbial growth.

57 The clinical evidence is largely based on observational studies on haemodialysis patients, which have yielded conflicting results, but taking a cautious approach the current Kidney Disease Improving Global Outcomes (KDIGO) Clinical Practice Guideline for Anemia in Chronic Kidney Disease advises avoiding i.v.

iron therapy in patients with active systemic infections.

58 Conversely, withholding iron therapy with consequent iron deficiency may increase the risk of infections.

57 Randomized controlled trials are needed to evaluate all impor- tant safety endpoints, including infection.

Ongoing monitoring of newer i.v.

agents such as ferric carboxymaltose, ferumoxytol, and iron isomaltoside 1000 is essential to establish an extensive evi- dence base similar to that which already exists for earlier prepa- rations such as iron sucrose.

Additional clinical trials are being undertaken to help provide such data, in response to requests from regulatory authorities.


The recent ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure 2012 recognize iron deficiency as a common co-morbidity in heart failure, independent of anaemia, which can contribute to muscle dysfunction and cause anaemia.

17 The guidelines recommend that assessment of all patients with suspected heart failure should include an evaluation of iron status to detect iron deficiency using ferritin and TSAT levels, but this is not yet regarded as standard practice.

59 A suggested algorithm for the diagnosis of iron deficiency in patients with heart failure is shown in Figure 1, based on typical thresholds for ferritin and TSAT in this setting.

15,17,60 Iron therapy should be considered in all patients with heart failure who have iron deficiency regardless of whether anaemia is present or not.

If ESA therapy is indicated for the treatment of anaemia-predominantly in patients with renal dysfunction-concomitant iron therapy may be necessary to meet the enhanced demand for iron during ESA-stimulated erythropoiesis.

5 The majority of heart failure patients receive oral iron therapy, with only ∼1 in 10 iron-treated patients being given i.v.

therapy.

61 Certain characteristics of patients with heart failure which can limit intestinal absorption of oral iron preparations can be partic- ularly pronounced and therefore i.v.

iron is the preferred option.


in 22 patients with heart failure and observed significantly greater thickness compared with controls, accompanied by significant changes in permeability.

63 In a subsequent study, the same team showed that intestinal blood flow is reduced by 30-43% in heart failure patients vs. controls, 64 an effect which would impair oral iron uptake from the gut.

Other changes can include overactiva- tion of the sympathetic nervous system, the effect of concomitant medications such as H 2 blockers, and complications of co-morbid conditions such as diabetes-related gastroparesis (Table 2).

Thirdly, where hepcidin levels are raised, expression of the iron transport protein ferroportin in enterocytes is suppressed such that release of absorbed iron into the bloodstream is restricted.

65 While more evidence is required, it appears that increased levels of circulat- ing hepcidin characterize early stages of heart failure, although iron deficiency becomes more prevalent in more severe stages of the disease.

14 This apparent paradox remains unexplained, but hepcidin levels appear to be elevated in most patients with heart failure.

14 Additionally, even after iron is absorbed, elevated levels of hepcidin inhibit ferroportin-mediated transport of stored iron out of enterocytes and macrophages, causing iron sequestration 46 and reducing the amount of iron available for haematopoiesis and other cellular functions.

The high intracellular iron levels achieved by i.v.

administration, however, can overcome this 'hepcidin block' by stimulating overexpression of ferroportin to release iron for transport in the plasma bound to transferrin 14,66 (Figure 2).


The median required dose of iron to achieve iron repletion in iron-deficient patients with heart failure is 1000 mg.

67 In patients with iron deficiency anaemia, the rate of iron bioavailability from a therapeutic dose of oral 100 mg ferrous sulfate, the most commonly used oral iron preparation, and iron[III]-polymaltose complex, is ∼10%.

68,69 Given that a typical dose of ferrous sulfate is 100-200 mg iron/day, in the 'best case' scenario, a patient who tolerates treatment of 200 mg/day might absorb only 20 mg of iron and thus require 50 days to correct the iron deficit.

In a 'worst case' scenario, given that patients with chronic heart failure may have as little as half the usual rate of active carrier-mediated intestinal absorption capacity, 62 a patient who can tolerate up to 100 mg iron/day might absorb only 5% (5 mg).

It would thus require 200 days to replenish iron stores.

In practice, correction of iron deficiency for heart failure patients will often take longer due to non-adherence or interruption due to gastrointestinal side effects, or missed doses due to the high pill burden.


Therefore, replen- ishment of iron stores with oral iron can be expected to require >6 months.

In contrast, one or two doses of i.v.

iron therapy can achieve rapid repletion (Table 3).

I.v.

administration, however, places additional demands on the treatment centre (Table 3).


Despite the fact that oral iron is still often the iron therapy of choice for many cardiologists, the evidence base related to oral iron therapy in heart failure is limited.

No randomized trials have compared oral iron vs. no iron therapy in the absence of ESA therapy.

In randomized trials of ESA therapy with oral iron vs. oral iron alone, mean iron parameters and Hb level, symptom severity, and exercise tolerance remained unaltered in the oral iron treatment groups 72 -75 (Table 4).


In the largest of these, 319 patients with symptomatic heat failure and Hb in the range © 2015 The Authors.

European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology.

9-12.5 g/dL were randomized to darbepoetin or placebo, both with daily oral iron therapy.

73 At the end of the 27-week study period, mean values for Hb were almost unchanged compared with baseline in the patients receiving only oral iron (11.6 g/dL vs. 11.3 g/dL at baseline), with little change in ferritin and TSAT levels.

Mean NYHA class was 2.54 compared with 2.67 at baseline, while maximum exercise duration decreased from 409 s to 363 s during the study.

73 The placebo-controlled IRONOUT study (NCT02188784) is currently underway to assess whether treatment with oral iron polysaccharide therapy can improve functional capacity in iron-deficient patients with heart failure.


Three single-arm trials have assessed the effect of i.v.

iron without ESA therapy on Hb level and iron indices in patients with heart failure, using iron sucrose or iron dextran preparations.

76 -78 The entry criteria for the three trials varied in terms of severity of heart failure, the presence or absence of anaemia and iron deficiency, and duration of follow-up (12-26 weeks), but in each case there was a significant increase in mean values for Hb, ferritin, and TSAT from baseline.

Upon treatment with i.v.

iron, various cardiac endpoints also improved significantly, including NYHA functional class, 76,78 LVEF, 77 and exercise tolerance.

76,78 These results are consistent .

with findings from single-blinded or double-blinded randomized trials of monotherapy with i.v.

iron in heart failure patients with iron deficiency, in which ferric carboxymaltose 15 or iron sucrose 32,79 were administered (Table 5).


Two randomized studies which used iron sucrose therapy demon- strated improved exercise capacity, cardiac function, symptom severity, and quality of life 32,79 compared with placebo.


Okonko et al. showed that the benefits of i.v. iron were seen in patients with or without anaemia at baseline, but were greater in anaemic individuals.


In a second trial, 40 patients with anaemia, iron defi- ciency, and LVEF <35% were randomized to placebo or i.v.

iron sucrose therapy for 5 weeks in a 6-month, double-blind study 79 then followed to 5 years.


At 6 months, the number of hospital- izations was significantly lower in the patients who had received i.v.

iron therapy compared with the placebo-treated group {0 events vs. 5 events (P < 0.01); relative risk 2.33 for hospitalization with placebo vs. i.v.

iron [95% confidence interval (CI) 1.59-3.42]}. 79


Hospitalization remained significantly less frequent with i.v.

iron vs. placebo at 1 year (10% vs. 50%) and 5 years (20% vs. 85%) (both P < 0.01) and, notably, mortality was significantly lower at 5 years (20% vs. 55%, P < 0.05).


These findings require confirmation.


FAIR-HF was a randomized, double-blind, placebo-controlled, 24-week multicentre study of 459 patients with heart failure (NYHA class II-III) and iron deficiency, with or without anaemia.

15 Patients were enrolled if their Hb level was between 9.5 and © 2015 The Authors.

European Journal of Heart Failure published by John Wiley & Sons Ltd on behalf of European Society of Cardiology.

non-dialysis-dependent patients.


90 Levels of Hb, ferritin, and TSAT were all significantly increased by i.v.

iron vs. oral iron, and there was a significant reduction in requirement in ESA dose in dialysis-dependent patients treated with i.v.

iron.

The scale of the mean between-group differences was clinically relevant: 0.90 g/dL for Hb, 243 ng/mL for ferritin, and 10.2% for TSAT.


This is consis- tent with recent randomized trials using i.v.

ferumoxytol in patients with chronic kidney disease with or without a requirement for haemodialysis 91,92 and using i.v.

ferric carboxymaltose in patients with non-dialysis-dependent chronic kidney disease, 93,94 each of which showed that the haematopoietic response was significantly greater with i.v.

iron vs. oral iron therapy.


In post-partum iron deficiency anaemia, numerous random- ized trials have compared the efficacy and safety of i.v.

vs. oral iron.

95 These have consistently shown a more rapid improvement in iron stores and Hb levels with i.v.

preparations, accompanied by improved fatigue and quality of life vs. oral iron therapy, with improved compliance.

Using non-dextran i.v. iron preparations, safety was good, with fewer gastrointestinal adverse events than oral iron.


Similarly, in chemotherapy-induced anaemia, a recent meta-analysis of 11 randomized trials of i.v. vs. oral iron ther- apy concluded that i.v. iron was associated with an improved haematopoietic response with no safety concerns, and that the requirement for blood transfusion was lower than with oral iron.


Lastly, several randomized trials testing the effect of i.v. vs. oral iron therapy in patients with iron deficiency anaemia associated with inflammatory bowel disease have shown a faster and more pro- longed haematopoietic response with i.v.

therapy, leading to recom- mendations from the 'Guidelines on the diagnosis and management of iron deficiency and anemia in inflammatory bowel diseases' that i.v. should be the preferred route for administration.

97 One recent large randomized trial of 338 patients with inflammatory bowel .

i.v.

iron isomaltoside 1000 in terms of the pri- mary efficacy endpoint of change in Hb from baseline to week 8 (P = 0.09), but patients with known intolerance to oral iron were excluded and i.v.

iron dosing may have been inadequate; therefore; further trial data are awaited.


Across a range of clinical settings, randomized trials have shown that oral iron therapy does not replenish iron stores, including non-dialysis-dependent chronic kidney disease, 91 -93 inflammatory bowel disease, 98,99 heavy uterine bleeding, 100 and post-partum anaemia.

101 -103


Iron deficiency is widespread in patients with heart failure.

Cur- rently, however, it is not universally monitored or managed despite convincing evidence that iron deficiency is a risk factor for poor outcome independent of anaemia status.

Iron is poorly absorbed from oral preparations, compounded by chronic inflammation and other contributing factors in patients with heart failure, which delays or prohibits iron replenishment.

Evidence for a clinical ben- efit using oral iron preparations in heart failure is lacking.

Clin- ical studies have instead focused on the use of i.v.

iron, which bypasses the problem of gastrointestinal absorption and poor com- pliance.

Early studies using iron sucrose consistently showed that correction of iron deficiency improved exercise capacity, cardiac function, symptom severity, and quality of life.

32,76,77,79 These stud- ies were recently confirmed in the large 6-month FAIR-HF study, where functional capacity, symptoms, and quality of life rapidly increased in patients with heart failure of NYHA class II-III and impaired LVEF 15 in comparison with placebo treatment.

Longer term data were provided by the CONFIRM-HF trial, in which symptomatic, iron-deficient patients with heart failure showed a sustained improvement in functional capacity, symptoms, and qual- ity of life, and reduced risk of hospitalization due to worsening heart failure over 1 year of i.v.

iron therapy.

16 The ESC Guide- lines point out that ferric carboxymaltose may be considered for treatment of these patients, although the effect in patients with pre- served LVEF remains unknown.

17 It should also be borne in mind that assessment of the hard endpoint of mortality is not feasible within the population sizes possible in controlled studies, and that a full assessment of safety would require more extended follow-up.


The structures of new i.v. preparations permit far larger doses of iron to be administered safely in a single visit.

Routine diagnosis and management of iron deficiency in patients with symptomatic heart failure, regardless of anaemia status, and prompt interven- tion using i.v.

iron therapy should now be considered, pending the outcome of larger ongoing randomized controlled trials and longer term safety data.


Anaemia is increasingly recognised as having an independent impact upon patient outcomes in cardiac disease.

The role of novel iron therapies to treat anaemia is increasing.

This systematic review and meta-analysis assesses the efficacy and safety of iron therapies for the treatment of adults with anaemia.

Electronic databases and search engines were searched as per Cochrane methodology.

Randomised controlled trials (RCTs) of iron versus inactive control or placebo, as well as alternative formulations, doses and routes in anaemic adults without chronic kidney disease or in the peri-partum period were eligible.

The primary outcome of interest was mortality at one year.

Secondary outcomes were blood transfusion, haemoglobin levels, quality of life, serious adverse events and length of hospital stay.


64 RCTs (including five studies of heart failure patients) including 9004 participants were included.

None of the studies were at a low risk of bias.

There were no statistically significant differences in mortality between iron and inactive control.

Both oral and parenteral iron significantly reduced the proportion of patients requiring blood transfusion compared to inactive control (risk ratio (RR) 0.66 (95% confidence interval (CI) 0.48 to 0.90) and RR 0.84 (95% CI 0.73 – 0.97) respectively).

Haemoglobin was increased more by both oral and parenteral iron compared to inactive control (mean difference (MD) 0.91 g/dL (95% CI 0.48 to 1.35) and MD 1.04 (95% CI 0.52 to 1.57) respectively), and parenteral iron demonstrated a greater increase when compared to oral iron (MD 0.53 g/dL (95% CI 0.31 to 0.75)).

In all comparisons there were no differences in the results comparing patients with and without heart failure.

Both oral and parenteral iron are shown to decrease the proportion of people who require blood transfusion and increase haemoglobin levels, without any benefit in mortality.

Further trials at a low risk of bias, powered to measure clinically significant endpoints are still required.

This paper updates the findings of a Cochrane review first published December 2014 in The Cochrane Database of Systematic Reviews (CDSR) Issue 12 (http://www.thecochranelibrary.com).


This article refers to ‘Effects of intravenous iron therapy in iron-deficient patients with systolic heart failure: a meta-analysis of randomized controlled trials’, by E.A. Jankowska et al., published in this issue on pages 786–795, and ‘Systematic review and meta-analysis of iron therapy in anaemic adults without chronic kidney disease: updated and abridged Cochrane review’, by B. Clevenger et al., published in this issue on pages 774–785.


Over the last 25 years, treatments for chronic heart failure (HF) with reduced systolic function (HF-REF) have emerged that significantly reduce both mortality and morbidity.

However, other phenotypes of the HF syndrome, particularly HF with preserved systolic function (HF-PEF), still lack effective therapies.

In addition there remains a substantial unmet need for further reductions in modifiable mortality and the avoidance of frequent and costly hospitalizations in HF-REF.

As HF is mainly a disease of the elderly, much attention is now being paid to management of the frequent co-morbidities which are so prevalent in HF patients.


Anaemia and iron deficiency (ID) have emerged as important co-morbidities as both are prevalent in chronic HF and their occurrence is associated with an increase in both morbidity and mortality [1], [2].


There is a growing body of evidence that treating ID in HF-REF patients, irrespective of whether they are anaemic or not, improves outcomes.

The majority of studies have used i.v.

iron preparations.

The use of oral iron preparations has been limited for a number of reasons: their high side effect profile, the time taken to correct ID, poor intestinal absorption which can be due to mucosal oedema, and emerging evidence that HF patients have high circulating concentrations of hepcidin which inhibits the release of stored iron for use in haematopoiesis [3].


Initially small, non-randomized open label studies and randomized controlled trials (RCTs) reported an improvement in surrogate endpoints such as symptoms, quality of life, LVEF, BNP concentration, effort capacity, and peak VO2 [4], [5], [6].


These were followed by two larger RCTs of i.v.

iron compared with usual care.

The FAIR-HF trial recruited 459 patients in NYHA class II/III, with an LVEF ≤40% (NYHA II) or ≤45% (NYHA III), haemoglobin concentrations in the range of 9.5–13.5 g/dL, with evidence of ID demonstrated by a serum ferritin <100 ng/mL, or <300 ng/mL if transferrin saturation (TSAT) was <20%.

Intravenous iron was associated with significant improvements in NYHA class, effort capacity, and quality of life [7].

It was followed by the CONFIRM-HF trial which demonstrated in 304 patients (with similar inclusion criteria), over a longer follow-up period of 1 year, sustained improvements in effort capacity and quality of life [8].

Neither trial was powered to demonstrate a reduction in harder, more traditional HF outcome measures, such as mortality or HF hospitalizations.


With the emerging evidence base, meta-analyses are being performed to try to determine whether the use of i.v iron may be associated with reductions in mortality as well as morbidity.

Two such aggregate data meta-analyses (random effects model) have now been published in this journal.

The first by Jankowska et al. focused on i.v iron therapy in patients with systolic HF and ID [9].

They identified five trials which fulfilled the inclusion criteria (509 patients received i.v iron therapy in comparison with 342 controls).

Intravenous iron therapy reduced the risk of a combined endpoint of all-cause death or cardiovascular hospitalization [odds ratio (OR) 0.44, 95% confidence interval (CI) 0.30–0.64, P < 0.0001] and a combined endpoint of cardiovascular death or hospitalization for worsening HF (OR 0.39, 95% CI 0.24–0.63, P = 0.0001).

It resulted in a reduction in NYHA class, an increase in 6-min walking test distance, and an improvement in quality of life [Kansas City Cardiomyopathy Questionnaire (KCCQ) score: +5.5 points, 95% CI 2.8–8.3, P < 0.0001; European Quality of Life–5 Dimensions (EQ-5D) score: +4.1 points, 95% CI 0.8–7.3, P = 0.01; Minnesota Living With Heart Failure Questionnaire (MLHFQ) score: 19 points, 95% CI −23 to −16, P < 0.0001; Patient Global Assessment (PGA): +0.70 points, 95% CI 0.31–1.09, P = 0.004].


The second, by Clevenger et al., differs in a number of aspects [10].

It looked at a wider group of patients with anaemia, and the iron therapy could be either i.v. oral.

They included RCTs of iron vs. inactive control or placebo, in anaemic adults without chronic kidney disease or in the peripartum period.

As such, patients were included with inflammatory bowel disease, gynaecological problems, those undergoing chemotherapy, as well as HF patients.

The primary outcome was mortality at 1 year.


They report results on 9004 patients in 64 RCTs (including five studies of HF patients).

None of the studies included was at a low risk of bias.

No statistically significant differences were seen in mortality between iron and inactive control.


Both oral and parenteral iron significantly reduced the proportion of patients requiring blood transfusion compared with inactive control [risk ratio (RR) 0.66, 95% CI 0.48–0.90, and RR 0.84, 95% CI 0.73–0.97, respectively].

Haemoglobin was increased more by both oral and parenteral iron compared with inactive control [mean difference (MD) 0.91 g/dL, 95% CI 0.48–1.35, and MD 1.04, 95% CI 0.52–1.57, respectively], and parenteral iron demonstrated a greater increase when compared with oral iron (MD 0.53 g/dL, 95% CI 0.31–0.75).

In all comparisons, there were no differences between patients with and without HF.


These two meta-analyses add to the growing body of evidence which shows that treating anaemia and ID is beneficial for patients.

The first, in HF-REF patients with ID, suggests that harder composite endpoints are reduced by i.v.

iron therapy.

Mortality alone was not reduced.

In the second study, where mortality was the focus, it was also not reduced by iron therapy.

This is perhaps unsurprising due to the heterogeneous inclusion criteria and the differing 1-year mortality rates of the myriad conditions included which ranged from menorrhagia to cancer.


So where does this growing body of evidence take us in terms of the management of ID in patients with HF-REF? In the European Society of Cardiology (ESC) guidelines of 2012 we are advised to investigate the cause of anaemia in HF patients and that the treatment of ID in HF-REF patients may be considered.

The accumulating evidence is pointing to a stronger recommendation in the future.

However, we still await large, adequately powered, studies to determine the effects of iron on mortality and hospitalizations in HF-REF and to determine the nature of the iron supplementation that should be given as well as its long-term safety.


Appelt and Kronfeld had studied the genera- tion of referring expressions, focussing on a range of difficult issues arising from the fact that referring expressions are an integral part of a larger communica- tive act.

The resulting algorithms generated expressions which took discourse context into account (e.g. Appelt and Kronfeld, 1987; Kronfeld, 1989) and of- ten contained more information than was necessary to identify the referent, for example in order to make it easier for the hearer to carry out the process of identification, or in order to meet additional communicative goals (Appelt, 1985).

Although their work offered important insights, its success as a model of human reference was essentially unproven, and this started increasingly to be seen when computational linguists became more data oriented.

For this reason, Dale and Reiter decided to re-focus, concentrating on what they saw as the core of the problem, which is to identify a referent, focussing on simple situations, where nothing matters except the properties of the referent and of the other objects in the domain (the distractors).

Other aspects of the utterance context were temporarily disregarded, not because they were deemed unimportant, but because these researchers thought it wise to focus on simple things first.

Even though this re-focussing on a narrower view of reference was not universal - some continued to investigate the effects of linguistic context (e.g. Passonneau, 1995; Jordan, 2002; Stone et al., 2003) and the interaction with other commu- nicative goals (e.g. Jordan, 2002; Stone et al., 2003) - it has exerted a strong influence on the direction of computational gre.


Let us spell out Dale and Reiter's assumptions in more detail.

The language- generating program takes as its input a knowledge base (kb) whose content is mutually known by speaker and hearer.

The kb ascribes properties to ob- jects, formulated as a combination of an attribute and a value.

If the domain objects are pets, for example, then one attribute might be type, with values such as dog and poodle.

The first aim of gre is to find properties that iden- tify the intended referent uniquely, using a semantic form like ( type: poodle , colour: brown ).

Computing such a semantic form is called Content Deter- mination.

The second aim is to express the semantic form in words, for example the poodle which is brown.


Dale and Reiter examined a number of algorithms, all of which were based on interpretations of the Gricean Maxims, which revolve around brevity, relevance and truth (Grice, 1975).

Roughly speaking, these algorithms sought to minimise the redundancy in a description.

In the most extreme case (e.g. Dale, 1989), the Gricean maxims had been interpreted as dictating the choice of the smallest set of properties of a referent that will uniquely identify it.

Dale and Reiter contrasted this with a more relaxed interpretation of the maxims, which gave rise to what has come to be known as the Incremental Algorithm (ia).

They argued that, in the situations envisaged, the ia produces referring expressions that resemble human-generated referring expressions better than its competi- tors.

Following Belz and Gatt 2008, we shall call this the criterion of human- likeness.

Dale and Reiter also argued that the ia is computationally tractable, and that this constitutes another argument in favour of it.

The two arguments can be seen as related if one assumes that neither people nor machines can solve computationally intractable problems in real time.


Dale and Reiter's empirical position was based on a reading of the psycholinguistic literature (particularly as summarised in Levelt, 1989).

Yet - consistent with existing practice in natural language generation at the time - their paper did not include an empirical test.

Later work (Passon- neau, 1995; Jordan and Walker, 2000) has tested some of their ideas, as we shall see, but this work has tended to concentrate on different (in fact, more complex) referential situations than the ones on which Dale and Reiter focussed.

We aim to put Dale and Reiter's original ideas to the test.

Like many other studies, our investigation will focus on Content Determination (unlike (Stone et al., 2003; Krahmer and Theune, 2002)), disregarding words and syntactic constructions.

Like Dale and Reiter, we shall focus on the task of identifying the referent, dis- regarding the well-documented fact that referring expressions can serve other communicative purposes (e.g. Jordan, 2002; Stone et al., 2003), and we shall focus on "one-shot" referring expressions, as produced in a null context where words outside the referring expression do not play a role.


In one respect we have been less conservative.

For although Dale and Reiter focussed exclusively on singular descriptions, many references to sets can be generated by variations of the classic gre algorithms (e.g. van Deemter, 2002; Gardent, 2002; Horacek, 2004; Gatt and van Deemter, 2007).

This is some- thing we considered worth testing as part of our general plan.

In order not to contaminate the discussion of Dale and Reiter's claims, we only discuss plural references of the kind that can be generated by a relatively simple extension of the ia; moreover, we report results on singulars and plurals separately.

For generality, we have studied referring expressions in two different domain types, one of which involves references to furniture (the furniture sub-corpus) and the other to photographs of faces of people (the people sub-corpus).


The plan for the experiment which is the main focus of this paper was first outlined in van Deemter et al. ( 2006).

The first tentative results were reported in Gatt et al. ( 2007) concerning the furniture sub-corpus, and in van der Sluis et al. ( 2007) concerning the people sub-corpus.


The tuna corpus and evaluation method have influenced the field considerably, in particular after they were chosen as the basis on which gre algorithms were evaluated in the First NLG Shared Task and Evaluation Challenge (stec) on Attribute Selection for Referring Expressions Generation (asgre), in the Spring and Summer of 2007 5 . 22 Different algorithms were submitted to this stec for comparative evaluation, coming from 13 different research teams (Belz and Gatt, 2007).

tuna also featured in a subset of the tasks organised for the second stec in this area 6 , where the number of submitted systems was even greater (Gatt et al., 2008a).

As of October 2008, the annotated corpus is available from the Evaluations and Language resources Distribution Agency (elda) 7 .


The present paper offers a definitive statement of the aims and set-up of the tuna experiment, the structure of the annotated corpus, and our analyses of the corpus (based on a larger number of subjects, in a greater number of conditions, and validated against a second corpus), superseding our earlier papers on the topic.

Consistent with our aim of testing Dale and Reiter's original hypotheses, this empirical analysis is combined with a discussion of the computational com- plexity of the Incremental Algorithm and its main competitors, and concludes with a discussion of the advantages and disadvantages of incrementality.


In order to test Dale and Reiter's claims, we focus our investigation on the algorithms that were most prominently discussed in their paper.

We introduce the algorithms briefly before discussing them in more detail.


• Full Brevity (fb).

Conceptually this is the simplest approach, discussed in a number of early contributions (e.g. Appelt, 1985), and formalised by Dale (1989).

fb minimises the number of properties in the description.

Technically, fb is not one algorithm but a class, since minimality can be achieved by many different algorithms.

fb is computationally intractable, because in the worst case, the time it takes to find a minimal description grows exponentially with the number of properties that the program can choose between (Reiter, 1990).


• Greedy Algorithm (gr).

This faster algorithm was introduced as an ap- proximation to fb by Dale (1989).

Rather than searching exhaustively, it selects properties one by one, always choosing the property that is true of the intended referent and excludes the greatest number of distractors.

gr does not always produce a minimal description, because a property that removes the maximum number of distractors at the time of its inclusion might not remove the maximum number of objects in combination with properties that will later be added.


• Incremental Algorithm (ia).

Like gr, this algorithm selects properties one by one, and stops when their combination identifies the referent.

Incre- mentality, in this broad sense which it shares with gr, was earlier hinted at in Appelt (1985) (where it is also pointed out that "Choosing a prov- ably minimal description requires an inordinate amount of effort").

In the ia, the order in which properties are added to the description is not dictated by their discriminatory power, as was the case for gr, but by a fixed po which tells us, broadly speaking, in which order these properties need to be considered.

It is important to realise that this po is logically distinct from the left-to-right order in which attributes occur within a noun phrase.

The idea, supported by psycholinguistic work (Pechmann, 1989, e.g.), is, rather, that some attributes are more prominent in speak- ers' minds than others, which makes them more likely to be included in descriptions.

( The fact that, in human language production, some at- tributes are selected before others, but realised after them, presents an interesting research problem for psycholinguists (e.g. REF Sedivy et al.) but it goes beyond the scope of the present paper.) Like gr, ia does not allow backtracking: once selected, a property is never withdrawn.

As others before us have observed (e.g. Jordan and Walker (2005); Fabbrizio et al. ( 2008b)), the outcome of the algorithm depends on the choice of po.


A specially constructed example will clarify how these algo- rithms can lead to different outcomes.

Imagine a small domain of objects {a, b, c, d, e, f, g}. Now consider the following five attributes, each of which has two values, denoted as val 1 and val 2 .

We choose values which are each other's complement.

Since complementary values do not overlap, this means that the choice between different values (of a given attribute) is always obvious once the referent is given.

Thus, here and elsewhere in this article, we will focus on the question of attribute selection, on which most research in this area has concentrated.


A different focus would have been possible, for example because one value (e.g., mammal) may be more general than another value (e.g., dog) of the same attribute.

( In section 4.2.2 we shall encounter this phenomenon.) The ia assumes that the different values of an attribute are always roughly equally preferred, with the choice between them depending on matters such as their discriminatory value (i.e., not on a fixed po).

This assumption can be questioned - If an elephant is pink its colour is more worth mentioning than if it is grey - and this can motivate a different ia, whose po defines an ordering on properties (i.e., combinations of an attribute and a value) rather than on attributes.


The choice of the best value of an attribute has, to the best of our knowledge, never been empirically investigated.

See Dale and Reiter (1995) (the FindBestValue function) for an algorithm and van Deemter (2002) for a logical analysis focussing on problems that arise when the values of an attribute overlap.

See also section 8.


Suppose the target referent is the dog e. fb will combine att2: val1 (black, which is true of the objects {a, b, c, e} only) with att4: val1 (outdoor, which is true of {d, e, f, g}), because these two properties jointly single out e while all other descriptions that manage the same feat happen to contain three or more properties.


The output thus consists of the properties black and outdoor.

Notice that realising this content into a description would not be straightforward, since neither property maps naturally to a head noun (it is usually the type of an en- tity which has this role).

This description might be realised as, for example, the black animal that lives outdoors, inserting the category or type of the referent (i.e., animal) artificially.


While gr often produces the same description as fb, this is not the case in the present example.

gr will start by selecting the property att1: val1 (poo- dle, corresponding to the set {c, e, f }), because it is the property that excludes most distractors.

Even though only two distractors are left, namely c and f , no single property manages to remove both of these at the same time.

As the next step, gr will select either att2: val1 (black, removing f ) or att3: val1 (bastard, removing c), but in each case a third property is required to remove the last remaining distractor.

gr does not generate the smallest possible de- scription, but something like the black poodle that lives outdoors instead.


What description is generated by ia? If the attributes are attempted in the order in which they were listed, so att 1 is attempted first, followed by att 2 , then a version of gr is mimicked precisely. But if att 2 is attempted first, followed by att 4 , then the result is the same as that of fb.

In other cases, much lengthier descriptions can result, for example when att 5 is attempted first (narrowing down the set of possible referents to {a, b, c, d, e, f }), and then att 2 (narrowing down the set of possible referents to {a, b, c, e}), followed by att 1 (resulting in {c, e}) and finally att 4 (resulting in the set {e} which contains only the target referent).

The resulting description might be realised as the black poodle with a tail, which sleeps outdoors.


This artificial example shows how dramatically the outcome of the ia can depend on po.

These differences make it difficult to test the hypothesis that ia's descriptions resemble human-generated descriptions.

Which of all the possible ias, after all, is the hypothesis about? It is possible that in most actually occurring situations, different pos lead to very similar descriptions, or ones that are similar in terms of their human-likeness.

We will show that this is not the case, and that there are important differences between the different versions of ia.

This outcome will raise the question how "good" pos might be selected.


The type attribute.

Types are usually realised as nouns.

poodle, for ex- ample, is a type.

Because a referring expression normally requires a noun (we tend to say the brown dog even when brown would identify the referent), Dale and Reiter gave types special treatment.


If type is not selected by the algo- rithm (e.g., because all objects are of the same type), ia adds the property to the description at the end of the search process.

Suppose, for example, the referent is a dog.

Now if the ia starts by selecting the property brown, and if this property happens to identify the referent uniquely, then the property type: dog is added, generating the brown dog.

This makes sure that every description contains one property realisable as a noun.


This special treatment of nouns - foreshadowed in Appelt (1985) - can be seen as independent of the search strategy.

In fact, we believe that the same considerations that make this a good move in combination with ia make it a good move in combination with other algorithms.

In our comparison between algorithms, therefore, we have levelled the playing field by making sure that fb and gr apply the same idea.


Each of the algorithms considered in this paper will therefore add a suitable value of the type attribute at the end of the algorithm.

This did not signifi- cantly change the outcome of any of the algorithms since, in the data we use for our evaluation, type was never either necessary or sufficient to identify a target referent, alone or in combination with other properties (see Section 4.2.3).


We decided to examine certain kinds of expressions that refer to sets as well.

We did this by applying a slightly generalised version of ia to these cases, simplifying a proposal in van Deemter (2002).

The original ia stopped adding properties to a description when they singled out the set whose only element is the referent.

The new ia does the same, stopping when these properties single out a set of referents.

The same idea extends naturally to fb and gr.

This tech- nique allows us, for example, to use the description ( att1: val1 , att2: val2 ) to refer to the set {c, e}. Note that this approach does not work for collective references (e.g. the parallel lines in this picture; cf.

Stone, 2000), which require drastic departures from the algorithms discussed by Dale and Reiter.


Another case which this algorithm will not handle is that requiring plural descriptions involving the union of two different sets, such as the dogs and the cats (van Deemter, 2002; Horacek, 2004; Gatt and van Deemter, 2007).

These cases too will be omitted from the present study.


Empirical work relevant to gre falls into two main classes.

On the one hand, psycholinguistic studies on reference span at least four decades, since the foun- dational work of Krauss and Weinheimer (1964).

On the other hand, partly as a result of increasing interest in comparative evaluation in Natural Language Generation, a number of studies have compared the ia and some other algo- rithms against corpus data.

We give an overview of the main findings from both types of research.


As we have seen, the starting point for contemporary gre research was the notion of Brevity.

Brevity was implicit in an early paper by Olson (1970), who argued that reference has a primarily contrastive function, so that the content of an identifying description would be expected to be determined by the distractors from which the referent is distinguished.


A substantial body of evidence now shows that brevity is not the only factor motivating speaker's choices of proper- ties in reference.

The phenomenon of overspecification, whereby speakers select properties which have little or no contrastive value, was observed in early de- velopmental studies (Ford and Olson, 1975; Whitehurst, 1976; Whitehurst and Sonnenschein, 1978); later studies on adult speakers confirmed the tendency.


Pechmann (1984) showed that adult speakers begin to articulate a reference be- fore their scanning of a visual domain is complete, and that perceptually salient attributes, such as colour, were apt to be selected before other properties, such as size, which required more cognitive effort because their value can be determined only by comparison to other objects.


Later work showed that type and colour were always used in referring expressions in visual domains, even when they had no contrastive value (Schriefers and Pechmann, 1988; Pechmann, 1989), a result not replicated for size.

According to these authors, type has privileged status not only because of syntactic constraints, but because speakers process a referent as a conceptual gestalt, central to which is the referent's object class; similarly, they argued that colour forms part of the speaker's mental representation of an object.


Similar results have been reported by Man- gold and Pobel (1988) and Eikmeyer and Ahlsèn (1996).

A slightly different interpretation is given by Belke and Meyer (2002), who interpret the finding in terms of an attribute's codability, that is, the ease with which that attribute can be included in a mental representation of an object.

Thus, relative attributes such as size are argued to have low codability.


Evidence for overspecification has also been found in connection with locative (Arts, 2004) and relational de- scriptions (Engelhardt et al., 2006).


The demonstration that speakers overspecify has obvious relevance to a study comparing the ia to its predecessors, since one of the consequences of a po is the potential for referential overspecification, with an increased likeli- hood that attributes which are given high priority (say, colour), will be used even though they are not required for a distinguishing description.


One of the first studies to systematically compare the ia to other algorithms fo- cussed on the coconut corpus, a collection of task-oriented dialogues in which interlocutors had to resolve a joint task (buying furniture on a fixed budget) (Jordan, 2000a).

9 Jordan's study compared the ia to two models.

Jordan's Intentional Influences (ii) model (Jordan, 2000a, 2002) views content determi- nation as heavily influenced by communicative intentions over and above the identification intention.

Thus, the intention to signal agreement with a pro- posal (in this case, to buy some item of furniture), may motivate the speaker to repeat a description produced by their interlocutor, for example.

The authors also included a computational implementation of the Conceptual Pacts model proposed by Brennan and Clark (1996), in which a speakers' choice of content for a referring expression is in part based on a tacit 'agreement' with the inter- locutor to refer to objects in a specific way.

The ia was outperformed by both models.

However, the comparison leaves open the question as to whether the ia is an adequate model of referent identification, since the models to which it was compared explicitly go beyond this.


Still within a dialogue context, Gupta and Stent (2005) carried out an evalu- ation on the coconut and the maptask (Anderson et al., 1991) corpora.

They compared the ia and a version of the Greedy algorithm by Siddharthan and Copestake (2004) against a baseline procedure that included the type of an ob- ject, and randomly added further properties until a referent was distinguished.

Additionally, each algorithm was augmented with dialogue-oriented heuristics, and coupled with procedures for the realisation of modifiers.

Thus, the eval- uation used an evaluation metric which combined (a) the degree of agreement between an algorithm's attribute selections and a human's, and (b) the extent to which the automatic realisation of attributes was syntactically a good match to the human realisation.


While these studies offer important insights, they do not directly address the questions outlined in the Introduction to our paper.

In the MapTask cor- pus, for example, most referents are named entities with no true distractors, which can explain why the baseline algorithm outperformed both ia and the Greedy algorithm on this data in Gupta and Stent's study.

In the coconut corpus, these two algorithms outperformed the baseline, but the original ia was outperformed by variants that incorporated dialogue-oriented heuristics.

This is exactly as one would predict, since identification is often not the only refer- ential goal of interlocutors, particularly in the coconut corpus, where other factors have been shown to be paramount (Jordan, 2000a; Jordan and Walker, 2005).

The evaluation metric used by Gupta and Stent incorporated syntactic factors, going beyond the purely semantic task-definition that the IA sought to address.

This, of course, is only a limitation from the viewpoint of a study like the present one, which focusses on Content Determination.


A study by Viethen and Dale (2006) offers a more straightforward compar- ison of the ia and the Greedy Algorithm.

Viethen and Dale stuck to the iden- tification criterion as the sole communicative intention.

They tested against a small corpus of 118 descriptions, obtained by asking experimental participants to refer to drawers in a filing cabinet, which differed on four dimensions, namely colour, row, column and whether or not a drawer was in a corner.

The primary evaluation metric used was recall, defined as the proportion of descrip- tions in the corpus which an algorithm reproduced perfectly.


The comparison of ia and gr revealed a recall rate of 79.6% for the latter, compared to a 95.1% for the ia (with both figures excluding relational descriptions).

Moreover, the corpus contained a limited number (29) of overspecified descriptions, of which the ia reproduced all but five.


Although these results seem favourable for the ia, they only tell us which descriptions are generated by one or more of all (4! =) 24 possible pos of the attributes.

This is because Viethen and Dale com- bined results from all 24 versions of the algorithm, a legitimate move, but one that obscures the extent to which a given version of the ia, with a particular po, contributes to the overall rate.


Recently, Di Fabbrizio and colleagues reported on a study that is closely related to the questions of the present paper (Fabbrizio et al., 2008b,a, see Bohnet (2008) for a related approach), comparing different versions of the ia. 10 One version uses a po that reflects the frequency of attributes in the tuna corpus as a whole, while the other attempts to model different speakers: when modelling a given speaker, the ia uses a po which reflects the frequency with which this speaker (i.e., this subject in the data collection experiment) used the attribute in question.

Although the findings of these studies are intriguing - speaker modelling improved the performance of the algorithm - they need to be treated with some caution.

This is because the set of speakers represented in the training set from which speaker-dependent preference orders were obtained was different from the set of speakers represented in the test set on which their algorithms were evaluated.

This makes it difficult to interpret the conclusion that 'speaker constraints can be successfully used in standard attribute selection algorithms to improve performance on this task' (Fabbrizio et al., 2008b, p.156).

One possible reason for the improvement is that the constraints in question are sufficiently general to apply to classes of speakers rather than individual speakers, and hence can be generalised from one sample (individuals represented in the training set) to another (those represented in the test set).


In summary, most existing gre evaluations do not address the question formulated at the beginning of this paper, because they either placed the ia within the context of a task (such as collaborative dialogue) in which reference is likely to go beyond the primary aim for which ia was designed, or because their evaluation criteria obscure the role of attribute preferences (e.g. by averaging over multiple pos).

The work of Dale and Reiter remains central to current work in gre.

To take an example, although the three stecs organised in this area over the past few years have led to novel proposals, with an emphasis on empirical methods Bohnet (2007); Theune et al. ( 2007); de Lucena and Paraboni (2008); Spanger et al. ( 2008) and the exploitation of novel frameworks such as genetic algorithms Hervás and Gervás (2009); King (2008), many submissions took the ia or one of its two 'competitors' as a starting point.

Gricean Brevity has also been emphasised as a desirable property of algorithms in recent years (Gardent, 2002; Bohnet, 2007).

It therefore seems crucial to put the claims made by Dale and Reiter to the test while maintaining their original assumptions.


The foregoing discussion raises a number of methodological issues that the eval- uation experiment reported below seeks to address.


First, the ia is a family of algorithms, since there are as many versions of it as there are pos.

The question then arises as to whether all these possible versions should be considered, with the combinatorial explosion that this brings about.


Our approach will be to select only those orders which are 'plausible'.

Where possible, we shall attempt to define plausibility in terms of earlier psycholinguistic work.


As we have seen in Section 3.1, however, much of this work has focussed on relatively simple, well-defined visual domains with attributes such as colour and size.


What of more complex domains in which the variety of attributes increases, and the determination of 'salient' or 'conceptually central' attributes is more difficult?


Psycholinguistic research has had less to say about preferences in these con- texts.


For this reasons, it seemed important to investigate the performance of algorithms in domains that are more complex than the ones that have typically been studied, as well as very simple ones.


Since Dale and Reiter's claims focussed on Content Determination, the aims that we set ourselves suggest that a comparison of gre algorithms should ab- stract away from differences in lexical choice and syntactic realisation.

Suppose an intended referent has the properties type: sofa and colour: red , and two human authors produce the descriptions the settee which is red, and the red sofa respectively.

An algorithm which selects both the above properties should be counted as achieving a perfect match to both descriptions.

A comparison should also rest on the knowledge that the algorithm and the authors share the same communicative intentions (namely, to identify the referent).

Of the stud- ies reviewed above, only Viethen and Dale (2006) and Fabbrizio et al. ( 2008b) satisfied this requirement.

The experiment we employed to collect data aimed to minimise potentially confounding communicative intentions.


In line with Dale and Reiter's starting point, the corpus-based evaluation on which we report here focusses on an assessment of the humanlikeness of the descriptions generated by a given gre algorithm.

In other words, we ask how well an algorithm mimics speakers.


An evaluation that compares automatically produced output against human data should take into account partial matches, something that a simple recall measure doesn't do.

We therefore adopt the Dice coefficient, a well-accepted distance metric which computes the degree of similarity between two sets in a straightforward way. ( Section 8 will briefly discuss alternative metrics.) In keeping will earlier remarks, we shall be assessing the similarity between sets of attributes (such as colour), rather than sets of properties (such as green).


The Dice metric is similar to the 'match' metric applied to gre algorithms by Jordan (2000b) (which was defined as X N , where X is 'the number of attribute inclusions and exclusions that agree with the human data' and N the maximum number of attributes that can be expressed for an entity).

Dice is computed by scaling the number of attributes that two descriptions have in common, by the overall size of the two sets: dice(D H , D A ) = 2 × |D H ∩ D A | |D H | + |D A | (1) where D H is (the set of attributes in) the description produced by a human author, and D A the description generated by an algorithm.

Dice yields a value between 0 (no agreement) and 1 (perfect agreement).

We will also report the per- fect recall percentage (prp), the proportion of times when an algorithm achieves a score of 1, agreeing perfectly with a human author on the semantic content of a description.

Finally, a description may contain an attribute several times.

For instance, the green desk and the red chair contains the attribute colour twice.

We treat these as distinct; hence, technically, our Dice coefficient is computed over multi-sets of attributes.


The experiment that led to the tuna corpus was carried out over the internet over a period of three months, yielding 2280 singular and plural descriptions by 60 participants.

Since this corpus was originally developed, two smaller datasets have been constructed, using the same overall methodology.

These were created in 2008 for the comparative evaluation of algorithms in some of the tasks in the second gre Shared Task Evaluation Campaign, tuna-reg'08; one of them was also reused in tuna-reg'09, the third and final gre stec involving tuna held in 2009 (see Gatt and Belz, 2010, for details).

The present study compares algorithms against human descriptions in the original tuna corpus and validates the results against one of the tuna-reg'08 test datasets.

The reason is that the original tuna data collection incorporated some design features - discussed below - which may have introduced complications in the interpretations of results.

Since the reg'08 test sets omitted these features, this comparison gives an additional measure of confidence in our interpretations.

We first describe the design, collection and annotation of the original tuna corpus, followed by a complete overview of the data sources and their use 4.8 below.


Let us use the term domain for a gre domain in the sense outlined in the previous sections, using domain type to refer to the kinds of objects in a domain, such as furniture.

Different domain types can lead to qualitatively different referring expressions (e.g. Viethen and Dale, 2006; Koolen et al., 2009).

We therefore let participants refer to objects in two very different domain types, (a) Furniture trial (b) People trial yielding two sub-corpora, the furniture sub-corpus (Section 4.2.1 and the people sub-corpus (Section 4.2.2).


We had to find a setting in which large numbers of reasonably natural human descriptions could be obtained, each referring to an object for the first time (i.e., without being able to rely on previous references to the same referent).

These constraints militate against the use of dialogues, which is why we opted for the following, more straightforward approach.


In the experiment, each trial consisted of one or two target referents and six distractor objects, with the targets clearly demarcated by red borders, as illustrated in Figure 1.

The participants were asked to identify the objects that were surrounded by the red borders.


Participants were told that they would be interacting with a language-understanding program which would interpret their description and remove the referents from the domain.

This was intended to introduce a measure of interactiveness into the proceedings.

It was emphasised that the aim of these descriptions was to identify the referent.


The system was programmed in such a way that one or two objects (depending on whether there were one or two target referents) were automatically removed from the domain after a participant had entered his or her description.

To emphasise that this task can be performed with different degrees of success, the system removed the correct referent(s) on 75% of the trials, but the wrong one(s) on a quarter of trials, which were randomly determined.


11 Koolen et al. ( 2009) replicated the TUNA experiment with Dutch speakers, manipulating communicative setting: some participants were in a non-dialogue setting while others were in a dialogue setting involving a confederate.

In terms of attribute overspecification, no effect of communicative setting was found.

Pilots with this scheme suggested that this did not discourage participants from taking the task seriously.

Nonetheless, this feature may have introduced confounding factors in our design.

For example, participants' referential be- haviour may have altered as a result of the system's 'misinterpreting' a descrip- tion, leading to less risk-taking behaviour (Carletta and Mellish, 1996).

This is one of the limitations of this methodology which motivated the use of the tuna-reg'08 test data as a second, validating test set, since this dataset was constructed without this feature.


The experiment was designed to produce a corpus that would serve as a lasting resource for gre evaluation; hence, it took into account a number of factors, of which only a few will concern us here.

In the following subsections, we describe the materials and design of the experiment, as well as the corpus annotation.

A more detailed explanation of the annotation procedure can be found in van der Sluis et al. ( 2006) and Gatt et al. ( 2008b).


Referential domains (corresponding to experimental trials) consisted of one or two images of target referents and six distractor objects.

Objects were dis- played in a sparse 3 (row) × 5 (column) matrix.

The positioning of objects was determined randomly at runtime, for each participant and each trial.


Our comparison of algorithms begins with the furniture sub-corpus.

We first identify candidates for 'plausible' incremental algorithms.

As indicated in Sec- tion 3.1, there are strong precedents in the psycholinguistic literature for hy- pothesising that, of the three attributes in this domain, colour will tend to be strongly preferred, while size is dispreferred (e.g. Pechmann, 1989; Belke and Meyer, 2002).

The situation is less clear with orientation.

Frequencies computed from our development dataset, displayed in Table 6, confirm the hypothesised trends, but also evince a tie between size and orientation.

We therefore expect that pos that put colour first will generally perform better, but there is the possibility that the relative order of size and orientation will show a less dramatic difference.

We can test these hypotheses by comparing all the possible ias, as follows: The top panel of Table 7 displays the mean Dice scores for each version of the ia within the two Cardinality conditions under consideration, as well as the prp obtained by the algorithms across conditions.

Overall means are reported for both test sets.


We now turn to the people sub-corpus, where the number of attributes is greater than in our previous analysis.

Hence, the possibilities multiply, both in terms of the number of possible versions of the ia, and in terms of the choices the authors had to describe objects.


Compared with the furniture sub-corpus, the larger number of attributes (nine excluding type) in the people sub-corpus makes testing all possible ias impractical.

Therefore, it is even more crucial to have an a priori estimate of the what pos might constitute 'optimal' and 'sub-optimal' ias.

Here, however, the psycholinguistic literature provides less guidance than before.

Most of the work cited in Section 3 focussed on references to objects with the kinds of attributes we find in the furniture domain type.

( Not a lot has been published on attributes such as hasGlasses.) Therefore, our reliance on frequencies based on the development dataset is greater than before.

Attribute usage frequencies are displayed in Table 10.

Aside from type, the table suggests a gap between a set consisting of the three attributes beard, hasGlasses and hair, and the others.


To construct different versions of the ia, we took all possible permutations of these three attributes, imposing a fixed order on the other six.

Additionally, we again used a version of the ia that reversed the hypothesised 'best' orders; this is our predicted sub-optimal version.

This resulted in the following versions of the ia, in addition to ia-rand: As before, Table 11 gives descriptive statistics for all the algorithms, with the different versions of the ia in the top panel.


The experiment consisted of 38 experimental trials, divided into 20 furniture trials and 18 people trials, each with one or two targets and six distractors in the sparse matrix.

Each trial displays a different domain.

For furniture, the domains were constructed by taking each possible combination of attribute- value pairs in each domain type 13 , and constructing a domain in which that combination was the minimally distinguishing description for the referent(s).

For example, in a domain in which the minimal description for the target referent was { colour: red , orientation: front } (red and facing front), at least one distractor would be a red chair, and at least one other distractor would be a chair facing front, but only the target referent would have both properties.

In the people sub-corpus, the minimal description was calculated based on a combination of three salient attributes found in the van der Sluis and Krahmer (2004) study, namely beard, hasGlasses and the attribute age.


The type was never part of the minimally distinguishing description because it was assumed, based on robust psycholinguistic findings, that it would be included anyway.

The available attribute-value pairs in a domain type were represented an approximately equal number of times.

For example, of the 12 furniture domains where orientation was part of the minimal description, a target faced front or back exactly half the time, and left or right in the rest.


Table 3 presents an overview of the experimental design, which manipulated one within-subjects, and two between-groups factors.

( The abbreviations used in the table will be introduced presently.) The within-subjects factor manipulated the Cardinality and Similarity of objects.

In the case of plural domains with two target referents, the two referents may or may not be sufficiently similar to be describable by means of the same minimal conjunction of properties.

Where this is not possible, one may have to split the description in two parts (set- theoretically, the union of two sets; logically, the disjunction of two conjunctions of properties) saying, for example, the red table and the blue sofa.

Accordingly, Cardinality/Similarity had three levels:


1. Singular (sg): 7 furniture domains, and 6 people domains, contained a single target referent.


2. Plural/Similar (ps): 6 furniture domains had two referents with identi- cal values for the attributes with which they could be distinguished from their distractors.

For example, two pieces of furniture might both be blue in a domain where the minimally distinguishing description consisted of colour.

This was also the case in 6 people domains where, for instance, both targets might be wearing glasses in a domain where glasses: 1 sufficed for a distinguishing description.

In furniture domains, the two referents in this condition had different values of type (e.g. one was a chair, and the other a sofa), while in people domains, they were identical (since all entities were men).


3. Plural/Dissimilar (pd): In the remaining 7 plural furniture trials, and the 6 plural people trials, the targets had different values for the mini- mally distinguishing attributes.

Thus, plural descriptions would always involve a disjunction (i.e., a set union) if they were to be distinguishing.

Since disjunction requires significant extensions to the classic algorithms discussed by Dale and Reiter (1995), we shall omit data in this condition.


The first between-groups factor, ±loc, consisted in whether participants were encouraged to use locative expressions.

Half of the participants were dis- couraged, though not prevented, from using locative expressions (−loc condi- tion), while the other half (+loc) were not.

The former were told that the language understanding program they were interacting with had access to the same domain representation, but had different information about the position of objects, so that using locatives would be counter-productive.

Participants in +loc were told that the system had access to the complete domain of objects, including location.


Locatives were not included in Dale and Reiter's discussion of gre algorithms.

In recent years, it has become increasingly clear that the location of a referent can play a special role in referring expressions (Arts, 2004), and that location requires special mechanisms for dealing with relations such as above (Dale and Haddock, 1991; Kelleher and Kruijff, 2006).

Given that the primary focus of this paper is an evaluation of the claims of Dale and Reiter (1995), we do not consider locative descriptions here.


The second between-groups factor sought to determine whether participants would perceive the communicative situation as fault-critical (±fc).

The group in the fault-critical (+fc) condition was told that the program would eventually be used in situations where accurate referent identification was crucial, and no opportunity to rectify errors would be available.

In this condition, participants could not correct the system's 'mistakes' when it removed the wrong referent(s).

Subjects in the −fc condition were not told this.

Instead, in the 25% of trials where the program removed the wrong referents, they were asked to click on the correct pictures to rectify the 'error' made by the program.


Once again, a full discussion of the effects of the fc factor would take us far beyond the scope of the present paper.

We shall therefore collapse descriptions from both ±fc conditions in our analysis based on the original tuna Corpus.

Although some preliminary analysis of the corpus data suggested that there was no difference between the two conditions in the likelihood of overspecified descriptions, some previous work (von Stutterheim et al., 1993; Maes et al., 2004) suggests that manipulation of communicative context may affect referential behaviour.

These complications further motivate our validation against the tuna-reg'08 test data, which did not manipulate a ±fc factor.


The experiment was run over the internet.

Participants were asked for a self- rating of their fluency in English (native speaker, non-native but fluent, not fluent).

Participants who rated themselves as not fluent were not included in the corpus.

Participants were then randomly assigned to a condition, and read the corresponding instructions.

The instructions emphasised that the purpose of their descriptions is to identify referents.

They were asked to complete the experiment (i.e. all 38 furniture and people trials) in one sitting.

Trials were presented in randomised order.


Each trial consisted of a presentation of a do- main, as shown in Figure 1, where participants were prompted for a description of the target referent(s).

This was followed by a feedback phase, in which the system removed the target referent.

60 participants completed the experiment, 15 in each group depicted in Table 3.


An xml annotation scheme was developed for the corpus, which pairs each corpus description with a representation of the domain in which it was pro- duced.

In the scheme, which is exemplified in Figure 2, a description is repre- sented in three different ways: (a) the original string typed by a participant (the STRING-DESCRIPTION node); (b) the same string with all substrings correspond- ing to an attribute annotated using ATTRIBUTE tags (the DESCRIPTION node); (c) a simplified representation consisting only of the set of attributes used by a participant (the ATTRIBUTE-SET node).


Evaluation required that the domains that were "seen" by humans and al- gorithms be compatible whenever possible.

This was not possible when human- produced expressions contained attributes that were not specified in the domain (e.g., where a person was described as being serious); these were tagged using name='other'.

In Sections 5 and 6, these attributes were treated as different from any system-generated properties.


The corpus was annotated by two of the authors based on consensus.

14 The reliability of our annotation scheme was evaluated by comparing a subset of 516 descriptions in the corpus to the annotations made by two independent annotators (hereafter a and b), postgraduate students with an interest in nlg, who used the same annotation manual.

The Dice coefficient was used as a similarity metric for comparing the three annotated versions of each description.

This allows us to measure the degree to which annotators agreed on the semantic content of a particular description (Passonneau, 2006, cf.).

Table 4 displays the pairwise mean and modal Dice scores.

In all three pair- wise comparisons, there is a high degree of similarity, with the most frequent score being 1.

However, agreement was slightly higher between the two annota- tors a and b than between either of them and the authors.


To take the likelihood of chance agreement into account, we used Krippendorf's α (Krippendorf, 1980), which has the general form shown in 2: α = 1 − D o D e (2) where D o is the observed disagreement between annotators, and D e is the disagreement expected when the units of interest are being coded purely by chance.

We follow Carletta (1996) in assuming that a value greater than 0.8 indicates high reliability (see Artstein and Poesio, 2008, for discussion).

In the present case, the disagreement on two descriptions D 1 and D 2 is calculated as 1 − dice(D 1 , D 2 ).

We followed Passonneau (2006) in adopting the following instantiation of (2): α = 1 − rm − 1 m i D1,D2 n D1 i n D2 i 1 − Dice(D 1 , D 2 ) D1 D2 n D1 n D2 1 − Dice(D 1 , D 2 ) (3) where r is the number of corpus descriptions, m the number of annotators (i.e., 3), i ranges over the individual corpus descriptions and n Dj i is the number of times the set of attributes D j has been assigned to description i (out of a maximum of 3).

The α value obtained was 0.85.

This implies that the three sets of independently annotated descriptions were in high agreement and the annotation scheme used is replicable to a high degree.


As stated above, we use Test Sets 1 and 2 from the tuna-reg'08 stec to further validate our results.

They were constructed via an online elicitation experiment based on the original design, with the following differences:


1. Participants were not told that they would be interacting with a natural language understanding system, and they received no 'feedback' of the kind given in the original experiment.

Rather, they were asked to identify objects as though they were typing them for another person to read.


The between-groups factor manipulating whether or not the communica- tive situation was fault-critical (±fc) was dropped;


3. Only singular domains were used in the experiment.

This is because the stecs did not include plural descriptions.


This experiment was completed by 218 participants, of whom 148 were native speakers of English.

The test sets were constructed by randomly sampling from the data gathered from native speakers only: both sets contain 112 different domains, divided equally into furniture and people descriptions and sampled evenly from both ±loc experimental conditions.

In Test Set 1, there is one description per domain, for a total of 112 descriptions (56 per domain type).

We use this as development data in the present study (sections 5 and 6).

In Test Set 2, there are two different descriptions for each domain, for a total of 224 descriptions (112 in each domain type).

This constitutes our independent test dataset, used for validation of the results on the original tuna corpus.


Table 5 shows the number of descriptions in our two test sets, and our de- velopment set.

Because the furniture and people sub-corpora vary so much in complexity, we focus on each one separately in what follows.

In each case, the data consists of (a) singular descriptions (sg); and (b) similar plurals (ps) elicited in the −loc condition.

These are the two classes of descriptions that can be handled by the algorithms without extensions to deal with disjunction.

Since participants in the −loc condition were not prevented from using loca- tive attributes (though they were discouraged), we further exclude from our test data all descriptions which include them.


As observed earlier, testing all possible versions of the ia would not be practical, particularly in a domain with a large number of attributes.

In each sub-corpus, we therefore selected a subset of the possible ias, focussing on those which prioritise preferred attributes.

Although such preferences can often be identified from previous psycholinguistic work, this is not always possible, especially in the case of the people descriptions.

For this reason, we used our development data to estimate frequencies with which different attributes were used.

The resulting frequency ranking for each subcorpus was used to determine a set of pos that were predicted to be 'optimal'.


For each subcorpus, we report mean Dice scores for algorithms based on both the original tuna Corpus and the tuna-reg'08 test data.

For validation pur- poses, we report correlations between these means.

Significant correlations are taken to suggest that the two datasets are compatible, in spite of the method- ological differences in the data collection.

Focussing on the main test dataset (that is, the original tuna data), our statistical analysis then proceeds in two steps.

First, we compare the set of optimal ias to the predicted sub-optimal ias, as well as a random baseline (hereafter referred to as ia-rand), which always se- lected the type of an attribute, and then incrementally added randomly-chosen properties to a description until the target referent was identified (this strategy follows Gupta and Stent, 2005).

In this part of the study, we are mainly con- cerned with the impact of different pos on the ia.

We then address the question of how the ia compares to other algorithms by identifying the two versions of the ia which are statistically the best and worst, and comparing them to gr and fb.

In each case, we report the results of a by-items anova with Tukey's post-hoc comparisons.


As before, there were strong positive correlations between the means obtained on the two datasets, both when the overall tuna means are compared to the tuna-reg'08 means (r 10 = .9; p < .001) and when the singular subset only is compared (r 10 = .9; p = .001).

Rankings of algorithms are identical for the singular subset of the tuna data and the tuna-reg'08 data.

The rankings on the overall tuna data display some variation in the middle ranks compared to the tuna-reg'08 data, but the two datasets give the same top and bottom rankings (that is, the top two and bottom two algorithms are the same).

This is confirmed by a strong positive correlation between ranks (ρ 10 = .88; p = .001).


At a glance, the table suggests some important differences between distributions of the scores and algorithm rankings in this sub-corpus compared to the previous one.

First, the overall Dice scores are more broadly distributed, with ia-rand and ia-sstaohbg scoring at or below .57.

Second, there does not seem to be such a sharp difference in performance between the sg and ps conditions, with only relatively small decreases in performance.

The exception is ia-sstaohbg, which performs worse in the sg condition compared to ps.

Third, although ia- rand once again ranks above the worst-performing ia, the random procedure peforms worse, in relative terms, than it did on the furniture sub-corpus.

Fourth, the worst-performing ia has an extremely low prp of 7, scoring 0 on this measure in the singular data, meaning that it does not achieve a perfect match with any of the descriptions produced by our subjects.


A 6 (algorithm) × 2(cardinality) univariate anova again showed a sig- nificant main effect of algorithm(F (9, 1876) = 118.47, p < .001) but no main effect of cardinality (F (1, 1876) = 2.22, p > .1).

However, the interaction was highly significant (F (6, 1876) = 6.37, p < .001).

These results confirm the preliminary impressions gleaned from the table, where the decline in perfor- mance on ps is not as sharp as it was on the furniture data.

The interaction is obtained primarily because ia-sstaohbg reverses the trend found for all the other algorithms.


Table 12 displays the homogeneous subsets obtained from the Tukey's pair- wise comparisons.

The table is most interesting in the difference at the two extremes.

At the top of the table, the two best-performing algorithms differ significantly from all other algorithms.

At the bottom, two distinct subsets identify the worst-performing algorithms, one of which is ia-rand.

Interest- ingly, the latter does not cluster with the worst-performing po.

Looking at the two best versions of the ia, the distinction between the pos appears subtle.

This is also evident in the overlap between groups b and c, which mirrors the overlap found in the furniture sub-corpus between algorithms that fall between the extremes (Table 8).


As before, the means for gr and fb in Table 7 suggest that they fall somewhere between the best and worst performing ias.

However, whereas their means were not significantly different in the furniture sub-corpus, for the people sub-corpus a significant difference was found, with fb outperforming gr (with a much higher prp.

In this section, we compare these two algorithms to one of the best ias (ia-bghoatss), and the worst (ia-sstaohbg).


A 4 (algorithm) × 2(cardinality) univariate anova revealed the same trends as with the comparison of the ias in the previous sub-section, with a main effect of algorithm (F (3, 1072) = 192.63, p < .001), no main effect of cardinality (F (1, 1072) = 1.14, p > .25) and a significant interaction (F (3, 1072) = 13.44, p < .001), the latter once again due to ia-sstaohbg.

Homogeneous subsets from the pairwise comparisons are shown in Table 13.

Apart from confirming the superiority of the top-ranked ia, these results also confirm that fb outperformed gr, and both are significantly better than the worst version of the ia.


The answers to our questions are generally more clear-cut on this dataset than on the furniture sub-corpus.

There is an obvious dependency of the ia on the po: ia-sstaohbg does not achieve a single perfect match with any description, and is significantly worse than all other algorithms, while once again, the versions of the ia which perform best are those which prioritise 'preferred' (i.e., in the present case, frequent) attributes.


The increased number of choices in this domain type also means that a ran- dom incremental procedure is more likely to select a distinguishing combination of attributes which a human author would not select.

Another observation con- cerns the distinction between gr and fb; the two algorithms are significantly different on this dataset, with fb performing better and achieving a prp which approaches that of some of the higher-ranked ias.

In general, gr is likely to overspecify, including attributes that are not minimally required for a distin- guishing reference.

The fact that fb achieves a respectable performance means that the validity of the generalisation that people are unlikely to be brief is strongly dependent on the domain type.


As we have argued, much of the psycholinguistic literature has shown pref- erences for attributes such as colour.

On one interpretation (Schriefers and Pechmann, 1988; Pechmann, 1989), this is because of a holistic, 'gestalt' mental representation of objects to which some attributes are central owing to their salience and the fact that they are not relative (unlike size).

A related inter- pretation is that these attributes have high codability (Belke and Meyer, 2002).

However, predictions of overspecification based on these theories do not seem to carry over straightforwardly to attributes such as whether a person wears glasses, or whether they are bald.

The investigation of these phenomena in different domains is an important area of future research.


The main reason why the two Cardinality/Similarity conditions differ less sharply in the people sub-corpus compared to the furniture data is that in the Plural Similar condition in the people domains, the two target referents were both of the same type (i.e., both men).

The kinds of 'partitioned' descriptions that were unavoidable in the furniture corpus (because the two referents in a plural target had different values for the type attribute) would sometimes still arise, as in (4a), where different properties were used to characterise each referent.

However, where the description ascribes the same properties to each referent as in the ps condition, the form will be as in (4b).


(4) (a) the man with a beard and the man with glasses


(b) the men with beards and glasses


Dale and Reiter claimed that the ia is superior to its competitors in two respects, namely human-likeness and computational tractability.

We have so far focussed on the first of these claims, but it is worth discussing the second one as well.


Firstly, suppose we accept tractability as an important consideration.

It is then far from clear that this rules out algorithms such as gr, or perhaps even fb.

To see why, let us assess the run-time complexity of each of these algorithms, use the following abbreviations:


1. n a = number of properties known to be true of the intended referent.


2. n d = number of distractors.


3. n l = number of attributes mentioned in the final referring expression.


Under Dale and Reiter's analysis, gr has a complexity of n a × n d × n l , because it needs to make n l passes through the problem, at each stage checking at most n a attributes to determine how many of the n d distractors they rule out.

By contrast, the ia has a complexity of n d × n l , because it requires n l passes, but does not look for the optimal attribute at each stage, since this is fixed in the Preference po.

Although this makes gr more computationally 'expensive' than ia, the standard view regarding the complexity of algorithms is that only the general shape of the function matters and not its fine details.

Because both algorithms are polynomial, the standard position suggests that they should probably be tarred with the same brush.

In other words, there are no strong computational reasons for preferring ia over gr.

A worse complexity class is only reached with fb, whose complexity Dale and Reiter assessed as n n l a (i.e., exponential).


It's unusual to assess complexity in terms of variables like n l (which is not known before the end of the calculation) and n a (which would require an al- gorithm of its own to calculate).

A similar picture emerges, however, if the al- gorithms are subjected to a more traditional worst-case analysis (van Deemter, 2002).

Such an analysis puts the complexity of ia at n x × n p , where n p equals the total number of properties expressible in the language (i.e., the number of attribute/value combinations), and n x is the number of objects in the domain.

In this analysis, it is easy to see that the order in which properties are tested is irrelevant, because in the worst case, all properties are tested.

The same conclusion follows, namely that both gr and ia have polynomial complexity.


17 Additionally, is debatable whether it makes sense to dismiss a gre algorithm purely because it is computationally 'intractable', and it might be partly for this reason that complexity has been discussed relatively rarely in gre in recent years, when most research has focussed on empirical tests of algorithms on miniature domains, as we have seen.

Suppose algorithm x produced better output than algorithm y, but at a much slower pace. Would we really want to prefer y over x under all circumstances?

The following arguments militate against such a position:


1. Current gre algorithms do not pretend to model procedural aspects of human reference production at all; at best, they offer a good approxima- tion of the descriptions produced by a human speaker.

Thus, the primary question that determines a choice between algorithms is which one mimics human output better, not which one is faster.


2. Computational tractability requires one to make assumptions which, in practice, can be debatable.

Suppose. for example, no referring expression can contain more than a hundred properties.

This reasonable assumption would instantly remove the variable n l from the formula for the complexity of fb, causing this algorithm to run in polynomial time.

An algorithm whose theoretical complexity is polynomial (but whose constants have high values) can easily take more time in practice than an exponential one (whose constants have low values).

Thus, it is often difficult to assess the practical implications of complexity results.


To sum up: the experiments reported in previous sections have led us to question the empirical superiority of the ia.

Our discussion of computational complexity now tends in the same direction.

Combining the evidence, greedy approaches to gre may have a lot to offer after all.


The Incremental Algorithm is by far the best-known gre algorithm to date.

Krahmer and Theune, for example, wrote that 'the Incremental algorithm has become more or less accepted as the state of the art for generating descriptions' (Krahmer and Theune, 2002, p.223).

Horacek wrote that 'the incremental al- gorithm is generally considered best now, and we adopt it for our algorithm, too' (Horacek, 1997, p.207).

Recently, Goudbeek et al. ( 2009) stated that 'Dale and Reiter's (1995) Incremental Algorithm is often considered the algorithm of choice (...), due to its algorithmic simplicity and empirical groundedness'.

Oberlander stated that the ia 'clearly achieves reasonable output much of the time' (Oberlander, 1998, p.506).

The ia has also become the basis of much recent work that seeks to widen the coverage of gre algorithms.

Examples in- clude work on relational descriptions (e.g. Areces et al., 2008), salience (such as Piwek (2009), which takes the incremental model 'as a point of departure'), vague descriptions van Deemter (2006), and spatial descriptions (e.g. Kelleher and Kruijff, 2006; Turner et al., 2007).

No other gre algorithm can boast a similar popularity, particularly for identifying a referent in a 'null context' (cf.

the Introduction section).

Similarly, the fb and gr strategies are still among the main competitors of the ia.

Perhaps the main other contender is the graph- based approach of Krahmer et al. ( 2003), but its main selling point is arguably that it can encode a wide variety of algorithms, including ia, fb and gr.