PhD report · Part II · Theoretical framework
Describing images
at several levels
The framework
Scope
1 Requirements
Which levels the retrieval system should handle.
2 Evaluation
The system’s ability, evaluated level by level.
Running example
Query example
Find photographsform and detailsform of the paintingform Adoration of the Shepherdsiconographic by Hugo van der Goesnon-visual, focusing on the expressionsdepicted of the shepherdsdepicted and the foreground still lifedepicted.
A system can find “shepherds” and miss “Hugo van der Goes”. A single recall score says that it missed, not which kind of information it missed.
Existing frameworks
Framework objectives
| Panofsky | interpreting works of art |
| Jaimes & Chang | indexing image collections |
| Hollink et al. | classifying how people describe images |
| Yu et al. | scoring critiques of artworks |
Applying them to queries is the adaptation. The same levels on both sides give retrieval a shared vocabulary: each phrase of a query has a level, and an image either supports a description at that level or does not.
Framework 1 · Panofsky (1939)
Three strata of meaning
| level | what it covers | in the example |
|---|---|---|
| I Pre-iconographical | what is factually and expressively depicted, recognised from everyday experience | A woman kneels before an infant in a manger; men rush in from the left; two bearded men draw back curtains. Haste and astonishment. |
| II Iconographical | conventional themes, stories and allegories, identified through texts | The Adoration of the Shepherds (Luke 2): the Virgin, Joseph, the Christ Child; Isaiah among the prophets. |
| III Iconological | intrinsic meaning: what the work reveals about its culture | A costly, exuberant altarpiece for the urban elite of Flanders, not New Devotion asceticism. |
| no level | how the image looks; facts about its making | colour, composition, medium, painter, date, size |
Of 300 queries
100%
ask for something no stratum holds: almost always the medium (96%), often the artist (60%) or a date (56%).
Framework 2 · Jaimes & Chang (2000)
A ten-level pyramid, plus non-visual facts
| level | what it covers | in the example |
|---|---|---|
| 1–4 syntactic | how it looks: type and technique, global colour and texture, local shapes, composition | oil painting; bright blue at the centre; a wide frieze framed by curtains |
| 5–6 generic | everyday objects, people and scenes | woman, infant, shepherds, angels, ox, ass, sheaf of wheat; a stable |
| 7–8 specific | objects, people and scenes identified by name | the Virgin, Joseph, Christ, Isaiah; the Adoration of the Shepherds |
| 9–10 abstract | what an object or the whole image symbolises or is about | the revelation of Christ, staged for the viewer |
| B / A / P non-visual | biographical (author, date), associated (text, context), physical (size, location) | Hugo van der Goes, ca. 1480; Gemäldegalerie Berlin; 97 × 245 cm |
Of 300 queries
14%
ask for something no level holds. Built to index images; semantic levels after Shatford (1986), who derives them from Panofsky.
Framework 3 · Hollink et al. (2004)
Built from how people describe images
| level | what it covers | in the example |
|---|---|---|
| Non-visual | creator, title, date, period, material, place, source | Hugo van der Goes; Adoration of the Shepherds; ca. 1480; oil on panel |
| Perceptual | colour, shape, texture, composition, type and technique | bright blues and reds; a wide frieze; painting |
| Conceptual · general | generic objects, events, places and times | woman, infant, shepherds, angels; kneeling, rushing; a stable |
| Conceptual · specific | the same, identified by name | the Virgin, Joseph, Christ, Isaiah; the Adoration of the Shepherds |
| Conceptual · abstract | the same, as meanings, emotions and ideas | awe, haste, devotion; revelation |
Of 300 queries
19%
ask for something no class holds. Each conceptual level is crossed with object, event, place, time and relation.
Framework 4 · Yu et al. (2026)
Five levels of depth in a critique
| level | what it covers | in the example |
|---|---|---|
| L1 visual perception | palette, line, composition, perspective, chiaroscuro | a bright palette; the still Virgin against the rush around her |
| L2 technical analysis | brushwork, medium, subject and genre, style and school | oil on panel; a Nativity altarpiece; Early Netherlandish |
| L3 symbolic | iconography, symbolism, inscriptions, patronage, literary references | the Adoration from Luke; Isaiah; the curtains as revelation |
| L4 historical context | period, artist biography, influence, provenance | Hugo van der Goes, ca. 1480, after his withdrawal to a cloister |
| L5 philosophical | aesthetic ideals, emotion, cross-cultural exchange, value | worldly splendour and devotion in one object |
Of 300 queries
4%
ask for something no dimension holds. Made to score critiques by 15 VLMs, not to index or retrieve; L2 mixes medium with subject.
Labels on 300 queries
What the queries ask for, by framework
| form | depicted | iconographic | abstract | non-visual | no place | |
|---|---|---|---|---|---|---|
| Panofsky | — | 65% | 62% | 1% | — | 100% |
| J&C | 98% | 66% | 57% | 16% | 94% | 14% |
| Hollink | 98% | 72% | 48% | 14% | 97% | 19% |
| L1–L5 | 33% | 98% | 55% | 6% | 93% | 4% |
Share of 300 of the 372 benchmark queries asking for each kind of information
(a query usually asks for several, so rows do not sum to 100%). Each whole query labelled
by gemini-3.6-flash under each framework, blind across frameworks; unreviewed.
Almost every query names a medium and a non-visual fact (creator, date, source); few ask
at the abstract level. L1–L5 files medium under L2, counted here as depicted.
Queries and images
Each level draws on a different source
| source | levels | in the example |
|---|---|---|
| the pixels | J&C 1–4, Hollink perceptual, L1 | oil painting, a wide frieze, bright blue |
| the pixels | Panofsky I, J&C 5–6, Hollink general, L2 | shepherds, angels, manger, sheaf of wheat |
| pixels + texts | Panofsky II, J&C 7–8, Hollink specific, L3 | the Adoration of the Shepherds; Isaiah |
| pixels + the culture | Panofsky III, J&C 9–10, Hollink abstract, L5 | revelation; splendour for an urban elite |
| metadata only | J&C B/A/P, Hollink non-visual, L4 | Hugo van der Goes, ca. 1480, 97 × 245 cm, Berlin |