“Upstream Annotations” and Data Standardization: Strengthening the Foundation of IIIF Image Annotations

4 October 2026

For this week’s blog post I decided to dig into Matthew J. Westerby’s “Annotating Upstream: Digital Scholars, Art History, and the Interoperable Image” (2024)¹. Westerby is the Digital Research Officer at the Center for Advanced Study in the Visual Arts (CASVA, National Gallery of Art, Washington D.C.). He works predominately in digital resources and linked-data, and his art historical work has concentrated in medieval and early modern illuminated manuscripts. Prior to taking on his current role as Digital Research Officer, Westerby also served as a postdoctoral fellow at CASVA in digital projects.

International Image Interoperability Framework, or IIIF, is a standardized application programming interface (API) that allows a user to link additional context about an image to the image’s metadata for viewing online². IIIF data is intended to make it easier for online users to manipulate images by editing the display of the digital object through viewpoints (i.e. zoom level) or annotations. According to IIIF, images with linked IIIF data can be viewed through specialized IIIF viewers such as Annona. Without digital viewers programmed to access attached IIIF data, a user cannot view any annotations or other edits made to the digital image.

I’ll be completely honest here, IIIF still confuses me to a certain extent. I consider myself more programming and data-forward than many in the field of art history due in part to my library science background, but many aspects of IIIF still go over my head. The diagram below from IIIF’s website helped me understand the connection to a digital image’s metadata (the lower flow), and I understand how linking the image metadata and the IIIF data works in theory. I’m not sure how a program like Annona pulls the IIIF data (how is the programming different?), nor do I fully understand how the two data files (the image metadata and the IIIF data) are linked. I would probably have to do a deep-dive into an image’s IIIF JSON file to see the linking process.

Westerby’s article focuses on annotations in IIIF data which allow researchers to add additional context to digital images in public-facing projects. He cites the Digital Mappa platform as one that has the capacity for IIIF annotations. Westerby argues that it is beneficial for digital art historians to re-use base-level metadata about objects in IIIF data. In the instance of Orazio Gentileschi’s The Lute Player that Westerby uses, this “base-level” metadata is the public domain description of the artwork generated by the painting’s owner (the National Gallery of Art). This description forms the basis of Westerby’s IIIF annotations, allowing him to link words in the description to their visual counterparts in Gentileschi’s painting.

“Upstream Annotations” and the Standardization of Annotation Data

The use of public domain descriptions is foundation of what Westerby refers to as “upstream annotations”. Rather than individual art historians and digital scholars generating their own IIIF annotations from scratch, Westerby argues for common resources to form the basis of IIIF annotations. If everyone starts from the same point (i.e. a public domain description of a painting), IIIF annotations can be better preserved and easier to access for more users. Westerby hopes that researchers generating IIIF annotations will be open to not only sharing those annotations publicly via open-source platforms like the Internet Archive, but that “upstream annotations” will also encourage researchers to be more intentional about their data creation.

Those who work with data know that the data-creation quality can vary drastically depending on who is going to use the data in the long term. I know that I have occasionally let my good data practices slip when I know that a particular dataset will only ever be used by me. If no one else ever sees the data, does it matter that it’s not perfect?

Westerby isn’t arguing for perfection in his article, just good practice. Data works best when it is standardized, that’s why library cataloging adheres to MARC 21 or RDA standards and there are widely-accepted best practices when working with spatial datasets. Standardized data is easier to search, easier to share, and ultimately easier to maintain. Westerby wants the starting point for the IIIF annotation data to be standardized and freely accessible, therefore ensuring the foundation of IIIF annotations is rock-solid. Institutions like the National Gallery of Art can promote best data practices by ensuring the starting point for users’ data projects is as seamless as can be.

Video Transcript Annotations

In addition to this week’s blog post, my class was also challenged to edit an auto-generated transcript using Panopto. For those unfamiliar, Panopto is a video content management platform commonly used by companies and universities like UNC Chapel Hill. Like other video platforms such as YouTube, Panopto auto-generates transcripts from videos using artificial intelligence. While oftentimes a solid baseline for improving accessibility, these transcripts can vary in accuracy to the video’s audio. I’ve edited transcripts in Panopto a bit in the past for work, and I’ve noticed that the auto-generated transcripts can struggle with speakers that have heavier accents (when the audio is in English as that’s all I’ve worked with) and with those who tend to talk faster or slur words together. As someone who grew up in and lives in the southeastern U.S., these traits are common amongst speakers with southern accents.

The video I elected to transcribe is one from a vacation I took a couple of years back. My family and I spent a week in the UK and one of our first stops in our rental car was the so-called “Magic Roundabout” in Swindon, UK. While I am the roundabout’s number one fan when it comes to urban planning, the Swindon Magic Roundabout is an absolute chaotic mess of traffic intersection design to my American eyes. I first learned about this intersection years ago from a Tumblr post, and my driving-enthusiast dad decided to make it his personal challenge to conquer the Magic Roundabout. In the middle of the evening rush. While driving a right-hand car. Enjoy what was at least his third attempt.

Matthew J. Westerby, “Annotating Upstream: Digital Scholars, Art History, and the Interoperable Image,” Open Library of Humanities Journal, vol. 10, no. 2 (2024) ↩︎
https://library.stanford.edu/iiif ↩︎

Comments

One response to ““Upstream Annotations” and Data Standardization: Strengthening the Foundation of IIIF Image Annotations”

October 5, 2026

Caroline Fenstermacher

I thought that your analysis of Westerby’s article about IIIF was especially thoughtful and thorough. Practically, I also do not fully understand IIIF and I also struggle to understand how scholars would be able to create the “thick data” that Westerby describes in his article. The graphic that you included is helpful to understand it better, though. At one point in Westerby’s article, he describes how decentralizing data has its own benefits as well. I think that this graphic of the IIIF components helps to visualize exactly how this data is decentralized.
The Image API is created by the owners of the image, likely a museum or similar collecting institution, but additional metadata can originate from a different source. If my understanding is correct, this is likely the place where the scholar can step in to contribute his or her own annotations. I agree that allowing scholars to add their own annotations is beneficial for greater public understanding of publicly accessible images. I do not know, however, how feasible it would be to distinguish the annotations produced by several different scholars with the standardized IIIF.
I think that there is also the challenge of convincing scholars to contribute the information they have. I relate to your experience of letting standardized data formats fall to the wayside when working on my own, individual research projects. I am sure that we have this in common with most other art historians as well. Adding publicly available scholarly annotations to IIIF would add another layer of scholarly labor necessary for a research project.
Thinking about editing transcripts and videos for public use, it was my first time using Panopto, but I found it to be a very user-friendly interface. I actually decided to download an mp4 file of the Rudolph the Red-Nosed Reindeer 1964 Christmas special from The Internet Archive, the open-source platform that you mention in your post. I myself am curious but do not have tremendous experience using different forms of digital data, but I noticed that The Internet Archive allowed the user to download different formats of data, at least for this particular piece of media. I do not know if it would be possible to re-upload my own annotations and corrected transcripts from Panopto back to The Internet Archive, allowing me to contribute to the “upstream” annotation process that Westerby proposes. I wonder if in Westerby’s vision more institutions would allow upstream annotations of their images in addition to the standardized IIIF.

Comments

One response to ““Upstream Annotations” and Data Standardization: Strengthening the Foundation of IIIF Image Annotations”