Intelligent Systems for Screen Archives

Working in partnership with moving image archives in the UK, ISSA aims to develop sector wide understanding about AI technologies in film and television archives, including the knowledge, skills and tools required to rethink and reshape large audiovisual collections from a computational perspective. The project falls within KDL’s Machine Learning and AI research theme, exploring ways to effectively use this technology with humanities research. A significant challenge of using AI in moving image archives is the lack of domain-specific systems that exemplify, end-to-end, how to go from collections to interactions. KDL research software engineers (RSEs) have built prototype frameworks and interfaces which use machine learning methods and tools including video language models to discover, access and exploit digital video content. One prototype analyses video files and breaks them into meaningful segments with descriptive metadata. Another searches for instances where named places are either mentioned or identifiably appear in a video. The partner archives on the project all have numerous items in their collections which are sparsely catalogued and the contents largely unknown, so these kinds of discovery aids will be immensely beneficial to them and their users. KDL is also looking at ways of at least semi-automatically providing audio description for video content. Audio description is a more sophisticated and complex activity than captioning, providing a narrative description of what is happening onscreen when no dialogue or voiceover is being spoken and offers visually impaired users greater access to the video content and a more immersive experience. Also, from a user perspective and building on the preceding work, KDL is thinking about creative ways that users can exploit video content that has been surfaced, identified and made manipulable. From a score of different sources which have been semantically searched and segmented, for example, an interface could help a user put together their own story about the place they live in, the work they do, the hobby they love.

Team

• Arianna Ciula ^KDL Research Software Analyst
• Daniel Chavez Herras Principal investigator, FAH Department of Digital Humanities
• Geoffroy Noel ^KDL Research Software Engineer
• Kirsty Warner Researcher, FAH Department of Digital Humanities
• Miguel Vieira ^KDL Research Software Engineer
• Paul Caton ^KDL Research Software Analyst
• Stefan Meier ^KDL Research Software Engineer
• Tiffany Ong ^KDL Research Software Designer

Project Status: Active