The cultural heritage sector is experiencing a digital revolution driven by the growing adoption of non-invasive, non-destructive imaging and analytical approaches generating multi-dimensional data from entire artworks. The ability to interrogate this wealth of data is essential to reveal an artist’s creative process, the works’ restoration history, inform strategies for its conservation and preservation and, importantly, present artwork in new ways to the public.
However, the availability of such rich datasets poses a major challenge: traditional approaches are not adequate to inspect the currently available wealth of data, so there is a strong drive and need to explore and adopt automated tools to interrogate these emerging large multi-dimensional datasets. This interest group intends to facilitate the cross-pollination of ideas between stakeholders in the area of data science and in cultural heritage institutions, with the overarching goal to advance the area of artificial intelligence for art.
Explaining the science
Techniques such as 'macro X-ray fluorescence' (MA-XRF) scanning, hyper-spectral imaging (HSI), and traditional (digital) imaging such as x-ray radiography (XRR) and infrared imaging (IRR) have enabled the heritage sector to greatly expand the amount of data they can use to understand an artist's creative process, develop conservation and preservation techniques, investigate art history and present artwork in new ways.
However, the ability to interrogate such multi-dimensional rich datasets calls for new supervised, semi-supervised and unsupervised machine learning approaches capable of ingesting multi-modal datasets in order to tackle a number of relevant challenges arising in the domain of art investigation. These range from:
- Revealing the distribution of materials present within a painting's layers
- Identification, characterisation, and visualisation of relevant sub-surface features present within a painting's layers (not always obvious by simple manual inspection of the data) such as preparatory sketches, pentimenti, or earlier concealed designs
See group organiser Miguel Rodrigues's talk at CogX 2019, detailing examples of challenges that can be addressed by machine learning technology in the area of art investigation:
Bringing together experts crossing various fields such as machine learning, computer vision, art history, art conservation, and cultural heritage, this research interest group aims to explore how emerging AI techniques can shape the general area of art investigation, including art history, conservation and preservation, and art presentation. It will encompass a number of activities including:
- Shaping an interdisciplinary scientific community cutting across areas that are traditionally siloed in the UK
- Serving as a platform to scope cross-disciplinary collaborations, form research consortia, and outreach activities such as research seminars, research workshops, and public oriented events
- Serving as a platform to build AI capability for cultural heritage institutions, contributing to the visibility of UK museums and galleries.
- Acting as a beacon of cross-disciplinary and co-created research within the UK and beyond.
Over the long-term, the 'AI for Arts' interest group has the ambition to also catalyse activities in the wide area of AI for the creative industries, facilitating collaborations in other disciplines beyond visual and fine arts, such as poetry and fiction, music, design and performance.
How can AI and data science techniques reveal a painting’s creative process (including an artist’s palette, materials, and methods), reveal a painting’s restoration history, or inform strategies for conservation and preservation?
Challenges: This requires new computational approaches capable of leveraging emerging imaging and analytical techniques such as macro x-ray fluorescence (MA-XRF) and hyper-spectral imaging (HSI) to deliver the distribution of materials present in a painting.
Example output: Machine learning algorithms capable of ingesting multi-dimensional datasets to deliver the distribution of materials present in a painting.
How can AI and data science techniques enable the visualisation of relevant sub-surface features present within a painting's layers such as preparatory sketches, pentimenti, or earlier concealed designs?
Challenges: This also requires new computational approaches capable of leveraging multi-modal multi-dimensional datasets to deliver visualisations of such features.
Example output: Machine learning algorithms delivering visualisations of sub-surface features support new public experiences in galleries, museums or other cultural heritage institutions.
How can AI and data science techniques be made available to GLAM (galleries, archives, libraries and museums) institutions through a UK-wide research infrastructure?
Challenges: Cultural institutions increasingly require advanced AI and data science techniques to be applied to their collections, to enrich and analyse them. This will require an engineering effort in order to expose them as a service, in order to reach all institutions, large and small, across the UK.
Example output: Image processing and feature extraction, pattern matching and advanced analytics techniques delivered as a service to institutions exposing their data following known standards.
Andre Piza, [email protected]
Owen Hopkin – Arts Council England
Tonya Nelson – Arts Council England
Max Saunders – University of Birmingham
Lindsey Askin – University of Bristol
Maja Maricevic – British Library
Torsten Reimer – British Library
Dominic Oldman – British Museum
Diana Tanase – British Museum
Caroline Bassett – University of Cambridge
Carola Bibiane Schönlieb - University of Cambridge
Brigitta Zics – University College London
Drew Hemment – University of Edinburgh
David Murray-Rust - University of Edinburgh
Fabrizio Nevola – University of Exeter
Arran Rees – University of Leeds
John Stell – University of Leeds
Lukas Nohrer – University of Manchester
Nick Bryan-Kinns - Queen Mary University of London
Mark Sandler – Queen Mary University of London
Catherine Higgitt – National Gallery
Marika Spring – National Gallery
David De Roure – University of Oxford
Seth Giddings – University of Southampton
Sunil Manghani – University of Southampton
Sean Hand – University of Warwick
Steve Ranford – University of Warwick