Introduction

The AI & arts interest group is a multi-disciplinary effort rooted within the Turing encompassing multiple stakeholders drawn from academia to cultural heritage institutions to the creative industries and policy makers.

The group is motivated by the fact that we are currently experiencing a digital revolution driven by the constantly increasing amount of data in our world. The ability to create, interrogate and use this wealth of data enables us to promote meaningful and insightful collaborations across the different disciplines represented by our group, sector partners, affiliated organisations and creatives.

In particular, the availability of rich datasets within the arts – including cultural heritage institutions –, the combined knowledge and experiences of our members and today’s computational methods enable us to explore new tools to investigate emerging large multi-dimensional datasets and creative practices; to understand how data science and the arts influence one another; and to create new knowledge driving forward research at the interface between AI, data science and the arts.

Explaining the science

The use of AI and computationally intensive methods, for example machine learning, have seen rapid growth in the last decade. We are currently producing more data than ever and this requires not just state of the art techniques to analyse such data, but also raises questions around its collection, usage and ethics.

These questions can’t be answered sufficiently by one discipline alone and therefore the need for cross-disciplinary investigations and collaborations is bigger than ever – we conduct research now to answer the questions of the future.

AI and Arts Practice

There is a mature tradition of work between art and technology innovation going back to the 1960s. Today we see the emergence of new artistic forms in which AI is a tool or topic, and increasing numbers of artists are experimenting with AI to support, enhance, simulate or replicate creativity. Some artists work with data and technology as material, generating formal and aesthetic outcomes by modifying a training dataset or parameters of a machine learning model, or to explore novel configurations of humans and algorithms.

AI for cultural heritage

For AI researchers, cultural heritage is a rewarding, high-profile area in which to develop, test and demonstrate new methods. For cultural heritage sector researchers, AI may offer entirely new ways of accessing and understanding collections at scale. Importantly, both sides bring different goals and skills to the collaboration. Traditionally, AI researchers bring the methods while cultural heritage professionals supply the materials, in the form of data. The research questions guiding the exercise can arise from either party, from a conversation between the two, or from third parties. AI is also performed internally, by institutions with their own capabilities in digital scholarship, or by third-party researchers benefitting from open datasets and the increased maturity of machine learning software.

AI in cultural heritage necessarily includes critical and ethical debate about the histories of collections, their classification schemes and related scholarship, and about diversity and representation within both AI and cultural heritage research and curation. The field is therefore a participant in the wider debate around AI’s capabilities and limitations; the protocols through which research should be transparently carried out and assessed; the accreditation of results and the allocation of benefits; and around what, ultimately, AI is good for.

AI in the creative industries

The Creative Industries are broadly defined by government and include many topics outside this group’s remit. The AI and Arts group is primarily interested in those scenarios where artistic content is created and then delivered and consumed. This means we look at the application of AI and Data Science to

- support cultural institutions to improve their understanding of their audiences

- support and enhance creative practice in music, performance, visual arts and moving image

- connect creative individuals and their creations to audiences

In helping cultural institutions to understand their audiences, AI and Data Science are enabled to help them connect better to those audiences, providing more meaningful and immersive experiences. By creative practice, we include not only authors, composers and artists, but also the producers, directors and engineers who shape and package the artistic creations. And we include broadcasting, archiving and discovery in our definition of how to bring creations to audiences.

AI and cultural policy

There is a need for cultural policy in the UK to move towards a strategic vision and capability for AI and the arts, and for data-led transformation more widely in the cultural sector. The AI & arts group will create opportunities for cultural professionals to engage on a palpable level in new and emerging research. This can in turn help to equip cultural agencies to better support, enable and guide positive change in the arts sector through policy development and by developing awareness, access and responses to AI technologies among arts organisations and practitioners.

Aims

We bring people from data science, heritage, the arts and creative industries, to work together to advance the interface between artificial intelligence, data science, and the art– acting as a beacon of cross-disciplinary and co-created research within the UK and beyond.

We see the arts as a broad field inherently encapsulating all forms of expression and creative practice including its study. Our broad and inclusive topics of interest therefore include:

  • AI and arts practice: new artistic forms in which AI is a tool or topic; use of AI to support, enhance, simulate or replicate creativity; AI as an autonomous creator and a creative partner.
  • AI for cultural heritage: the role of AI in the investigation, interpretation, preservation and management of the arts and cultural heritage.
  • AI in the creative industries: AI used to create, produce, distribute, and consume creative industry products.
  • Critique and commentary on AI: the role of the arts in improving understanding and explanation of AI.
  • AI in cultural policy: transformations in the cultural sector through applications of AI.

The AI & arts Group therefore acts as a platform to shape an interdisciplinary scientific community serving multiple purposes. We aim to facilitate collaborations, form research consortia, and host outreach activities such as seminars, research workshops, and events orientated to the public.

The use of AI and computationally intensive methods, for example machine learning, have seen rapid growth in the last decade. We are currently producing more data than ever and this requires not just state of the art techniques to analyse such data, but also raises questions around its collection, usage and ethics.

These questions can’t be answered sufficiently by one discipline alone and therefore the need for cross-disciplinary investigations and collaborations is bigger than ever – we conduct research now to answer the questions of the future.

 

Talking points

How can the arts augment and help to answer questions for AI?

Challenges: Deriving meaningful and actionable insights, including scientifically rigorous requirements, through experiences created with artists.

Example output: Insights, strategies, methods and tools for AI practitioners.

How can AI and data science techniques reveal a painting’s creative process (including an artist’s palette, materials, and methods), reveal a painting’s restoration history, or inform strategies for conservation and preservation?

Challenges: This requires new computational approaches capable of leveraging emerging imaging and analytical techniques such as macro x-ray fluorescence (MA-XRF) and hyper-spectral imaging (HSI) to deliver the distribution of materials present in a painting.

Example output: 

Visual AI for Cultural Heritage Research

Much of the attraction of cultural heritage for machine learning lies in the fact that museums and galleries possess large collections of images labelled with features such as the subject, date or place of origin of an image. In a so-called ‘supervised’ machine learning system, a deep neural network can learn visual features from these labels. The system can then detect such features within unlabelled images, permitting a form of automated classification. A common application is visual search, where an object, individual, location or a more abstract quality of an image (such as colour or texture) is identified within an unlabelled image collection.

Challenges: Beyond the convenience of being able to find relevant images, machine-classification of visual materials is of technical interest to AI researchers as the results provide a very literal way of seeing how particular methods work in practice. AI systems can also uncover latent similarities, patterns or outliers within image collections that may not have been apparent beforehand. Researchers and curators in turn can see existing systems of classification put to the test when extended to materials that have not been classified, perhaps usefully extending those classification schemes, or else revealing their gaps, inconsistencies or biases. The usefulness of the results will depend on the choice of method, the extent and relevance of the training data and, importantly, the assumptions or biases of those providing and using both the search system and the data.

Example output: 

paintings

Searching for bearded figures within paintings using Convolutional Neural Networks (CNNs). A research collaboration between the Visual Geometry Group at the University of Oxford and ArtUK.

 

Handwriting Recognition Technology, Machine Learning, and AI for Archives

Handwritten text recognition has the potential to transform access to our written past through the use of computers to process and search scanned images of historical papers.

Challenges: Making it easier for everyone to read, transcribe, process and mine historical documents.

Example output: The Transkribus project, an inherently inter-disciplinary one, applies deep neural network models to manuscript material. With Transkribus, historical manuscripts of all dates, languages and formats can be read, transcribed and searched by means of automated recognition.

 

Semantic Image Segmentation

With the development of smartphones and unprecedented internet interconnectivity, crowdsourcing has become a main data source in the ‘era of big data’. Compared to traditional computer algorithms, tools based on Artificial Intelligence hold significant promise to automatically extract useful information from massive amounts of unstructured crowdsourced data. Semantic image segmentation is an approach which not only classifies objects in an image, but also partitions and labels them into distinct segments. That is to say image segmentation assigns probabilities of labels to each pixel in the image. Current state-of-the-art image segmentation is based on deep neural networks combined with transfer learning, a method enabling users to fine-tune a generalized model for a specific task. Locating boundaries between objects in an image is the main advantage of image segmentation over image classification, as it extracts useful information from an image that is more meaningful for researchers and practitioners, such as the position, shape and area of the object within the image. In addition to the application in this project, image segmentation receives significant attention in areas such as automatic driving, medical image processing and robotics.

Challenges: Automatically segment objects with images of heritage sites to detect and monitor changes over time, reducing resources that would otherwise be allocated to traditional monitoring.

Example output: The current implementation of this project is to automatically monitor vegetation growth in Bothwell Castle in Scotland by comparing the areas of growth in similar images submitted by visitors over time. This is also broadly applicable to monitor other objects in other sites by using different training images to fine-tune the model.

SEGMENTATION FIGURE

How to get involved

To join us, please email [email protected]

Webinars

Our AI for Arts YouTube channel is currently under construction. We will shortly be posting videos of our past webinars, please check back soon.

Upcoming webinars

17/09/2021 Dr Patricia Vitoria (Universitat Pompeu Fabra, ES) and Professor Shira Faigenbaum-Golovin (Duke University, US) Register now

22/10/2021 Professor Mick Grierson (University of the Arts London, UK) and Dr Rebecca Fiebrink (University of the Arts London, UK)

26/11/2021 Dr Dominic Oldman (British Museum, UK)

Past webinars

25/06/2021 Francien Bossema (Centrum Wiskunde & Informatica, NL) Watch now

25/06/2021 Robert Laidlow (Royal Northern College of Music, UK) Watch now

23/04/2021 Professor Aleksandra Pizurica (Ghent University, BE) Watch now

21/05/2021 Dr John Delaney (National Gallery of Art, US) with Dr Tania Kleynhans (Rochester Institute of Technology) Watch now

Organisers

Contact info

Andre Piza, [email protected]

Editorial team

Flora Roumpani
Lukas Noehrer

Simone Parisotto
Sunil Manghani

Representatives

Owen Hopkin – Arts Council England
Tonya Nelson – Arts Council England
Max Saunders – University of Birmingham
Lindsey Askin – University of Bristol
Maja Maricevic – British Library
Torsten Reimer – British Library
Dominic Oldman – British Museum
Diana Tanase – British Museum
Caroline Bassett – University of Cambridge
Carola Bibiane Schönlieb - University of Cambridge
Brigitta Zics – University College London
Drew Hemment – University of Edinburgh
David Murray-Rust - University of Edinburgh
Fabrizio Nevola – University of Exeter
Arran Rees – University of Leeds
John Stell – University of Leeds
Lukas Noehrer – University of Manchester
Nick Bryan-Kinns - Queen Mary University of London
Mark Sandler – Queen Mary University of London
Catherine Higgitt – National Gallery
Marika Spring – National Gallery
David De Roure – University of Oxford
Seth Giddings – University of Southampton
Sunil Manghani – University of Southampton
Sean Hand – University of Warwick
Steve Ranford–University of Warwick
Elena Simperl – Kings College London