Introduction
The AI & Arts interest group is a multi-disciplinary effort rooted within the Turing encompassing multiple stakeholders drawn from academia to cultural heritage institutions to the creative industries and policy makers.
The group is motivated by the fact that we are currently experiencing a digital revolution driven by the constantly increasing amount of data in our world. The ability to create, interrogate and use this wealth of data enables us to promote meaningful and insightful collaborations across the different disciplines represented by our group, sector partners, affiliated organisations and creatives.
In particular, the availability of rich datasets within the arts – including cultural heritage institutions – the combined knowledge and experiences of our members and today’s computational methods enable us to explore new tools to investigate emerging large multi-dimensional datasets and creative practices; to understand how data science and the arts influence one another; and to create new knowledge driving forward research at the interface between AI, data science and the arts.
Explaining the science
The use of AI and computationally intensive methods, for example machine learning, have seen rapid growth in the last decade. We are currently producing more data than ever and this requires not just state of the art techniques to analyse such data, but also raises questions around its collection, usage and ethics.
These questions can’t be answered sufficiently by one discipline alone and therefore the need for cross-disciplinary investigations and collaborations is bigger than ever – we conduct research now to answer the questions of the future.
AI and Arts practice
There is a mature tradition of work between art and technology innovation going back to the 1960s. Today we see the emergence of new artistic forms in which AI is a tool or topic, and increasing numbers of artists are experimenting with AI to support, enhance, simulate or replicate creativity. Some artists work with data and technology as material, generating formal and aesthetic outcomes by modifying a training dataset or parameters of a machine learning model, or to explore novel configurations of humans and algorithms.
AI for cultural heritage
For AI researchers, cultural heritage is a rewarding, high-profile area in which to develop, test and demonstrate new methods. For cultural heritage sector researchers, AI may offer entirely new ways of accessing and understanding collections at scale. Importantly, both sides bring different goals and skills to the collaboration. Traditionally, AI researchers bring the methods while cultural heritage professionals supply the materials, in the form of data. The research questions guiding the exercise can arise from either party, from a conversation between the two, or from third parties. AI is also performed internally, by institutions with their own capabilities in digital scholarship, or by third-party researchers benefitting from open datasets and the increased maturity of machine learning software.
AI in cultural heritage necessarily includes critical and ethical debate about the histories of collections, their classification schemes and related scholarship, and about diversity and representation within both AI and cultural heritage research and curation. The field is therefore a participant in the wider debate around AI’s capabilities and limitations; the protocols through which research should be transparently carried out and assessed; the accreditation of results and the allocation of benefits; and around what, ultimately, AI is good for.
AI in the creative industries
The creative industries are broadly defined by government and include many topics outside this group’s remit. The AI & arts group is primarily interested in those scenarios where artistic content is created and then delivered and consumed. This means we look at the application of AI and data science to
- support cultural institutions to improve their understanding of their audiences
- support and enhance creative practice in music, performance, visual arts and moving image
- connect creative individuals and their creations to audiences
In helping cultural institutions to understand their audiences, AI and data science are enabled to help them connect better to those audiences, providing more meaningful and immersive experiences. By creative practice, we include not only authors, composers and artists, but also the producers, directors and engineers who shape and package the artistic creations. And we include broadcasting, archiving and discovery in our definition of how to bring creations to audiences.
AI and cultural policy
There is a need for cultural policy in the UK to move towards a strategic vision and capability for AI and the arts, and for data-led transformation more widely in the cultural sector. The AI & arts group will create opportunities for cultural professionals to engage on a palpable level in new and emerging research. This can in turn help to equip cultural agencies to better support, enable and guide positive change in the arts sector through policy development and by developing awareness, access and responses to AI technologies among arts organisations and practitioners.
Aims
We bring people from data science, heritage, the arts and creative industries, to work together to advance the interface between artificial intelligence, data science, and the art – acting as a platform of cross-disciplinary and co-created research within the UK and beyond.
We see the arts as a broad field inherently encapsulating all forms of expression and creative practice including its study. Our broad and inclusive topics of interest therefore include:
- AI and arts practice: new artistic forms in which AI is a tool or topic; use of AI to support, enhance, simulate or replicate creativity; AI as an autonomous creator and a creative partner.
- AI for cultural heritage: the role of AI in the investigation, interpretation, preservation and management of the arts and cultural heritage.
- AI in the creative industries: AI used to create, produce, distribute, and consume creative industry products.
- Critique and commentary on AI: the role of the arts in improving understanding and explanation of AI.
- AI in cultural policy: transformations in the cultural sector through applications of AI.
The AI & arts group therefore acts as a platform to shape an interdisciplinary scientific community serving multiple purposes. We aim to facilitate collaborations, form research consortia, and host outreach activities such as seminars, research workshops, and events orientated to the public.
The use of AI and computationally intensive methods, for example machine learning, have seen rapid growth in the last decade. We are currently producing more data than ever and this requires not just state of the art techniques to analyse such data, but also raises questions around its collection, usage and ethics.
These questions can’t be answered sufficiently by one discipline alone and therefore the need for cross-disciplinary investigations and collaborations is bigger than ever – we conduct research now to answer the questions of the future.
Talking points
How can AI-arts collaborations solve societal challenges?
AI entrepreneurs are experts in the development and delivery of solutions, products and services using large quantities of data and sophisticated algorithms. As innovators, they deal in business plans, go-to-market strategies, supply chains, and marketing campaigns to disrupt and reach customers at global scale. Artists innovate differently - they use data and AI as resources to create new experiences and communicate complex issues in a universally accessible language to various audiences. Both skillsets are crucial to tackle any modern societal challenge, from climate change and science education to online polarisation and mistrust in experts and established institutions.
Challenges: Startups and artists stand to learn a lot from one another, and can achieve so much more together than apart. Such collaborations require time and resources, as well as novel formats that go beyond established programmes in each field, like incubators or residencies.
Potential solution: In order to understand and address these challenges, we need research frameworks and methods that capture how these collaborators go about addressing a challenge. For example, participatory AI frameworks, tools that allow for broader stakeholder engagement, as well as qualitative and quantitative methods to study collaborations, identify common patterns, or suggest areas of improvement and tool support.
The MediaFutures toolkit allows us to study these collaborations systematically. It offers funding, training, specific support and evidence-based impact assessment as a way to bootstrap interactions. A match-making service enables artists and innovators to meet and compare notes about shared areas of interest, and arts/industry collaborations are actively encouraged through higher funding rates than artists or innovators would receive on their own. This enables both artists and entrepreneurs to learn about each other's language and world views in collaborations. If successful, these interactions can broaden perspectives on both sides, and form the foundation for fruitful partnerships.
Example output: Smoking Gun is an interactive artistic experience, delivered through a mobile app. It was developed by the artist group FastFamiliar and the Data Stories research project. The collaboration was enabled through a STARTS residency. Smoking Gun engages its audience through a series of interactions with data.
How can the arts augment and help to answer questions for AI?
Challenges: Deriving meaningful and actionable insights, including scientifically rigorous requirements, through experiences created with artists.
Example output: Insights, strategies, methods and tools for AI practitioners.
How can AI and data science techniques reveal a painting’s creative process (including an artist’s palette, materials, and methods), reveal a painting’s restoration history, or inform strategies for conservation and preservation?
Challenges: This requires new computational approaches capable of leveraging emerging imaging and analytical techniques such as macro x-ray fluorescence (MA-XRF) and hyper-spectral imaging (HSI) to deliver the distribution of materials present in a painting.
Example output:
Visual AI for cultural heritage research
Much of the attraction of cultural heritage for machine learning lies in the fact that museums and galleries possess large collections of images labelled with features such as the subject, date or place of origin of an image. In a so-called ‘supervised’ machine learning system, a deep neural network can learn visual features from these labels. The system can then detect such features within unlabelled images, permitting a form of automated classification. A common application is visual search, where an object, individual, location or a more abstract quality of an image (such as colour or texture) is identified within an unlabelled image collection.
Challenges: Beyond the convenience of being able to find relevant images, machine-classification of visual materials is of technical interest to AI researchers as the results provide a very literal way of seeing how particular methods work in practice. AI systems can also uncover latent similarities, patterns or outliers within image collections that may not have been apparent beforehand. Researchers and curators in turn can see existing systems of classification put to the test when extended to materials that have not been classified, perhaps usefully extending those classification schemes, or else revealing their gaps, inconsistencies or biases. The usefulness of the results will depend on the choice of method, the extent and relevance of the training data and, importantly, the assumptions or biases of those providing and using both the search system and the data.
Example output:
Searching for bearded figures within paintings using Convolutional Neural Networks (CNNs). A research collaboration between the Visual Geometry Group at the University of Oxford and ArtUK.
Handwriting Recognition Technology, Machine Learning, and AI for Archives
Handwritten text recognition has the potential to transform access to our written past through the use of computers to process and search scanned images of historical papers.
Challenges: Making it easier for everyone to read, transcribe, process and mine historical documents.
Example output: The Transkribus project, an inherently inter-disciplinary one, applies deep neural network models to manuscript material. With Transkribus, historical manuscripts of all dates, languages and formats can be read, transcribed and searched by means of automated recognition.
Semantic image segmentation
With the development of smartphones and unprecedented internet interconnectivity, crowdsourcing has become a main data source in the ‘era of big data’. Compared to traditional computer algorithms, tools based on artificial intelligence hold significant promise to automatically extract useful information from massive amounts of unstructured crowdsourced data. Semantic image segmentation is an approach which not only classifies objects in an image, but also partitions and labels them into distinct segments. That is to say image segmentation assigns probabilities of labels to each pixel in the image. Current state-of-the-art image segmentation is based on deep neural networks combined with transfer learning, a method enabling users to fine-tune a generalised model for a specific task. Locating boundaries between objects in an image is the main advantage of image segmentation over image classification, as it extracts useful information from an image that is more meaningful for researchers and practitioners, such as the position, shape and area of the object within the image. In addition to the application in this project, image segmentation receives significant attention in areas such as automatic driving, medical image processing and robotics.
Challenges: Automatically segment objects with images of heritage sites to detect and monitor changes over time, reducing resources that would otherwise be allocated to traditional monitoring.
Example output: The current implementation of this project is to automatically monitor vegetation growth in Bothwell Castle in Scotland by comparing the areas of growth in similar images submitted by visitors over time. This is also broadly applicable to monitor other objects in other sites by using different training images to fine-tune the model.
How to get involved
Webinars
Our AI for Arts YouTube channel is currently under construction. We will shortly be posting videos of our past webinars, please check back soon.
Upcoming webinars
Upcoming webinars will be posted here.
Past webinars
07/12/2022 Dr Drew Hemment (University of Edinburgh and Turing Fellow, UK)
17/09/2021 Dr Patricia Vitoria (Universitat Pompeu Fabra, ES) Watch now
17/09/2021 Professor Shira Faigenbaum-Golovin (Duke University, US) Watch now
25/06/2021 Francien Bossema (Centrum Wiskunde & Informatica, NL) Watch now
25/06/2021 Robert Laidlow (Royal Northern College of Music, UK) Watch now
21/05/2021 Dr John Delaney (National Gallery of Art, US) with Dr Tania Kleynhans (Rochester Institute of Technology) Watch now
23/04/2021 Professor Aleksandra Pizurica (Ghent University, BE) Watch now
Organisers
Lukas Hughes-Noehrer
Researcher, University of Manchester
Léllé Demertzi
Research Project Manager The Turing Way | AI&Arts Interest Group Organiser
Professor Sunil Manghani
Organiser of the AI and Arts Turing Interest Group
Semeli Hadjiloizou
Researcher in Data Justice and Global Ethical FuturesContact info
For any queries, please contact [email protected]
Representatives
Owen Hopkin – Arts Council England
Tonya Nelson – Arts Council England
Christopher Haworth – University of Birmingham
Lindsey Askin – University of Bristol
Maja Maricevic – British Library
Torsten Reimer – British Library
Dominic Oldman – British Museum
Diana Tanase – British Museum
Caroline Bassett – University of Cambridge
Carola Bibiane Schönlieb – University of Cambridge
Brigitta Zics – University College London
Drew Hemment – University of Edinburgh
David Murray-Rust – University of Edinburgh
Fabrizio Nevola – University of Exeter
Arran Rees – University of Leeds
John Stell – University of Leeds
Caroline Jay – University of Manchester
Nick Bryan-Kinns – Queen Mary University of London
Mark Sandler – Queen Mary University of London
Catherine Higgitt – National Gallery
Marika Spring – National Gallery
David De Roure – University of Oxford
Seth Giddings – University of Southampton
Sunil Manghani – University of Southampton
Sean Hand – University of Warwick
Steve Ranford – University of Warwick
Elena Simperl – King's College London