Artificial intelligence systems, known as ‘kratt’* in Estonian, are revolutionizing the way people tackle big, data-intensive, and repetitive tasks by providing faster and easier solutions. The government has a major role in developing these systems, as AI is increasingly being deployed in diverse areas.
Estonia’s Heritage Board is one of the cultural institutions that recognizes the significant benefits of utilizing these AI systems. Last year, STACC partnered with the Board to create Sälli, a museum preservation ‘kratt’ that employs an artificial intelligence developed by data scientists to assist museum personnel staff in preserving old artifacts that require special care. As a follow-up to the successful collaboration, the development of a prototype of the Folli ‘kratt’ for describing museum objects was launched.
“STACC has been a reliable and responsive partner. The Heritage Board has had a very positive working relationship with STACC, and I believe the most crucial factor is that we share a common set of values. They prioritize creating something new, useful, and necessary, and that is essential for us. During the Folli project, I witnessed how the STACC team’s enthusiasm grew as they explored the possibilities. Data science plays a critical role in today’s world, and I’m pleased that Estonia has such strong players in this field. They are highly motivated to make history with their projects, and that inspires us as well! It is also essential to note that they are always meticulous in their administration and meet deadlines.”
Mirjam Rääbis, Head of the Heritage Board
Why does a museum need a KRATT?
The goal of the heritage board is to enhance the quality of museum object descriptions by implementing image recognition technologies to automate the process of describing digital objects.
MuIS is a museum information system that employs approximately 200 individuals responsible for describing museum objects on a daily basis. These professionals describe both newly acquired items and retrospectively archived objects in the museum’s collection. The MuIS database currently holds descriptions of 3.7 million museum objects, more than 1 million of which include accompanying illustrations.
On average, one individual can describe 20-30 museum pieces in a single day. In 2019, for instance, 214,000 museum objects were described across 60 different museums. Despite the high volume of descriptions, the quality of the descriptions is inconsistent, which impedes the widespread utilization of cultural heritage resources.
Annually, approximately 1 million pages of digital material are added to DIGAR, a digital repository. A significant portion of this material includes images that currently lack detailed descriptions, making them difficult for users to search for in a contextualized manner.
How can you describe museum pieces with KRATT?
During the KRATT project, STACC’s data scientists conducted tests on machine learning solutions designed to analyze museum image collections, seeking models that would be practical to implement in this specific context. Machine learning is a rapidly evolving branch of artificial intelligence that focuses on analyzing images and extracting useful information from them. With machine learning methods, it’s possible to approach image collections from various perspectives. For example, it can automatically identify the image’s type (whether it’s a photograph, painting, postcard, document, or other), recognize the objects, landscapes, and people in the image, and even read the text within the image. Typically, each of these tasks requires a different AI model. By combining multiple models, a ‘kratt’ can be constructed that is capable of providing a versatile description of museum collections.
Folli predicts objects and locations
The Folli project conducted tests on various segmentation models, which analyze images and predict the appropriate category (i.e., what each pixel represents) for each pixel in the image. This helps determine both the WHAT and WHERE of the objects in the image. Figure 1 provides an example of the segmentation model’s predictions, showing its ability to recognize key elements of an image, such as people, sky, buildings, and nature. However, it’s important to note that artificial intelligence has its limitations and can make mistakes, as demonstrated in Figure 2, where the AI identified a Russalka as a bird with 59% probability due to the shape of its wings. Therefore, it’s crucial for humans to evaluate and interpret AI predictions. Without such follow-up, this work cannot yet be fully trusted to robots.
Folli clusters and sorts
The second experiment of the Folli project was image sorting and clustering. The sorting model was given the task of analysing an entire image collection at once and placing visually similar images in a single group and similar groups close together. After sorting, the model formed a large “image cloud” of all the photos. This is an exciting way to visualise and categorise large image collections. Figure 3 shows an example where, while exploring the collections of the Museum of History, the model has brought together images of chess sets, and Figure 4 shows a group of Christmas cards from the Digital Archives collection. Since the model itself decides what to sort and what not to sort, it never knows exactly which image clusters will form. An interesting example is shown in Figure 5, where the model has decided to group together pictures of skis, garden flags, gravestones, and soldiers standing in a line. The model likely found that all of these images have recurring vertical elements (rows).
Explore one of these image clouds for yourself at http://folli.stacc.cloud/demo, featuring over 250,000 images collected from various museums’ image collections. Can you discover any commonalities in the clusters?
Vahur Puik, an expert in visual heritage from the NGO Estonian Photographic Heritage and leader of the Ajapaiga collaborative creation platform, played a significant role in the project’s success. Ajapaiga’s aim is to enhance historical visual material, mainly old photos, with various types of additional content. In addition to MuIS data, the analysis relied on hundreds of thousands of tags added to images by Ajapaiga’s volunteer contributors.
Around 2.5 quintillion bytes of data are generated globally every day
As the amount of data being generated continues to grow, distinguishing between valuable data and noise is becoming more challenging. However, STACC has a proven track record of collaborating with both public and private sectors, ranging from large enterprises to start-ups, to successfully solve problems and uncover new opportunities across various fields. Our data scientists are passionate about leveraging cutting-edge technologies to tackle complex challenges and drive innovation.
* Kratt is a magical creature in old Estonian mythology built from hay or old household items to serve its master. In Estonia, kratt is used as a metaphor for AI. Sälli comes from “säilitamine”, the Estonian word for preservation.
