Data Visualization | Second Look

Data visualization project for Curatorial A(i)gents, an exhibition curated by metaLAB (at) Harvard that took place at the Harvard Art Museums’ Lightbox Gallery in 2022. The below visualizations were programmed in JavaScript using the d3.js library. Visit the interactive website here.

Overview

Not unlike animal forms of intelligence, artificial intelligence relies upon pattern recognition. But its understanding of patterns is dependent upon rigid and rigorous categories predefined by human programmers and upon far more limited data inputs than those provided by the sensory apparatus of a sentient being. Second Look calls attention to the circularity of how an artificial intelligence “sees” and “knows” by asking it to infer gender and sentiment in paintings from the Harvard Art Museums. The result is a curatorial exercise that, as it exposes aspects of how gender has been interpreted by painters across time within the Harvard collections, showcases the powers and limitations of AI as an analytical tool. For conventional AI tools and services like the ones on exhibit, there exist only eight emotions and two genders; conflicted emotions and interstitial gender identities exist only as degrees of uncertainty along the path to a definitive single identification. By making a show of how an AI “sees” and “knows,” Second Look seeks not only to probe how an AI’s predetermined codes filter and interpret the perception of emotion and binary gender identity, but also how similar assumptions propagate cultural norms within contemporary society. 

VISUALIZATION 1/4

WHAT ARE THE MOST FREQUENT SENTIMENTS IDENTIFIED BY AI FOR THE GIVEN SELECTION OF PORTRAITS?

When AI assesses a subject’s sentiment, it provides confidence ratings for the following emotions: happy, sad, calm, angry, disgusted, confused, surprised, and fear. Interested in revealing at a high level the sentiment of the portrait collection, the first visualization groups the paintings according to their most dominant emotion, or emotion with the greatest confidence rating. The size of each circle is relative to that confidence rating, so a “SAD” painting with a 92% confidence score for that emotion will be larger than a “SAD” painting with a similar score of 44%. This visualization reveals that the portrait collection is predominately “CALM” according to AI.  

VISUALIZATION 2/4

HOW CONFIDENT IS AI IN ITS SENTIMENT ANALYSIS OF THESE PAINTINGS?

Each circle represents a painting; circle sizes are relative to AI’s confidence that the painting portrays the given emotion. Because the circles are drawn concentrically, the darker rims help to reveal the average confidence score for each grouping. For example, the overall large size of the “MALE-CALM” grouping and its darkened outer rim reveals that AI consistently classifies “MALE” subjects in paintings as “CALM” with a high degree of confidence.

VISUALIZATION 3/4

AI ONLY "SEES" (SEPARATELY) BINARY GENDERS 

When AI assesses a subject’s gender, it provides a confidence rating for how sure it is about which gender the subject is. Its analysis is purely based on the subject’s physical characteristics, as AI cannot infer a person’s subjective gender identity. Therefore, a higher confidence rating may be correlated with AI recognizing greater outward masculinity or femininity in a subject. This speculative correlation is visualized here via the circle rays. The rings circumscribed by the outer white circumference visualizes the different relationships between the gender categories of female, male, both, and neither, which are umbrella terms for an array of gender identities. The rays extending beyond the white circumference show the frequency that AI recognizes a subject’s gender for a given confidence rating between 1% and 100%. Reflecting the logic of the rings, a 1% confidence correlates with “less masculine” or “less feminine” and a 100% confidence correlates with “more masculine” or “more feminine”, hence the layout of the rays. Because the AI used here has not been trained to "see" non-binary genders, there are no rays extending from the top or bottom sections of the chart. 

VISUALIZATION 4/4

WHAT ARE THE MOST PROMINENT COLORS FOR SPECIFIC GENDERS AND SENTIMENTS?

Interested in revealing at a high level the colors associated with a given gender and/or emotion in the portrait collection, the fourth visualization first observes each painting’s most dominant color, then creates a frequency chart in the form of a color bar. By clicking on one of the colors within the bar in the interactive version, the mosaic chart below is updated with images of paintings whose dominant color matches the selected color. By selecting “Hue” from the lefthand filter menu, the chart can be replicated using the dominant hues of the paintings. 

Second Look Demo Video