Mind vs. Machine: Storytelling Using Neural Image Style Transfer
Deep Dream Generator is a free online application that just about anyone can use to create images using machine learning. Machine Learning (ML) is a subset of artificial intelligence in which computers are able to learn and adapt without following explicit instructions, by using algorithms to analyze and draw inferences from patterns in data. A machine learning model is a digital file that has been trained to recognize certain types of patterns. You train a model over a set of data, providing it with an algorithm that it can use to reason over and learn from those data.
One of the key aspects of the Deep Dream/ML process is the use of and (in my case) layering of output images. I start with a source image, then I do a run of one or more output imagers based on different styles and this process is referred to as an image style transfer. To create my self portrait (see above), I use multiple styles and had the ML/image style transfer algorithm create the outputs. Then, I layered the output images to create one composite image. This composite was done using additional image processing/editing software.
Additionally, there is a storytelling aspect in my process. For example, I created a portrait of the late jazz maverick Chick Corea using two styles (see above). The final composite image was created by me, using additional software. The styles I chose were inspired by Corea’s contribution to jazz fusion, which developed in the late 1960s when musicians combined jazz harmony and improvisation with rock music, funk, and rhythm and blues.
To tell the story visually I used image styles based on two color palettes, including one from the 1960s when the original source photo of Chick Corea was taken and another one based on Afrofuturism. Afrofuturism is a cultural aesthetic and movement that explores the developing intersection of African diaspora culture with science and technology. A color palette that represents Afrofuturism could include violets (blue and red), deep blues, and other complimentary colors. Jazz fusion reminds me of Miles Davis (who Corea played with) and I see those colors in my mind when listening to his music.
The decisions I’m making when choosing image styles and the layering of the image outputs is part of an iterative loop that repeats the process in order to generate certain outcomes. I’m using my knowledge of color theory, as well. Color theory is both the science and art of using color. It explains how humans perceive color; and the visual effects of how colors mix, match or contrast with each other. I took two semesters of Light, Color and Design courses at Pratt Institute and this learning is coming in handy now.
Back then we purchased expensive Color Aid paper that was used for exercises pertaining to color theory. Many of these exercises were outlined in Josef Albers 1963 book, Interaction of Color, a volume that is considered the definitive text on color. Color-Aid is essentially a packet of large paint chips, each a separate gradation of color along an entire spectrum. These piece of paper could be torn, cut, and collaged into designs that, depending upon their complexity, could result in… a long night.
So in many ways, the visual storytelling and color theory experiments are almost as important as the tools themselves. Deep Dream and machine learning are the first step (and sometimes the last). I often go through several rounds of image style transfer to create one image. I can predict which styles could work even before I click the “Generate” button. I applied this method to my work for the Smithsonian. I used Deep Dream Generator to create a series of 11 portraits of “Featured Futurists.” You can see them here.