ITP Blog

My journey in making things

Self Portrait – ‘Mean Emotion’

The idea of a self portrait is a rather challenging one. It holds the premise of conveying the ‘DNA’ of an artist, but more importantly to me, it tells the story of someone through the period of time when the portrait was taken/made. Therefore, portraits are inherently a document of a period of time that no longer exists for someone, and on the shoulders of this understanding, I decided to pursue ‘Mean Emotion’.

The story

I moved to New York city a little over a year ago, to attend ITP. Before coming to NYC, I lived in Israel with my partner for (around that time) over 5 years, Katia. When we got the message informing us about my acceptance into ITP, we both knew what it means. The reason behind this mutual understanding is that Katia was still attending school in Israel, and so we decided we would split but do everything we could to support and maintain that relationship. Fast forward, one year after as I sat down thinking about the portrait I realized one of the tools that helped us in this crazy period of time, is our ability to express emotion through selfies we send to each other.

Data-set collection from WhatsApp

The project

Recently I have begun digging deeper into data science and machine learning, with an emphasis on graphics and imagery. One thing I found myself doing recursively is aligning and averaging data-sets to be able to clearly see variances  and deviations that a so-called ‘learning’ model could potentially pick up. And so I decided to use the aforementioned selfie data to try and average a ‘year’s worth of emotions’ . The data set came to be 230 images taken from the 8th of 2016 till the 8th of 2017.

I collected all the images while paying special attention to WhatsApp’s naming convention, which stores date and time in the file name itself. From that point I started build a face aligner that would be able to parse the images and align them in 3D, which makes for a much better face matches. I started by examining Leon Eckert’s ‘Facemesh Workshop’ github repo and started build my own FaceMesh class in python. The project required Python 3+, openCV, dlib and numpy, which all have very good installation guides online. With dlib in place, you can use pretrained face landmarks .dat file which gives very good results out of the box for detecting faces. Another great resource was pyimageresearch’s facial tracking tutorial.

The console application running

Github repository for the project and the tool could be found here.

After some coding and testing I was able to run the script and start aligning faces on a reference image. My plan was to run the app and have it output a print of the entire year, 8/2016 – 8/2017.

The failure

After getting the code to work, and spending hours on alignment, resolution and compression (haaaaaaa), I was able to produce a good mean image from all the data set. Feeling uplifted I decided to go to Laguardia Studio and print the portrait on high quality paper using their museum grade inkjet printer. Upon getting there, and after a brief ‘up-to-speed’ tutorial by the staff I realized my image doesn’t meet the DPI requirement by the printer and set down to redo the part of the code that had to do with that. Unfortunately, after adjusting it the program took 3 hours on my computer and Laguardia studio already closed.

Lesson learned: “measure twice and cut once

Shooting a printer I never got to print anything on (meta)


The result

Here is the printed result of 8/2016- 8/2017, of 228 images shot during the year averaged into a single image.

I also took the time to sort the averaging process by month to create 12 more images each containing the selfies sent during that month.

I also created a speculative video of how the piece could be displayed:

Hi I am Or, I am a director, developer and artist. My current interest in research is sound interaction, computer vision & immersive media development.

Leave a Reply

Your email address will not be published. Required fields are marked *