ITP Blog

My journey in making things

Multiplayer Augmented Reality

For my final project in Adv. Computer Graphics I decided to pursue with my current interest in augmented reality. Through our discussions in class I got interested in experimenting with participatory augmented reality and being able to “share” a space. After proposing the idea I started building the infrastructure needed to experiment with multiplayer Augmented Reality experiences.

Multiplayer AR

Early on, I recognized both ARKit and ARCore to be great at persistency and user experience and decided to combine the marker less approach ARKit offers with QR code detection (a more traditional computer vision application) using OpenCV to try and calculate the “offsets” in each client’s camera to the origin of his world matrix. This would be turned into a transformation matrix that I can then share between client and transform shapes so they appear at (roughly) the same spot in the real world, ok enough with the technical stuff this is how I imagined it would work:

I decided to use Apple’s Swift language for simplicity and built the project specifically for ARKit. Upon getting a stable version to work I documented a video of the engine with a simple cube placing experiment (Thanks to Roi for helping out)


Thinking about an AR Google Drive

After the class discussion on participatory Augmented Reality I got really interested in how we perceive our digital assets through 2D abstractions. I have been using Google Drive for quite some time and it always struck me how intuitive Google was able to make Google Drive feel even though storing images, audio and video files in a folder structure is unintuitive by it’s nature, these elements were once tangible. Through a discussion sparked in class after suggesting that, Ken mentioned he is able to find an item in his home even though he could potentially be one of 20,000 or more items and using a service like Google Drive means you are bound to loose files because of how insignificant the action of storing them in some folder is. It’s nothing compared to putting something on a specific shelf in your home.

Since one of the main uses of storing files on a cloud is sharing, once I was able to build the multiplayer system I started thinking about how this could be used for storing files in Augmented Reality and inviting people to explore the files with you.

I was able to prototype the idea of “uploading” the files to a specific location in space, here is a image of that:

What’s next?

  1. I would like to combine the multiplayer engine with a prototype for organizing digital assets in augmented reality.
  2. I would like to add sharing capabilities where people could enter the space and be exposed to the files your shared with them

Hi I am Or, I am a director, developer and artist. My current interest in research is sound interaction, computer vision & immersive media development.

Leave a Reply

Your email address will not be published. Required fields are marked *