May - July 2023 · XR · Commission/Exhibition · Creative Technologist @ NEEEU Spaces GmbH
During the summer of 2023, I undertook a two-month internship at NEEEU Spaces, a Berlin-based creative technology and design agency. In my capacity as a creative technologist, I actively contributed to a wide variety of projects. These included the development of a Unity prototype for a spatial audio-based game, the creation of character concepts in Cinema4D for a chat LLM-based application, and the finalization of responsive layouts in Figma for a museum installation. My biggest project however was working on an augmented reality advertisement for Sigur Rós affiliated perfume brand Fischersund, to be published in BMW's yearly lifestyle magazine, Freude Forever.
The project started out with Fischersund sending out an initial concept and a render to us. The render involved this abstract floral entity surrounded with pearls to reflect the ethos behind Fischersund - creating scents based on memories and feelings rather than corporeal objects. With the creative director of NEEU, Raffael Moco Schiller, I set up a plan to create an Instagram filter using Meta Spark studio that would involve a series of animations going off and forming the flower when someone scans the flower's picture in the magazine.
First, we set off to create a 2D ripple shader using Meta Spark's node system. Then due to the memory/size limitations of creating a graphic-based application as an Instagram filter, we had to use a lot of workarounds. For example, to create the illusions of pearls I set up a billboard shader of flat circles which would track the rotation of the phone, hence giving the illusion of a sphere. After that, I had to write another shader that would mimic the refraction that a sphere does onto a flat circle by converting coordinates from world space to camera space and then passing the normal texture of a sphere into this calculation.
Bringing together the final shader involved finally animating the elements of the original flower sent to us by Fischersund. The stalks were animated to grow in Meta Spark Studio while to animate petal movement, we rigged them with bones and then set off animation clips that get triggered after the flower is formed. After prototyping this implementation and making it smoother and less buggy for Instagram, we handed this over to Fischersund who presented this in BMW's magazine along with a render on their website.
Meta Spark filters have to ship under a tight memory budget, so a real 3D sphere mesh for each pearl was out. I used a billboard of flat circles instead, with a shader that read the phone's rotation to keep the surface 'facing' the camera. From a few angles the same shader tricks the eye into seeing volume.
The pearls also needed to look wet. With no actual sphere geometry to refract through, I wrote a shader that converts world-space coordinates into the camera's space and samples a sphere normal map at the result. It mimics the colour-shift you get from light bending through glass without any of the geometry being there.