Whiteboard Synth Project Page
This project is about turning an ordinary whiteboard marker into a magic pen, and the whiteboard into a magic canvas, on which we can draw waveforms and then hear them played back to us as sound waves.
The general principle we aim to demonstrate is the ability to use augmented reality to turn traditional materials (such as a whiteboard) into an interactive control surface. The relatively simple application shown here is just one example of what is possible using Ogglebox's computer vision technology.
This project is also an exercise in applying Ogglebox software technology to obtain useful results even using the relatively low-resolution cameras found on some smart glasses.
Whiteboards let us quickly sketch ideas in a visual way. And vision-based augmented reality lets us align virtual content with visual information. As a by-product of this process, vision-based augmented reality takes measurements of the world around us. In this project we combine these concepts in a new example application.
The on-board camera of the smart glasses is used to measure content drawn on a whiteboard in real-time. As the user draws lines and curves on the whiteboard, the smart glass display gives a graphical confirmation that the system has registered what was drawn. The shape of curves seen by the camera is then processed by the smart glass computer and output as sound waves.
This allows the user to draw sounds and hear them. We may think of this as a ‘whiteboard sound synthesizer’, hence the project name.
Although wearable smart glass technology has advanced hugely in recent years, there remain design compromises in off-the-shelf products. In this project we will be working with the Epson Moverio BT-200 - a product which has given the public unprecedented access to mobile 3D augmented reality at an accessible price point. Developer documentation for Moverio suggests that the built-in camera was intended for decoding QR codes, and therefore did not need to be particularly high resolution. This seems reasonable. Unfortunately it means the camera has limited performance for vision-based augmented reality.
In this project we will use Ogglebox algorithms to extract information from the camera in a robust and reliable way, albeit with the constrained environment of a whiteboard. Our purpose is to show that even with a very basic camera, there is a useful role for vision-based augmented reality on smart-glasses; and at the same time demonstrate some of our own unique capabilities.
The first stage in the processing pipeline is to process incoming video to find those segments of the image corresponding to the waveform of interest, ready for analysis. We achieved this using Ogglebox’s real-time region-based background extraction library, which compensates for the noise and illumination variation across the video images.
Recogntion and Tracking
To extract the waveform, we adopt an approach known as ‘tracking by recognition’. In each video frame, Ogglebox’s oriented line extraction algorithm was used to identify the vertical end stops across multiple possible locations of the head camera relative to the whiteboard. This allows us to compensate for the varying pose of the whiteboard as the user moves their head (and hence the video camera) around. Once found, it is a relatively simple matter to extract a coarse representation of the waveform that the user has drawn between these bounds.
We have conducted initial tests for real-time overlay of the detected waveform and will post results here soon.
Future project updates will show how the final waveform shape is computed and overlaid; and finally how it is output as sound.