Kromophone
About
The Kromophone is a program developed at Gordon College by Zach Capalbo to provide color information through an auditory stimulus. It maps one pixel of color to several overlayed sounds. The program takes the input from a webcam and chooses the center pixel or an averaged area around the center pixel and converts the intensity of the red to a high pitch noise in the right ear, the green to a middle pitch noise in both ears, and the blue to a low pitch noise in the left ear. The intensity of the white is then seperated into distinct high pitch sound. The resulting sounds allow the wearer to distinguish colors.
The Kromophone can interface with a Nintendo Wiimote for use as a remote control. Using the wiimote, a user can change preferences such as coarseness of vision, contrast, and light autofocusing. The user can also activate a special sweep mode. The sweep mode moves the area to be translated back and forth over the image, mimicking the neck motion of human subjects during early trials. This sweep motion proved to be very helpful in allowing subjects to navigate paths.
We have had great success so far. Both blind and blind-folded sighted subjects were able to identify fruit in a normally lit environment with only a few minutes to a few hours of training. Blind folded subjects were able to navigate the campuses sidewalks by distinguishing between the grey concrete and the green grass. We even had one subject who was able to navigate Harvard square with the device.
Read the paper published in the Asia Pacific Computing and Philosophy Conference 2009.
I gave a talk on the Kromophone at the PSYPHIR workshop in October 2009:
For an artistic interpretation/demonstration of the device, check out this awesome video done by Sara Hendren: http://vimeo.com/11064419
Also, check out David Botticello's Reflection on using the device.
Kromophone Generations
The orignal Kromophone only runs on linux. The dependencies are gtk, Video for Linux 2,FreeImage, Portaudio Version 18, and libcwiid. To use a webcam, the webcam must support v4l2 and yuyv compression.
The original code was written on a lark over a weekend while I was a freshman at Gordon College. It was subsequently expanded and used as a proof of concept test bed for new ideas.
The next iteration of the Kromophone was developed by Bill Rosser for his senior project to port the Kromophone to Android. See the sourceforge page.
The current generation of the Kromophone is kept at Kromophone.com. The code has been completely rewritten, and is now written in Qt, and is cross platform and friendly.
A VR Kromophone is included in the experimental features section of VARTISTE