The following is the tale of a 17 year old high school student from Utah and his quest to create a Macintosh compatible multi-touch display; the hardware he created and the software coding involved.
The potential for scholarship money can make us do things that we wouldn't ordinarily have the motivation to do. When Brdiger Maxwell discovered Symantec was giving away scholarship money to science fair winners in the state of Utah the 17 year old decided he needed to get involved. It didn't take him long to figure out what his project would be. He drew his inspiration from multi-touch pioneer Jeff Han and his demonstration videos that appeared on YouTube a few years ago. He decided he would make a Macintosh compatible Mac-compatible multi-touch interface.
On January, 1st of this year the graduating junior who attends high school in a double wide trailer began his multi-touch expedition that would later send him to Atlanta to take part in an international science fair.Bridger, who was named after legendary 19th century mountain man James Bridger, learned to program working at the Space Center, a common destination for Utah grade school students that teaches science through space travel simulations. The Space Center, deemed "cooler then space camp" by Maxwell, uses missions straight out of Star Trek and Battle Star Galactica:
"It's supposed to be educational, thats what they run it under and thats where they get their funding. [...] they have simulators there that they base in the Star Trek universe and so you get a crew and you go on a mission there. Everyone gets their uniform and everyone gets a position like captain or tactical officer and you run through a story straight out of Battlestar Galactica or Star Trek."
Some students are lucky enough to be brought on as an intern of sorts where individuals learn to program in Revolution; a sort of great grandchild of HyperCard, and according to the 17 year old uses the same syntax. It is no wonder that with a background like this, multi-touch, a technology that even ten years ago would have seemed straight out of a science-fiction novel, piqued his interest.
The first step for the student was to decide which method he would use for his multi-touch system. According to Maxwell, there are two commonly used methods that most home-brew multi-touch creators use. The first method is called direct illumination where an infrared source is shone directly onto the surface of the display, and a camera capable of viewing the IR spectrum reads the position of the fingers. The advantage to this method is that any object can be detected on the surface of the table.
The second method deals with Frustrated Total Internal Reflection and is better explained by the 17 year old:
"The other method uses Frustrated Total Internal Reflection, or FTIR. The theory is that when infrared light is shined inside the edge of a material such as glass or acrylic, it will be reflected internally to the other end of the material, never escaping. However, when a finger touches this material, it "frustrates" the internal reflection, and causes the infrared light to escape through the surface of the material. An infrared camera then tracks this on the opposite side of the material."Bridger chose the former method citing stability, calibration issues, and outside interference of the direct illumination technique.
The first step was to procure the camera that would eventually be used to deliver the touch information to the computer. I was a bit surprised when Maxwell informed me that nearly any web camera's sensor will pick up the infrared spectrum; the filter for the spectrum just needs to be removed. So he picked up the cheapest Logitech web camera he could find, took it apart, and removed the IR filter. The next step was to replace that filter with exposed photo negatives which would filter out all visible light allowing the camera to only see the blobs of Evanescent waves escaping from the plexi-glass when fingers touch the surface of the table.
Unfortunately Bridger later discovered that the bandwidth of the USB camera wasn't sufficient enough to provide real time results based on the input. Later in his testing process he had to replace that camera with a FireWire camera which solved his latency issues.
Next he ordered his display, a 19" Acer AL196W and immediately dismantled it, breaking the outer casing in the process. Since all he really needed was the controller and the panel its self it really didn't matter much. Once the backlight is removed from the back of the LCD panel the panel itself is transparent, this is important as it allows the camera mounted behind the display to see the finger input on the top of the table. Unfortunately without a backlight you won't get much of an image on the LCD, so Bridger had to add normal everyday fluorescent lighting to his project so the user could actually see the objects that he was manipulating.
The IR portion of the project didn't go as smoothly. After some frustration with cheap LEDs in his initial try he found some that were triple chip that emit light at 850 nanometers that he thought would do a much better job. After some thinking he and a friend came up with a means for mounting the LEDs inside his box. He took 3/4" x 3/4" boards and routed out a piece that would hold the piece of plexiglass that would sit over the screen and drilled out holes every two inches to hold the IR light emitting diodes. While he recommends that readers don't do the wood working in their bedroom, as the wood chips are hard to get out of the carpet, he almost had bigger problems on his hands:
"My first attempt at designing a circuit to power all of these LEDs was somewhat unfortunate. There was 28 LEDs. I wired them up with 4 parallel circuits of 7 LEDs, along with a resistor to drop the leftover voltage. What I forgot to figure in was the amount of power dissipated by each resistor, which ended up being over 1/2 watt for a 1/4 watt resistor. Needless to say, the resistors started getting pretty hot. I rethought my circuit and re-soldered everything into two circuits of 10 and one of 8. This worked much better."
After building the rest of the box, with a depth of about two feet, Bridger could plug his new table into his MacBook via Firewire and output the video to the table using VGA or DVI. He later covered the entire table in vinyl to give it a more finished look. For the high school student the most difficult part of the hardware phase was building the box to house it all; apparently they don't have shop class in the double wide.
Building a multi-touch system isn't anything new. Many have created similar devices using similar methods. What makes this case different is Bridger's work with the software and libraries available on his operating system of choice Mac OS X. Since the science fair was quickly approaching and his experience with image processing was minimal he set out looking for software that could interpret those blobs of light into information he could use.
With the help of the Natural User Interface (NUI) group he found Pawel Solyga's Google Summer of Code project OpenTouch a platform based upon the mentioned earlier frustrated total internal reflection. Open Touch passes messages via tangible user interface objects (TUIO) the issue was receiving those messages in Objective C. While the project did have examples on how to work with the information in other programing languages there was no library for Objective C. Those libraries were necessary to interpret Open Sound Control information which TUIO is based upon and then is sent over TCP.
After some googling he found an older project:
"I did find one that was released in like 2001 I think, so it was kind of outdated and it used a lot of things that didn't work anymore because they had been changed. So I spent forever trying to figure that one out. All the bits were coming in flipped, so I had to read up on processors which was something I didn't really want to do."
He then used this new information to rewrite a library to interpret TUIO messages in Objective C and then wrote an Objective C application called Corkboard which uses core animation to display xbcd comics on OS X. Inside the application the comics can be manipulated, shrunk, enlarged, rotated, moved, via a user's fingers. He has also done some further work:
"I could show movies on there, I have a little demo of that. and then instantly they're multi-touch so you can rotate or scale them. So what I was hoping by releasing my source was that anyone else could pick this up, the groundwork is already laid and someone could make a multi-touch enable application pretty fast."
How fast? Bridger estimates a company like Delicious Monster, with his code, could make Delicious Library multi-touch enabled in a day or two.
Maxwell has no intent to stop now that science fair season is over. While he isn't looking for funding, or to mass produce his multi-touch device, he is creating a device for the Space Center he started at so if folks would like to help monetarily with that project please contact him. Instead, he wants to rewrite OpenTouch and eliminate the dependence on TCP, write a gesture recognition library, and get one of his displays working in the Space Center. At 17 he has plenty of time.
Currently Bridger is working on porting his Mac game Lumen to the iPhone. He has been contacted and interviewed with Toby Patterson an Apple iPhone frameworks programmer regarding a summer internship, but unfortunately it did not work out because of Apple's busy schedule with so much work being put into WWDC preparation.
So how did Maxwell faire in the land of science fairs? Round one went well enough, his school threw a science fair just for him, and in the words of KRS One he wasn't just number one he was "one, two, three, four, and five." The district fair; however, didn't go so well. After he thought he didn't place he soon after found out that the judges just forgot to announce his name during the competition. At regionals, hosted by Brigham Young University, he finished in the top six and was sent all expenses paid to Atlanta, Georgia for the international fair. While he didn't take first, he did finish fourth in the computer science category. The overall winner? Efficient Hydrogen Production Using Cu-Zn-Al Catalysts Prepared by Homogeneous Precipitation Method. I'll let the Noble Intent folks explain that one.'