June 10, 2005 @ 10:01AM - posted by Jeff Smykil
Part the I. Interview with Larry Yaeger
In the Loop: interview with Larry Yaeger (Part I)
Recently I had the oppurtunity to ask Larry Yaeger a few questions, this is part one of a three-part series where we discuss handwriting technology, the future of Inkwell and the original Newton project. In case you're in the dark, here is a brief bio for Larry—all a part of keeping you In The Loop!
Infinite Loop: Working on the Newton project, did you think it was going to change the world?
Larry Yaeger: Perhaps a little. But my perspective on Newton is different than most people associated with it. I'd met Steve Sakoman1 and talked with him about his vision for the Newton — which included multiple instantiations from a very small form factor (today's "PDA") to laptop-sized (a "slate") to desktop-sized to wall-sized, all sharing information seemlessly and wirelessly. It was a neat vision. However, the project lingered and lingered and never produced anything viable, for quite a few years.
Larry Yaeger has used computers to solve a wide variety of problems throughout his career. Having studied aerospace engineering with a focus on computers, he carried out pioneering computational fluid dynamic flow studies over the space shuttle and submarines.
As Director of Software Development at Digital Productions, he used a Cray X-MP supercomputer to generate the special effects for Hollywood films The Last Starfighter, 2010, and Labyrinth, as well a number of Clio Award-winning television commercials. While with Alan Kay's Vivarium Program at Apple Computer, he designed and programmed a computer "voice" for Koko the gorilla, helped introduce Macintoshes into routine production on Star Trek: The Next Generation, and created a widely respected Artificial Life computational ecology ("Polyworld") that evolves neural architectures resulting from the mutation and recombination of genetic codes, via behavior-based, sexual reproduction of artificial organisms.
He also coauthored possibly the first book+CD-ROM title, the award-winning Visualization of Natural Phenomena. Also at Apple, in the Advanced Technology Group, he was Technical Lead in the development of the neural network-based handprint recognition system, the world's first genuinely usable hadwriting recognition system, showcased in second generation Newton PDAs and Mac OS X's "Inkwell." He currently resides in scenic Beanblossom, Indiana, and teaches and performs research in Artificial Life at Indiana University.
It's been so long, now, that I can't quite piece together the exact chronology, but I think it was before I even began working on handwriting recognition, or perhaps only slightly after, that I got invited to a meeting at which the Paragraph2 guys were demoing their technology (that ultimately ended up in the first generation Newton). The Newton project was now under Larry Tesler3 , and he led this meeting. I got a chance to use their handwriting recognition. It got 1 out of 10 things I wrote correct. It was terrible. The experience was the same for pretty much everyone. Yet Tesler beamed. I never understood it. Later I learned he'd simply become fascinated by the technology, and its potential, and I guess that blinded him to the realities of the technology at the time.
(To be fair, the debacle of the first-generation Newton wasn't entirely ParaGraph's fault. There again, Tesler and some of the highest level decisions put them into a situation from which it was impossible for anything good to come. But that's another story.)
I was in ATG4 , not the Newton group, at the time. I began working on handwriting recognition. Much time passed. I had some conversations with Newton engineers that were intriguing, and suggested possibilities of us working together. But handwriting recognition is a hard problem, and we were just getting going. It took us years to produce a viable technology. Meanwhile Newton was ramping up and even shipped its first-generation project. I don't think we ever seriously contemplated being part of that first generation, as we knew we weren't ready. Of course, neither was ParaGraph, as history all too starkly made clear.
So while I thought the original, sweeping Newton concept was exciting, and even the reduced vision of what first shipped was intriguing, I had my doubts about whether it could succeed. I also, at various points and with others, lobbied for a smaller form factor, but never seemed to have any impact. (Very, very late in the game I got the then-CEO, Shane Robison5 , to agree to investigate what we now call a palm-sized device, but it was too late, as Newton was soon to be reeled back into Apple and killed off.)
Still, yes, there were so many things done right about the Newton, both hardware and software, that it seemed wonderfully, radically different, in all the right ways. (Except handwriting.) The concept of the data soup, the UI, nicely architected system software, the portability... It really was a brilliant design, in many ways. In fact, if they had left handwriting recognition out entirely, forcing people to use the built-in, on-screen keyboard, I retroactively predict the Newton would have been a success from day one. As it was, the handwriting recognition was so bad, and so touted as the one true input method for the device, that it was very nearly doomed to failure from day one.
Our internal handwriting recognition work didn't get into the Newton until much later, with the release of Newton 2.0 and the MP130 (though I believe there was a mid-generation release of MP120s with Newton 2.0 and our software, but I'm not 100% certain of that). We kind of thought we might have "saved the Newton", and I still think the company and product could have been rescued, but that's yet another story.
2. ParaGraph was responsible for the first generation hand writing recognition in the NewtonOS. They later developed a cursive recognition system for the Newton 2.x a system the later liscenced to Microsoft who now uses it in their Windows XP Tablet PC Edition. The software is still available under the name Callligrapher. ParaGraph is now PhatWare.
3. Larry Tesler was the original head of the Lisa group at Apple Computer. He later took charge of the Newton Group after Steve Sakoman left Apple. Tesler later took over the Advanced Technology Group and then later killed the group as Chief Scientist at Apple. Since Apple he has worked at Amazon.com as the VP of Shopping Experience and Design, and at Yahoo! Research Labs.
4. ATG or the Advanced Technology Group briefly became Apple Research Laboratories before being disbanded by Tesler in 1987. ATG was at least partially responsible for such innovations as "Toby's Frame Buffer", QuicTime, QuickTime VR, QuickDraw3D, speech recognition and synthesis software, handwriting recognition software (later became InkWell), "v-twin" that became Apple's 'core document search technology', an Ensoniq chip for the Mac, the Midi Manager and a DSP chip for the Mac among other things.
5. Shane Robinson is currently the Executive Vice President and Chief Strategy and Technology Officer at Hewlett Packard. Robinson was briefly the manager of the Newton Group before it was killed by Apple.
IL: Did you laugh the first time you saw the infamous Newton spoof on the Simpsons, where Martin writes "Beat up Martin" into his message pad and it comes out as "Eat up Martha"?
LY: Oh yes! I used to never miss an episode of the Simpsons, and was heartily bemused, if also a bit saddened, when I saw that. And even earlier, I think, I was a huge Doonesbury/Gary Trudeau fan, and laughed and cringed at his "egg freckles" jab at the Newton. In fact, once we started seriously working on a Newton version of our recognizer, I kept a photocopy of the Doonesbury panels lampooning the Newton on the wall by my desk, as a reminder that we had to do better.
IL: Was it an involved process porting the Newtons handwriting technology to OS X?
Yes it was. Especially since we started a port to OS 9 first. Brought it to an official alpha release (and a very good, unusually stable alpha, with all known bugs of any significance fixed), and within a couple of days, certainly less than a week, got the official word from Marketing that there would be "no new features on OS 9." So it was back to the drawing board, as very little of the approach or the code for our OS 9 approach could work in OS X.
OS X, then, involved a complete redesign and a lot of new code, but I'm happy with the event-based approach we used to let both system and apps intercept the data handling at any level... points, strokes, words, phrases, with phrase termination determined in a variety of ways. It turned out very flexible, but with a very compact API.
Tune in tomorrow for Part II of the three part series!
