BEIJING, April. 7 --If Wang Jian had his way, everything would be digital.
"I hate printers, because they turn digital files into printouts," he said, holding a rectangular, silvery pen slightly larger than the size of an ordinary one at Microsoft Research Asia in Beijing.
Fortyish and lanky, the computer scientist, head of the multimodal user interface group, specializes in inventing new computer interfaces to bridge the gap between printouts and digital files.
What he did was to change the printed papers into a part of the computer's hard disk.
Zhang Chunhui, one of Wang's research group assistants, scribbled some corrections on a paper document. But this is no ordinary pen. A few seconds later, his comments appear on a nearby computer screen, superimposed on the electronic version of the document in the exact spot where he wrote on the paper.
Then Zhang drew several musical staves on a piece of paper and wrote down some notes. Simultaneously, what he wrote appeared in a document on the computer screen and the musical staves could be played and these emerged as a piece of nursery rhyme.
The magic pen could transform the way people interact with computers, said Wang.
Unlike gizmos that write on computer displays or special pads of paper, Wang's invention uses regular ink, works with regular paper, and lets users combine handwritten text and diagrams with digital content from reports, magazines, and Web pages.
An executive on a plane trip, for instance, could mark up a paper copy of a report and later transfer the changes to the file on his or her computer automatically.
Concern over a world full of stacks of paper sourced from dwindling woods and forests, prompted Wang and his colleagues to embark on the pen project in 1999.
A professor of engineering psychology at Zhejiang University in Hangzhou, Wang made his name in human-computer interfaces and virtual-reality systems.
In 1998, at the height of his academic career, he received a cryptic e-mail from Li Kaifu, a renowned researcher who was just starting up Microsoft's Beijing laboratory. Li suggested they meet but was too busy to explain why.
"I don't know how he got my name," said Wang, "and he didn't explain who he was."
Intrigued, Wang decided to visit Beijing. He quickly discovered that he and Li shared the ambition to create a user interface for computers based on handwriting.
The pen project was inspired in part by Wang's desire to enable mobile computers to handle handwritten Asian languages. But from his academic research on 3-D interfaces, he had learned that if technology is not designed to be practical and appealing to a variety of users, it will not be widely adopted.
"We wanted to make a breakthrough, not improvements," he said.
With that motivation, Wang left Zhejiang University and joined the Beijing lab in the autumn of 1999.
The key idea, advocated by Wang, was that a document could be kept digital even when printed, with the right kind of pen interface and software.
It took four years to make it work. After a few false starts, taking advantage of advances in computer vision algorithms Wang's team based the pen's sensing on a simple digital camera.
The researchers' first challenge was to find a way to determine the pen's position on the page. Their solution involves special software that puts a barely visible background pattern, like a watermark, on standard copy paper as a document is printed.
That enables the computer to figure out not only exactly where the pen is relative to the document, but also which document is being modified, because each page has a unique code.
A pressure sensor in the pen triggers a tiny embedded camera, which snaps pictures of the user's writing. The images are stored in the pen on a memory chip like those found in digital cameras. When brought within a few metres of a PC or laptop that has the proper software installed, the pen transmits the images wirelessly using a Bluetooth connection.
However, interpreting those images and incorporating them into digital files turned out to be a tougher problem.
The trick, Wang explains, is getting the computer to recognize different types of writing and drawings -- to know what's a box, what's a sentence, and what's a doodle -- just from a series of photos.
First, computer vision algorithms classify sequences of marks as words, diagrams, or shapes, all of which can be manipulated. Then character recognition software, the subject of years of intensive research at Microsoft and elsewhere, makes sense of the handwritten text.
On the computer screen, the user's marks show up as handwriting embedded in the document. Software tools can then convert the writing into typed text and rendered graphics. With these tools, the user can manipulate, say, boxes and text from a hand-drawn flow chart.
In addition to enabling users to import and manipulate handwritten text, tables, and charts, said Wang, the pen will allow multiple collaborators to make comments on separate printouts of a document. The computer can then integrate them all into the same file.
ˇˇˇˇFor computer users, there are many situations where fiddling with a keyboard is challenging, such as attending a meeting, working out of doors, and sitting in a cramped aeroplane seat.
Now smart digital ink technology enables them to work in ink and write in their own handwriting.
Holding a tablet PC and a pen in his hands, Li Yantao, a researcher in the multimodal user interface group, could do whatever he wants on the screen, such as taking notes, inserting a sketch or drawing, and jotting down a chart. What is more, he can freely edit any content he writes.
This process, called inking, enables users to add "digital ink" to the screen which appears as natural-looking handwriting.
"The notes people have made generally have a very complex structure. One of the core techniques with smart digital ink is its ability to make a distinction between handwritten words, charts, and drawings," said Li.
The digitized handwriting can be converted to standard text, or it can remain as handwritten text.
Now more and more people are using Internet cameras to have a chat with others. But scientists are exploring a more interactive way of communicating with computers through a connected digital camera.
Zhang Weiwei, a researcher in the visual computing group, was standing attentively before a computer screen a meter away, with his two fists punching the air.
He was playing a new computer game they had just developed. In the game, he was a diver encountering repeated shark attacks. His mission was to fight against them and also replenish his oxygen.
The camera was capable of recognizing Zhang's face, which appeared a mask on the screen and when he punched out in the right direction, he scored a shark kill or gained oxygen.
The seemingly simple game involves complex face recognition technology.
Zhang and his research team tried to develop algorithms and technology for real-time, automated and highly accurate face detection, tracking, and recognition under variations of light, pose and expression.
Without the keyboard and mouse, the computer could also understand the users intentions by analyzing their body movement through the cameras.
"The new camera-based interaction with computers is more direct, natural and intimate," said Zhang.
(Source: China Daily)