Tokyo, Japan - 01 Aug 2011: IBM (NYSE: IBM) today announced that it is helping the National Diet Library of Japan -- the country's only national library -- digitize its literary artifacts on a massive scale to make them widely available and searchable online by all information seekers.

The prototype technology, created by IBM Research, allows full-text digitization of Japanese literature to be quickly realized through expansive recognition of Japanese characters and enabling users to collaboratively review and correct language characters, script and structure. Additionally, the full-text digitization system is designed to promote future international collaborations and standardization of libraries around the world.

“Nearly two decades ago in his book Digital Library, Dr. Makoto Nagao, the director of the National Diet Library, shared his vision that digitized and structured electronic books will dramatically change the role of libraries and the way knowledge will be shared and reused in our society,” said Dr. Hironobu Takagi, who led the development of the prototype technology at IBM Research – Tokyo. “Until now, the breadth of the characters and expressions within the Japanese language had posed a series of challenges to massive digitization. In order to enable this transfer of knowledge from print to online, we realized the need for both machine and human intelligence to understand information in every form.”

Compared to other languages, which rely on just a few dozen alphabetical characters, Japanese is extremely diverse in terms of script. In addition to syllabary characters, hiragana and katakana, Japanese includes about 10,000 kanji characters (including old characters, variants and 2,136 commonly used characters), in addition to ruby (a small Japanese syllabary character reading aid printed right next to a kanji) and mixed vertical and horizontal texts.

Aside from ensuring quality recognition of Japanese characters, IBM researchers aimed to optimize the amount of time needed to review and verify the accuracy of the digitized texts. By introducing unique collaborative tools via crowdsourcing, the technology allows many users to quickly pour through the texts and make corrections at a much higher rate of productivity and efficiency.

“Through collaboration technology and user tools, we now have the potential to populate a global collection of literature and information,” said Dr. Takagi. “From small community libraries to national institutions, people everywhere can leverage this standardized system to help preserve and share their cultural works for years to come.”

The architecture of the full-text digitization prototype system provides the following two key collaborative features:

• Collaborative Correction -- This feature allows simultaneous corrections by multiple users via web browsers and improves the accuracy of optical character recognition (OCR). The collaborative character correction interface displays a group of OCR-scanned characters which were recognized as the same character, allowing reviewers to make a batch correction. The result of the correction work can be fed into the OCR engine as training data, so the system can learn from the mistakes to improve accuracy down the line.
• Collaborative Data Structuring -- This feature is essential to digitizing texts for visually impaired people to read books using a voice browser. Reviewers are able to add structural information as well as correct read-out order, which are both supported by inference engines to reduce workload. For document structuring, the system automatically detects repetitive patterns and suggests similar structures throughout pages. To correct read-out order, the visual editor technology uses smoothly-connecting arrows to show in what order voice browsers would present content, allowing reviewers to go over reading flow at a glance and make necessary edits by simply dragging, dropping and re-arranging the arrows. 

The full-text digitization prototype system was realized based on two streams of technologies. IBM researchers in Tokyo applied an innovative approach called Social Accessibility, which allows large groups of reviewers to work collaboratively via Web browsers regardless of location. Also, the COoperative eNgine for Correction of ExtRacted Text(CONCERT) technology -- developed by IBM Researchers in Haifa, Israel -- was leveraged to significantly improve productivity through the repetition of simple operations.