Unicode Project

Goal

History

Methodology

Study and Findings

Tables from Innovative

One Big5 and two utf8 tables are received from Innovative. All the analyses of UTF-8 mappings are based on UTF8.new. (Since more sites obtained the Big5 and UTF8, only UTF8.new is attached here for reference.)

• BIG5 (current)
• UTF8 (current)
• UTF8.new (for Phase 3)

UTF-8

Doubtful cases are extracted from utf8.new and re-mapped by cross-checking with the Unicode standard. A revised UTF8 (UTF8.rev) is created. From UTF8.rev, many-to-one pairs are extracted and peferred positions are determined. To simplify the remapping, word frequency is not taken into account. Selection of preferred positions follows Unicode standard. That is, whenever possible, the EACC code chosen in Unicode standard is chosen as the preferred code.

For output, three mapping tables are generated: a many-to-one re-mapping for data conversion; a pure CCCII re-mapping for data conversion; and a new one-to-one mapping table to replace existing UTF8.

Study

• Cases updated by CityU (57 cases)
• Revised UTF8 (23666 entries) -- revised from utf8.new with doubtful cases re-mapped
• Revised UTF8 Many-to-one (1150 cases)

Output

• Revised UTF8 Many-to-one Re-map (1182 pairs) -- used for data conversion
• Revised UTF8 Pure CCCII (6888 pairs) -- re-mapped to placeholders; used for data conversion
• Revised UTF8 One-to-one (15366 EACC) -- proposed to III to replace UTF8 mapping table

Big5

The BIG5 table from III does not contain Unicode information. Unicodes are extracted from UTF8.rev by mapping the CCCII/EACC. If not found, they are extracted from the Unicode site by mapping the Big5. With unicodes exist, doubtful cases could be easier to identify. They are re-ordered, added or re-mapped. A revised BIG5 (Big5.rev) is created.

There are two ways to define many-to-one pairs in Big5: have the same Big5 and the same Unicode; or have the same Big5 only. The former set is safe for use in data conversion. For the later set, multiple entries of EACC are studied. 181 pairs of this kind are identified. Most have the same low-order bytes. Our analysis shows that only one case is dubious in meaning. The rest are included in data conversion.

The outputs of Big5 are similar to those of UTF8: two many-to-one re-mappings for data conversion; a pure CCCII re-mapping for data conversion; and a new one-to-one mapping table to replace existing Big5.

Study

• Cases updated by CityU (102 cases)
• Revised Big5 (14807 entries)
• Revised Big5 Many-to-one -- same Big5, same Unicode (394 cases)
• Revised Big5 Many-to-one Re-map (399 pairs)
• Revised Big5 Many-to-one -- same Big5, different Unicode -- all EACC (181 cases)

Output

• Revised Big5 Many-to-one Re-map (same Big5, same Unicode) (not found in utf8.rev) (100 pairs) -- used for data conversion
• Revised Big5 Many-to-one Re-map (same Big5, different Unicode) (all EACC) (187 pairs) -- used for data conversion
• Revised Big5 Pure CCCII (4669 pairs) -- re-mapped to placeholders; used for data conversion
• Revised Big5 One-to-one (8952 EACC) -- proposed to III to replace Big5 mapping table

Final tables to Innovative

The following are for patching (data conversion) only

• Many-to-one Re-map (Big5 + Unicode) (total 1469 pairs) -- for data conversion
• Pure CCCII Re-map (Big5 + Unicode) (total 8202 pairs) -- for data conversion

The following are to replace current mapping tables

• Revised UTF8 One-to-one (15366 EACC)
• Revised Big5 One-to-one (8952 EACC)