Character encoding in corpus construction

X

xiaoz

永远的超级管理员
Staff member
Character encoding in corpus construction

This chapter first briefly reviews the history of character encoding. Following from this is a discussion of standard and non-standard native encoding systems, and an evaluation of the efforts to unify these character codes. Then we move on to discuss Unicode as well as various Unicode Transformation Formats (UTFs). As a conclusion, we recommend that Unicode (UTF-8, to be precise) be used in corpus construction.

http://www.lancs.ac.uk/postgrad/xiaoz/papers/character encoding.doc
 
回复：Character encoding in corpus construction

WordSmith 4 似乎只处理UNICODE文本的汉语
文件，UTF-8格式的汉语文本WS4不能处理。对否？
 
Exactly. WS4 is Unicode-based (UTF-16). If your Chinese data is encoded in UTF-8, WS4 will prompt you to convert it into Unicode. If the data in GB2312 or other native encoding, WS4 does not recognizes it and cannot deal it properly. UTF-8 text cannot be dealt with directly by WordSmith 4 reliably.

But it is adviable to backup your data before conversion as WS4 overwrites the data.

[本贴已被 作者 于 2005年07月06日 01时00分53秒 编辑过]​
 
回复：Character encoding in corpus construction

以下是引用 xiaoz 在 2005-7-6 0:50:08 的发言：
Exactly. WS4 is Unicode-based (UTF-16). If your Chinese data is encoded in UTF-8, WS4 will prompt you to convert it into Unicode. If the data in GB2312 or other native encoding, WS4 does not recognizes it and cannot deal it properly. UTF-8 text cannot be dealt with directly by WordSmith 4 reliably.

But it is adviable to backup your data before conversion as WS4 overwrites the data.

[本贴已被 作者 于 2005年07月06日 01时00分53秒 编辑过]​
Click to expand...

Important warning! Thanks.
 
回复：Character encoding in corpus construction

以下是引用 xiaoz 在 2005-7-6 0:50:08 的发言：
Exactly. WS4 is Unicode-based (UTF-16). If your Chinese data is encoded in UTF-8, WS4 will prompt you to convert it into Unicode.
Click to expand...

When does it warn you? When you use the conversion program? Otherwise it doesn't display any warning when you use its tools (concord, wordlist, etc.)

以下是引用 xiaoz 在 2005-7-6 0:50:08 的发言：

But it is adviable to backup your data before conversion as WS4 overwrites the data.



I prefer using the MS Notepad tool to do the conversion.
 
1) After you have chosen texts, click on the button for "test Unicode" (the third from right on the toolbar, shaped like 人). If your corpus is in UTF-8, you will get a warning as in the screen dump.

2005070620362989.jpg


(The status for encoding in the table when you load your corpus: U for Unicode, 8 for UTF-8, a for any other encoding)

2) You can of course do conversions using Notepad etc, but only one file can be converted a time. This approach is not recommended for large corpora. For Chinese data, Scott Piao's MLCT (available at this site) is recommended to first convert GBK or Big-5 to UTF-8 (will not do if you convert directly to UTF-16 as, I think, MLCT and WS4 are using different types of UTF-16 - big-endian vs. little-endian), and then load the UTF-8 data into WS4, allowing WS4 to do the final conversion.
 
Thanks for the useful tips.

PS: These standards never fail to amaze me: just when you think Unicode is good enough, you got UTF-8, 16, 32 ...

[本贴已被 作者 于 2005年07月07日 11时33分01秒 编辑过]​
 
For Chinese corpora, UTF-8 can save disk space as all English letters (used for markup and annotation, for example) are still one byte while all Chinese characters are two bytes. In UTF-16, everything becomes double-byte. UTF-32 is currently used rarely. It is in place to ensure upward compatibility with the 64-bit system.
 
回复：Character encoding in corpus construction

以下是引用 xiaoz 在 2005-7-7 21:06:49 的发言：
For Chinese corpora, UTF-8 can save disk space as all English letters (used for markup and annotation, for example) are still one byte while all Chinese characters are two bytes. In UTF-16, everything becomes double-byte. UTF-32 is currently used rarely. It is in place to ensure upward compatibility with the 64-bit system.
Click to expand...

Understood.
Yes. Any complexity has either motivated by efficiency or stupidity.
 
i also notice this problem, most of the Chinese texts are in ASCI code, and wordsmith4 only recognizes Chinese texts in Unicode, and some other concordance programs recognize different codes, my question is:
how can we swith from one code to another freely? esp. for large texts
 
You must log in or register to reply here.
Back
顶部