[讨论]WordSmith Tools v4.0处理中文要加空格
- 主题发起人 lngzlz
- 时间
4. 哪些软件可以自动作编码转换（GB/BIG5/UTF-8/UNICODE=UTF-16）？
a) Multilingual Corpus Tool by Scott Piao, 成批转换
http://www.lancs.ac.uk/staff/piaosl/research/download/download.htm
b) WordSmith Tools 4, GB/BIG5 -> UNICODE (UTF-16) 成批转换
c) 南极星NJ Star 文本转换器, 单个转换
http://www.njstar.com
d) b) Chinese Annotation Tool可在线处理简体汉语文本, 单个转换
http://www-rohan.sdsu.edu/~chinese/annotate.html
Perl 版本：http://www.mandarintools.com/segmenter.html
e) MS Word/Notepad, 单个转换
Find more at
http://www.corpus4u.org/showthread.php?t=699
Character encoding in corpus construction
http://www.corpus4u.org/showthread.php?t=416
a) Multilingual Corpus Tool by Scott Piao, 成批转换
http://www.lancs.ac.uk/staff/piaosl/research/download/download.htm
b) WordSmith Tools 4, GB/BIG5 -> UNICODE (UTF-16) 成批转换
c) 南极星NJ Star 文本转换器, 单个转换
http://www.njstar.com
d) b) Chinese Annotation Tool可在线处理简体汉语文本, 单个转换
http://www-rohan.sdsu.edu/~chinese/annotate.html
Perl 版本：http://www.mandarintools.com/segmenter.html
e) MS Word/Notepad, 单个转换
Find more at
http://www.corpus4u.org/showthread.php?t=699
Character encoding in corpus construction
http://www.corpus4u.org/showthread.php?t=416
In WordSmith 4, go to "Utilities - Text converter" in the main menu.
Check Text conversion Activated.
Select the filefolder to be converted and make other adjustments as desired (keep the original data and create an extra coppy in the temp directory?).
In the Conversion type, select "into Unicode based on", and select "Chinese (People's Republic of).
Click on the OK button at the bottom.
Check Text conversion Activated.
Select the filefolder to be converted and make other adjustments as desired (keep the original data and create an extra coppy in the temp directory?).
In the Conversion type, select "into Unicode based on", and select "Chinese (People's Republic of).
Click on the OK button at the bottom.
The best thing to do with Chhinese text is, of course, to tokenise the data. Yet you still need to convert the data into Unicode. Running texts without segmentation will work with Concord, not Wordlist or Keyword. (I only tested running test with Concord.)
The problem you encountered probably has to do with settings. You will need to select language and font properly before loading the texts (in Adjust settings).
The problem you encountered probably has to do with settings. You will need to select language and font properly before loading the texts (in Adjust settings).
The discussions in this thread are related to Wordsmith 4, but version 3.
With version 3, as long as your Chinese data are tokenised, there is no need to convert to Unicode - By the way, if you use "save as" and select "Unicode document" in Notepad, it is Unicode (UTF-16). But you do need a simplied Chinese version of Windows or a language pack. Still only Concord will work, but not Wordlist or Keyword.
In Wordsmith 4, you must convert Chinese data into Unicode (you can use the Text Converter in Utilities of WS4). In this case, even texts not tokenised will work with Concord. But the data must be tokenised if you want to make a wordlist or extract keywords.
With version 3, as long as your Chinese data are tokenised, there is no need to convert to Unicode - By the way, if you use "save as" and select "Unicode document" in Notepad, it is Unicode (UTF-16). But you do need a simplied Chinese version of Windows or a language pack. Still only Concord will work, but not Wordlist or Keyword.
In Wordsmith 4, you must convert Chinese data into Unicode (you can use the Text Converter in Utilities of WS4). In this case, even texts not tokenised will work with Concord. But the data must be tokenised if you want to make a wordlist or extract keywords.
回复：[讨论]WordSmith Tools v4.0处理中文要加空格
The problem you have encountered, I suspect, is most likely that your data have not been converted into Unicode properly. You have to the Text Converted in WST4 to do the conversion (see my earlier replies). Or you can use MLCT to convert GB2312 (GBK) data into UTF-8 (not UTF-16, Mike cannot explain why the UTF-16 data converted using MLCT can be processed by wordsmith 4), and then click on the icon for "test Unicode" when choosing texts.
Here are a few screen dumps that show how WST4 works well with untokenised Unicode Chinese data.
The problem you have encountered, I suspect, is most likely that your data have not been converted into Unicode properly. You have to the Text Converted in WST4 to do the conversion (see my earlier replies). Or you can use MLCT to convert GB2312 (GBK) data into UTF-8 (not UTF-16, Mike cannot explain why the UTF-16 data converted using MLCT can be processed by wordsmith 4), and then click on the icon for "test Unicode" when choosing texts.
Here are a few screen dumps that show how WST4 works well with untokenised Unicode Chinese data.
[本贴已被 作者 于 2005年10月06日 10时40分28秒 编辑过]