用Wordsimth可以统计汉语语料吗？

[hd_lwj]

用Wordsmith中的WordList统计汉语文本文件为什么出来的是乱码呢？WordSmith不是支持汉字词语的统计吗？

 

[清风出袖]

http://www.corpus4u.com/forum_view.asp?forum_id=7&view_id=420 has details about the question you asked. tokenized all the Chinese you want to process in wordsmith is probably the way out.

 

[xiaoz]

Version 3 or 4?

 

[xujiajin]

WordSmith version 4 processes unicodified Chinese texts reliably.

 

[hd_lwj]

我用的是经过词性标注的人民日报语料库，是经过分词和词性标注的，想先用CWordList统计出来一个词语频率列表，用的是version 4，结果为乱码

 

[hd_lwj]

是不是因为有词性标注的原因呢

 

[xiaoz]

Use Text Converter in Utilities to convert your data from GB into Unicode first.

 

[hd_lwj]

回复：用Wordsimth可以统计汉语语料吗？

好了，多谢了

 

[xujiajin]

Unicode here, acceptable by WS4, refers specifically to utf-16, not utf-8.