A Corpus Worker`s Toolkit：语料库工具箱-0908 更新
- 主题发起人 动态语法
- 时间
回复：A Corpus Worker`s Toolkit：语料库工具箱
J. Clear says that:
the null hypothesis,
(f(post) * span ) * relative_freq(the)
which is
(2579 * 8) * (1 / 20) = 20632 / 20 = 1031
And in calculating both MI/T-Score, the notion of span is used as a
variable. My question (and confusion) is, why choosing 8, why not
other numbers? is there an optimal number to use?
[/quote]
This is Jen Clear's reply to the inquiry
The decision to use 4 left and 4 right (giving a span of 8) was based
on work done at Birmingham University in the 1970s by Prof. John
Sinclair (using a rather small computer corpus of only a few hundred thousand
words!) which led him to conclude that the "influence" of a lexical
item on its surrounding words dropped quite sharply beyond 4 words in
both directions, but within the 4:4 span the level of "influence" was
not significantly different whichever position was selected. Based on
the data obtained from this preliminary corpus study, the Cobuild
project used 4:4 as a standard span for almost all its collocational
work.
Of course, MI can be calculated for any two lexical items separated by
any number of intervening words, and Ken Church demonstrated in the
mid-1980s that statistically significant (*and* interesting!)
collocations can be calculated over a span of 100 or 200 words.
Best wishes
Jem Clear
29 School Road, Moseley, Birmingham, B13 9TF, UK
J. Clear says that:
the null hypothesis,
(f(post) * span ) * relative_freq(the)
which is
(2579 * 8) * (1 / 20) = 20632 / 20 = 1031
And in calculating both MI/T-Score, the notion of span is used as a
variable. My question (and confusion) is, why choosing 8, why not
other numbers? is there an optimal number to use?
[/quote]
This is Jen Clear's reply to the inquiry
The decision to use 4 left and 4 right (giving a span of 8) was based
on work done at Birmingham University in the 1970s by Prof. John
Sinclair (using a rather small computer corpus of only a few hundred thousand
words!) which led him to conclude that the "influence" of a lexical
item on its surrounding words dropped quite sharply beyond 4 words in
both directions, but within the 4:4 span the level of "influence" was
not significantly different whichever position was selected. Based on
the data obtained from this preliminary corpus study, the Cobuild
project used 4:4 as a standard span for almost all its collocational
work.
Of course, MI can be calculated for any two lexical items separated by
any number of intervening words, and Ken Church demonstrated in the
mid-1980s that statistically significant (*and* interesting!)
collocations can be calculated over a span of 100 or 200 words.
Best wishes
Jem Clear
29 School Road, Moseley, Birmingham, B13 9TF, UK
回复：A Corpus Worker`s Toolkit：语料库工具箱
ACWT Updates!
-Updated August 18, 2005:
* Added NEUCSP 东北大学自然语言实验室汉语分词器 & ICTCLAS 中科院计算所词法分析系统
to the TxtUtils group.
* Corrected some user guide inaccuracies.
* Added links to the relevant programs referenced in the clips.
ReadMe portions about the additions:
6) NEUCSP 东北大学自然语言实验室汉语分词器 can be downloaded from
http://www.nlplab.cn/cipsdk.html
Install the program to directory
where neucsp.exe and all other system files should be stored.
This program provides Parts of Speech (POS) tagged output for the currently
open file. (In a Windows-DOS console environment, which is not the case here,
it can also handle multiple files.)
7) ICTCLAS 中科院计算所词法分析系统 can be downloaded from
http://www.nlp.org.cn/categories/default.php?cat_id=12
Install the program to C:\ictclas, where ictclas.exe can be found. There should
be a subdirectory called C:\ictclas\data, where all other system files should be
stored.
___________________________________
For the latest information about ACWT, see page 1.
ACWT Updates!
-Updated August 18, 2005:
* Added NEUCSP 东北大学自然语言实验室汉语分词器 & ICTCLAS 中科院计算所词法分析系统
to the TxtUtils group.
* Corrected some user guide inaccuracies.
* Added links to the relevant programs referenced in the clips.
ReadMe portions about the additions:
6) NEUCSP 东北大学自然语言实验室汉语分词器 can be downloaded from
http://www.nlplab.cn/cipsdk.html
Install the program to directory
where neucsp.exe and all other system files should be stored.
This program provides Parts of Speech (POS) tagged output for the currently
open file. (In a Windows-DOS console environment, which is not the case here,
it can also handle multiple files.)
7) ICTCLAS 中科院计算所词法分析系统 can be downloaded from
http://www.nlp.org.cn/categories/default.php?cat_id=12
Install the program to C:\ictclas, where ictclas.exe can be found. There should
be a subdirectory called C:\ictclas\data, where all other system files should be
stored.
___________________________________
For the latest information about ACWT, see page 1.
回复：A Corpus Worker`s Toolkit：语料库工具箱
说明：因为文件大小超限，我无法上传这两个相关程序文件。用户可以自行免费下载：
1) NEUCSP 东北大学自然语言实验室汉语分词器
http://www.nlplab.cn/cipsdk.html
2) ICTCLAS 中科院计算所词法分析系统 （本站有），也可以下载：
http://www.nlp.org.cn/categories/default.php?cat_id=12
说明：因为文件大小超限，我无法上传这两个相关程序文件。用户可以自行免费下载：
1) NEUCSP 东北大学自然语言实验室汉语分词器
http://www.nlplab.cn/cipsdk.html
2) ICTCLAS 中科院计算所词法分析系统 （本站有），也可以下载：
http://www.nlp.org.cn/categories/default.php?cat_id=12
http://www.nlplab.com. is a dead link.
回复：A Corpus Worker`s Toolkit：语料库工具箱 -已更新
Try this one. I'll update the files.
http://www.nlplab.cn/cipsdk.html
Try this one. I'll update the files.
http://www.nlplab.cn/cipsdk.html
回复：A Corpus Worker`s Toolkit：语料库工具箱-0819更新
Oops, thanks for pointing it out. It's helpful to get all the feedbacks.
By the way, if anyone (not necessarily Dr Xu, who has done so much
already) is interested in translating into Chinese the
Readme file (essentially my 'user guide'), feel free to contact me.
It would help to make the Toolkit accessible to more users. Thanks.
Oops, thanks for pointing it out. It's helpful to get all the feedbacks.
By the way, if anyone (not necessarily Dr Xu, who has done so much
already) is interested in translating into Chinese the
Readme file (essentially my 'user guide'), feel free to contact me.
It would help to make the Toolkit accessible to more users. Thanks.
回复：A Corpus Worker`s Toolkit：语料库工具箱-0819更新
If you want to look at everything on one page, go to:
http://www.humnet.ucla.edu/alc/chinese/ACWT/ACWT.htm
If you want to look at everything on one page, go to:
http://www.humnet.ucla.edu/alc/chinese/ACWT/ACWT.htm
每天翻一点
先翻了一点贴出来，以免造成重复劳动。请先不要对我的翻译进行修改润色，我会在都翻译完后统一校对修改的。当然欢迎纠错。
7. History
-2005年8月18日最后更新
* 在文本处理单元（TxtUtils group）中增加了中国东北大学自然语言实验室汉语分词工具NEUCSP和中科院计算所词法分析系统ICTCLAS。
* 修正了原使用说明中的个别错漏。
* 增加了“新增模块”中应用到的相关软件的链接。
-工具箱的首次发布时间：2005年8月15日。
- 1998年秋于纽约的Ithaca开始收集“模块”。
陶红印
Email: ht_ling@sbcglobal.net.
本readme文件于2005年8月18日最后更新。
先翻了一点贴出来，以免造成重复劳动。请先不要对我的翻译进行修改润色，我会在都翻译完后统一校对修改的。当然欢迎纠错。
7. History
-2005年8月18日最后更新
* 在文本处理单元（TxtUtils group）中增加了中国东北大学自然语言实验室汉语分词工具NEUCSP和中科院计算所词法分析系统ICTCLAS。
* 修正了原使用说明中的个别错漏。
* 增加了“新增模块”中应用到的相关软件的链接。
-工具箱的首次发布时间：2005年8月15日。
- 1998年秋于纽约的Ithaca开始收集“模块”。
陶红印
Email: ht_ling@sbcglobal.net.
本readme文件于2005年8月18日最后更新。
majorlv511
高级会员
非常非常感谢这么细心的图示说明