从网页中提取文本制作语料库时会发现网页中包含大量重复性内容，如copyright, ads, headers, footers等等。这些显然不是这类语料库使用者所关心的内容。我近期就遇到这个问题。由于建设这类语料库要处理的网页数量往往特别多，手工删除这些并不现实。
推荐使用jusText工具，是用python实现的，有效去除语料中的多余杂质，保持肌肤活力。
http://corpus.tools/wiki/Justext
Quick start
wget -O page.html http://planet.python.org/
justext -s English page.html > cleaned-page.txt
经我亲自试验效果非常好。
This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with financial support from PRESEMT and Lexical Computing Ltd. It also relates to Jan Pomikálek's PhD research.
Justext is licensed under the BSD 3-Clause License
推荐使用jusText工具，是用python实现的，有效去除语料中的多余杂质，保持肌肤活力。
http://corpus.tools/wiki/Justext
Quick start
wget -O page.html http://planet.python.org/
justext -s English page.html > cleaned-page.txt
经我亲自试验效果非常好。
This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with financial support from PRESEMT and Lexical Computing Ltd. It also relates to Jan Pomikálek's PhD research.
Justext is licensed under the BSD 3-Clause License