1. Effectively and efficiently detect web page duplication
- Author
-
Zhongming Han, Qian Mo, Jianzhi Sun, and Hongzhi Liu
- Subjects
Information retrieval ,Text mining ,Similarity (network science) ,business.industry ,Computer science ,Web page ,The Internet ,Static web page ,Data mining ,business ,computer.software_genre ,computer ,Statistic - Abstract
There are a lot of redundant web pages on Internet. Based on tag statistic and text similarity comparison, we present a novel multilayer framework for detecting duplicated web pages in this paper. We propose two similarity text paragraphs detection algorithms and implement our framework. The experimental results show that our approach achieves high performance, which means that duplicated web pages can be efficiently detected simply by tag statistic and text comparison.
- Published
- 2009