Back to Search Start Over

The Research of Web Page De-duplication Based on Web Pages Reshipment Statement

The Research of Web Page De-duplication Based on Web Pages Reshipment Statement

Authors :
Dong-sheng Liu
Min-yan Wang
Source :
DBTA
Publication Year :
2009
Publisher :
IEEE, 2009.

Abstract

Web page de-duplication module is an important part of search engine system, which can improve its performance and quality with filtering the web pages downloaded by crawler system of search engine and eliminating the duplicated web pages. This paper from the source of duplicated web pages--reshipment proposes a web page de-duplication method that the information including original websites and web titles are extracted to eliminate duplicated web pages based on feature codes. Experiments show that this method can achieve satisfactory results in eliminating large-scale duplicated web pages.

Details

Database :
OpenAIRE
Journal :
2009 First International Workshop on Database Technology and Applications
Accession number :
edsair.doi...........c443fe150421a2490815db0bd516def5
Full Text :
https://doi.org/10.1109/dbta.2009.64