Research Paper | Computer Science & Engineering | India | Volume 3 Issue 7, July 2014
Reducing Duplicate Content Using XXHASH Algorithm
Rahul Mahajan, Dr. S. K. Gupta, Rajeev Bedi
Users of World Wide Web utilize search engines for information retrieval in web as search engines play a vital role in finding information on the web. With the rapid growth of information and the explosion of Web pages from the World Wide Web, it gets harder for search engines to retrieve the information relevant to a user. However, the performance of a web search is greatly affected by flooding of search results with information that is redundant in nature. Removing redundant content is an important data processing operation in search engines and other web applications. The existing architecture of WWW uses URL to identify web pages. A large fraction of the URLs on the web contain duplicate (or near-duplicate) content. Web crawlers rely on URL normalization in order to identify equivalent URLs, which link to the same web pages. Duplicate URLs have brought serious troubles to the whole pipeline of a search engine, from crawling, indexing, to result serving. De-duping URLs is an extremely important problem for search engines, since all the principal functions of a search engine, including crawling, indexing, ranking, and presentation, are adversely impacted by the presence of duplicate URLs. In this we have proposed a new technique for reducing duplicate content during web crawling and saving only unique pages in the database.
Keywords: Duplicate, Duplicate Content, Normalization, Web, Web Crawler
Edition: Volume 3 Issue 7, July 2014
Pages: 610 - 612
How to Cite this Article?
Rahul Mahajan, Dr. S. K. Gupta, Rajeev Bedi, "Reducing Duplicate Content Using XXHASH Algorithm", International Journal of Science and Research (IJSR), https://www.ijsr.net/search_index_results_paperid.php?id=2014951, Volume 3 Issue 7, July 2014, 610 - 612
Similar Articles with Keyword 'Duplicate'
CBCD Methods in Video Copy Detection
Jan Mary Thomas
Enhance QoS of Lossy Wireless Sensor Network by In-Network Data Aggregation
Harshada Kupade, Madhav Ingle
Similar Articles with Keyword 'Normalization'
A Review on Data Mining, Its Applications and Approaches
Anu Verma, Jyoti Arora
Image Normalization Robust using Histogram Equalization and Logarithm Transform Frequency DCT Coefficients for Illumination in Facial Images
Dr. V. S. Manjula
Similar Articles with Keyword 'Web'
Secure Methods for Supplychain Management to Protect from Attacks in Blockchain
B. Ratnakanth, K. Venkata Ramana
Krashi Prabhandak (Agricultural Manager)
Prafful Mundra, A V Pavan Krishna, Swarnalatha P, Venkata Sumanth Kakollu
Similar Articles with Keyword 'Web Crawler'
A Survey on Types of Crawlers and Web Searching Algorithms
T. Yogameera, Dr. D. Shanthi
Web Data Extraction by Using Trinity
Sayali Khodade, Nilav Mukharjee