Downloads: 0 | Views: 62

Research Paper | Computer Science | India | Volume 12 Issue 4, April 2023

AI - Based Solution for Web Crawling

Prashanth Kumar HM | Dr. Subramanya Bhat S

Abstract: Web crawling, also known as web scraping or spidering, is the process of automatically gathering data from the internet. It involves using automated software tools using AI to visit websites, download data like web pages, pdf, videos, metadata, or images. Then store it in a structured format for later use. Web crawlers, also called spiders or bots, follow links from one webpage to another with AI validation. The information gathered by web crawlers can be used for a variety of purposes, including data mining, content aggregation, search engine indexing, market research or Plagiarism detection. Here our crawling is only for plagiarism detection, and our new AI based algorithms help us to do the fastest and most accurate data downloading.

Keywords: Web Crawling, Structured Data, Link Validation, URL, Uniform Resource Locators, Artificial Intelligence

Edition: Volume 12 Issue 4, April 2023,

Pages: 179 - 183