International Journal of Science and Research (IJSR)

International Journal of Science and Research (IJSR)
Call for Papers | Fully Refereed | Open Access | Double Blind Peer Reviewed

ISSN: 2319-7064


Downloads: 0 | Views: 67

Research Paper | Computer Science | India | Volume 12 Issue 4, April 2023 | Rating: 4.3 / 10


AI - Based Solution for Web Crawling

Prashanth Kumar HM | Dr. Subramanya Bhat S


Abstract: Web crawling, also known as web scraping or spidering, is the process of automatically gathering data from the internet. It involves using automated software tools using AI to visit websites, download data like web pages, pdf, videos, metadata, or images. Then store it in a structured format for later use. Web crawlers, also called spiders or bots, follow links from one webpage to another with AI validation. The information gathered by web crawlers can be used for a variety of purposes, including data mining, content aggregation, search engine indexing, market research or Plagiarism detection. Here our crawling is only for plagiarism detection, and our new AI based algorithms help us to do the fastest and most accurate data downloading.


Keywords: Web Crawling, Structured Data, Link Validation, URL, Uniform Resource Locators, Artificial Intelligence


Edition: Volume 12 Issue 4, April 2023,


Pages: 179 - 183


How to Download this Article?

Type Your Valid Email Address below to Receive the Article PDF Link


Verification Code will appear in 2 Seconds ... Wait

Top