Crawler and Its Linguistic Challenges in the Arabic Language Sites A Case study of Syrian Newspapers.

Crawler, a Web indexing program or an Internet robot/bot (Spetka, 2004), is a software application that runs automated scripts over the Internet. The Web engines use it to update the content and sites via copying all the accessed pages and processing them into indexes so that the users can search much more sufficiently. Crawling is the first stage that downloads Web documents, which the indexer indexes for later use by searching module, with feedback from other backgrounds. This module could also provide on-demand crawling services for search engines. Yet, with the massive amount of data that has been fed on the web, we still encounter some problems and challenges while crawling data. Subsequently, through the wide-open access to all search engines, Arabic content is hitherto scantily accessible. This paper descriptively details the stances and challenges that the Arabic language, the fifth most spoken language, might grapple with while crawling data. [ABSTRACT FROM AUTHOR]