Supporting e-business research with web crawler methodology

In economic and social sciences it is crucial to test theoretical models against large and reliable databases. The general research challenge is to build up a well-structured database that suits the given research question well and that is cost efficient at the same time. In this paper we focus on crawler programs that proved to be an effective tool for data base building in very different problem settings. We present three structurally different research models where crawler programs can be applied successfully: exploration, classification and time series analysis. In the case of exploration we present findings about the Hungarian web agency industry where no previous statistical data was available about their operations. For classification we show how the top visited Hungarian web domains can be divided into predefined categories of e-business models. In the third research we used a crawler to gather the values of specific pre-defined records containing ticket prices of low cost airlines from one single site. Based on the experiences we highlight some conceptual conclusions and opportunities for crawler based research in e-business.