World Wide Web Robot for Extreme Datamining with Swiss-Tx Supercomputers

Roehrl, A.S.A., Frey, M., & Roehrl, R.A. (1999). World Wide Web Robot for Extreme Datamining with Swiss-Tx Supercomputers. IIASA Interim Report. IIASA, Laxenburg, Austria: IR-99-020

[thumbnail of IR-99-020.pdf]

Download (461kB) | Preview


This paper discusses the software and hardware issues of designing a highly parallel robot for extreme datamining on the Internet. As a sample application, a World Wide Web server count experiment for Switzerland and Thailand is presented. Our platform of choice is the SwissTx, a supercomputer built from commodity components that runs NT and COMPAQ tru64 UNIX. Hardware and software of this machine are discussed and benchmark results presented. They show that NT is a feasible choice even under the given extreme conditions. Using statistical modelling for optimizing the search process, the inevitable bandwidth problem is reduced to some extent to a computation problem. We suggest that our approach to Web robots is a robust bet for a multitude of future Internet applications which might lead to a large-scale and cost-efficient usage of Web robots.

Item Type: Monograph (IIASA Interim Report)
Research Programs: Institute Scholars (INS)
Depositing User: IIASA Import
Date Deposited: 15 Jan 2016 02:11
Last Modified: 27 Aug 2021 17:16

Actions (login required)

View Item View Item