World Wide Web Robot for Extreme Datamining with Swiss-Tx Supercomputers

Roehrl, A.S.A., Frey, M., & Roehrl, R.A. (1999). World Wide Web Robot for Extreme Datamining with Swiss-Tx Supercomputers. IIASA Interim Report. IIASA, Laxenburg, Austria: IR-99-020

Preview

Text IR-99-020.pdf Download (461kB) | Preview

Abstract

This paper discusses the software and hardware issues of designing a highly parallel robot for extreme datamining on the Internet. As a sample application, a World Wide Web server count experiment for Switzerland and Thailand is presented. Our platform of choice is the SwissTx, a supercomputer built from commodity components that runs NT and COMPAQ tru64 UNIX. Hardware and software of this machine are discussed and benchmark results presented. They show that NT is a feasible choice even under the given extreme conditions. Using statistical modelling for optimizing the search process, the inevitable bandwidth problem is reduced to some extent to a computation problem. We suggest that our approach to Web robots is a robust bet for a multitude of future Internet applications which might lead to a large-scale and cost-efficient usage of Web robots.

Actions (login required)

View Item