World Wide Web Robot for Extreme Datamining with Swiss-Tx Supercomputers

Roehrl ASA, Frey M, & Roehrl RA (1999). World Wide Web Robot for Extreme Datamining with Swiss-Tx Supercomputers. IIASA Interim Report. IIASA, Laxenburg, Austria: IR-99-020

[img]
Preview
Text
IR-99-020.pdf

Download (461kB) | Preview

Abstract

This paper discusses the software and hardware issues of designing a highly parallel robot for extreme datamining on the Internet. As a sample application, a World Wide Web server count experiment for Switzerland and Thailand is presented. Our platform of choice is the SwissTx, a supercomputer built from commodity components that runs NT and COMPAQ tru64 UNIX. Hardware and software of this machine are discussed and benchmark results presented. They show that NT is a feasible choice even under the given extreme conditions. Using statistical modelling for optimizing the search process, the inevitable bandwidth problem is reduced to some extent to a computation problem. We suggest that our approach to Web robots is a robust bet for a multitude of future Internet applications which might lead to a large-scale and cost-efficient usage of Web robots.

Item Type: Monograph (IIASA Interim Report)
Research Programs: Institute Scholars (INS)
Depositing User: IIASA Import
Date Deposited: 15 Jan 2016 02:11
Last Modified: 18 Nov 2016 12:00
URI: http://pure.iiasa.ac.at/5919

Actions (login required)

View Item View Item

International Institute for Applied Systems Analysis (IIASA)
Schlossplatz 1, A-2361 Laxenburg, Austria
Phone: (+43 2236) 807 0 Fax:(+43 2236) 71 313