An Efficient and Resilient Approach to Filtering and Disseminating Streaming Data Shetal Shah, Shyamshankar Dharmarajan, and Krithi Ramamritham   View Paper (PDF)   Return to Internet/WWW (Session A1) Abstract Many web users monitor dynamic data such as stock prices, real-time sensor data and traffic data for making on-line decisions. Instances of such data can be viewed as data streams. In this paper, we consider techniques for creating a resilient and efficient content distribution network for such dynamically changing streaming data. We address the problem of maintaining the coherency of dynamic data items in a network of repositories: data disseminated to one repository is filtered by that repos- itory and disseminated to repositories dependent on it. Our method is resilient to link failures and repository failures. This resiliency implies that data fidelity is not lost even when the repository from which (or a commu- nication path through which) a user obtains data experi- ences failures. Experimental evaluation, using real world traces of streaming data, demonstrates that (i) the (com- putational and communication)cost of adding this redun- dancy is low, and (ii) surprisingly, in many cases, adding resiliency enhancing features actually improves the fi- delity provided by the system even in cases when there are no failures. To further enhance fidelity, we also pro- pose efficient techniques for filtering data arriving at one repository and for scheduling the dissemination of fil- tered data to another repository. Our results show that the combination of resiliency enhancing and efficiency im- proving techniques in fact help derive the potential that push based systems are said to have in delivering 100% fidelity. Without them, computational and communica- tion delays inherent in dissemination networks can lead to a large fidelity loss even in push based dissemination. ©2004 Association for Computing Machinery