An Improved Approach in Data Warehousing ETLM Process for Detection of Changes in Source Data Rosana L. de B. A. Rocha, Leonardo Figueiredo Cardoso, and Jano Moreira de Souza   View Paper (PDF)   Return to Data Warehousing Techniques and Applications Abstract The ETLM process (extraction, transformation, load, and materialization) comprises the basis for data acquisition and organization of a data warehouse (DW) environment. In the DW area, a well-known issue consists of the detection of changes in source data. This problem arises when the data source does not possess or does not provide the information about the changes that have occurred. The main scenarios in which this problem is found may be characterized by: (i) the impossibility to instrument the DBMS (triggers, transaction log, stored procedures, replication, materialized views, old and new versions of data, etc), due to security policies, data property or performance issues; (ii) the lack of instrumentation resources on the DBMS; (iii) the use of legacy technologies such as file systems or semi-structured data; (iv) the existence of application proprietary databases and ERP systems. In order to deal with this issue, controls have to be implemented to enable the detection of these changes and to reflect them into the DW environment. In this paper, a framework is proposed for the organization of the DW environment, and some of the main issues involved in the ETLM process are commented on. Having this framework as a basis, the main approaches to solve the detection of changes in source data are described. Our technique, using signatures to mark off and detect changes is presented. This technique was derived from the comparison of database snapshots approach: the technique is simple and can be applied to all four scenarios above. The paper also presents a case study in the DW project developed for Rio Sul Airlines, a regional aviation company owned by the Brazilian Varig group. ©2004 Association for Computing Machinery