Implementation of the Web Scraping as Extract-Transform-Load (ETL) Module in the Data Warehouse Simulator

Abstract

Data and information are the main assets in an organization. Especially when the data is very fast, it causes the demand for information to also increase. This phenomenon must be anticipated by the organization, because by paying attention to proper data growth it can increase the organization's profits. However, it takes a different architecture from conventional databases to store this kind of data. This storage architecture is called the Data Warehouse. One of the important Data Warehouse components is the Extract Transform Load (ETL) section. The purpose of ETL is to collect, filter, process and combine relevant data from various sources for storage into a data warehouse. In this paper, we propose a simple ETL model that uses web scraping technique for data retrieval. Web scraping are techniques that have been used to collect data from web sites. Its reliability in data collection, as well as its accuracy in sorting data makes it the right model for the ETL process. However, there are still some adjustments that must be made so that the desired data can be obtained. Among other things, the accuracy in sorting HTML elements and knowledge of finding the exact location of the desired data.