Abstract:
The success of an organization depends on its ability to acquire accurate and timely data about its operation, to manage this data effectively, and to use it to analyze its own activities. The need for powerful and flexible data management has grown much in all fields of life. This Final year project is an industrial-based project assigned by a data warehousing company “TeraData”. In this project main objective was to investigate research and design an Extract Transform Load (ETL) architecture which would segregate ETL process into smaller and more manageable steps.
Main Objectives of the project is to:
Design, engineer and development of an end to end ETL process for a Telecommunication industry as follows:
o Using TeraData as RDBMS for loading data during ETL process.
o Designing physical Data Model for optimal Business Intelligence queries & data loading.
o Engineering a complete ETL process and load source data into a logical data model by following data warehousing principles used during ETL process.
o Capturing changes coming from sources on daily basis and merge them into already existing data warehouse. (Change Data Capture, (CDC))
Developing GUI based Enterprise Data Warehouse Management & Monitoring System (EDWMM).
o EDWMM will monitor all EDW loading jobs which trigger as part of ETL process.
o EDWMM will control the loading of data based on static metadata registered in metadata repository inside Teradata.
o EDWMM will register all data errors in error tables and generate email & SMS alerts for Database Administrators to take action.
o EDWMM will be able to schedule ETL jobs on a fixed time & recurring basis.
The primary reason behind developing this project is the mutual interest of both the group members in data Warehousing. Secondly inspiration by the reputation and performance of TeraData. So, all we wanted was to get a data Warehousing project from TeraData which was more like a dream come true.
