Enterprise Data Warehouse
A near realtime data warehousing system built on serverless technology for a pharma organization including an operational UI.
Situation
The client worked in the healthcare industry. They used a data warehousing system for storing their Pharmaceutical Sales Representatives’ data. The client did not process or refine the collected data, or there was no sorting of the data according to the business/fact/dimension criteria. Additionally, there was no system in place that allowed the company reps to use the data.
Problem Statement
In order to evaluate the data, the previous data warehousing system lacked a suitable user interface (UI). Correct functionalities weren’t utilized in the data warehousing system to deal with validations and standardization. Furthermore, Fact GPO & Dimensions were not properly implemented. A mechanism was required in the new data warehousing system that would provide them with the following three alternatives for resolving the following issues:
- It was hard to validate, modify, and add new and existing data via the UI interface
- There was no precise address standardization method
- Fact, GPO, and Dimensions layers weren’t used.
Approach/Solution
For the new data warehousing system, the AWS Lambda service was used to execute straightforward alerts of incoming files in order to accomplish real-time processing of input data files. An input definition system had also been created, making it simple to add or change an existing input data source. As part of the input definition implementation, the team also leveraged the AWS Glue service and the idea of an external schema. By leveraging the SmartyStreets API for address verification and data collection surrounding a specific address, accurate address standardization was made possible. Using a user interface to edit, update, add, or remove existing data simplified the data validation process. Additionally, the processed data was synchronized with the existing CRM system to make it simple for the marketing and sales teams to check data. The client worked in the healthcare industry. They used a data warehousing system for storing their Pharmaceutical Sales Representatives’ data. The client did not process or refine the collected data, or there was no sorting of the data according to the business/fact/dimension criteria. Additionally, there was no system in place that allowed the company reps to use the data.
Outcome
Once the new data warehousing system was finished, the Incentius team found that the data accuracy had increased and that the data validation had become simple with the new UI. With the assistance of the AWS Lambda service, real-time data processing was accomplished. Since the data warehouse system was used to develop the data needed for the client’s existing CRM system, data synchronization between the two systems was accomplished easily. Sequential lambda functions were used to apply business, fact, and dimensions rules to the incoming data with simplicity.
