Data Quality for Migrating to a New System

Improving data quality through automated information cleaning

We work with a global leader in transactional solutions in the field of vouchers and prepaid services cards for businesses and enterprises. It offers solutions for 50 million users and 2 million affiliated merchants in 46 countries through 850,000 corporate customers. Below, we will discuss the Case of Success Data Quality for Data Migration.

Challenges for the company

This company was in the process of implementing a new Core Business System for all of LatAm, which was managed from Mexico. One of the biggest challenges they encountered in the project was migrating large volumes of data to this new platform.

They assigned an internal team to perform this task; however, they realized that the existing data required thorough cleaning to perform a successful migration. Additionally, the volume of data was significantly large, which made it evident the need to hire a provider specializing in migrating and managing large volumes of data.

Definition of the project

When analyzing the need for data migration from multiple domains, in this case customers and affiliates, we proposed a data quality service with multiple iterations and testing, that would align with the windows of the corresponding areas and the release dates of the project.

Data profiling

Once we started the project, we faced several limitations. The business team could provide very little time to support the IT department in terms of defining requirements, needs, and rules for data cleansing. However, our experience and expertise in managing data quality projects allowed us to identify other data sources that also needed to be migrated and that had not been considered, which added additional value to both the IT department and the Project Management team.

We conducted an exhaustive profiling of the existing data and discovered a series of formatting errors, such as incorrect dates, and content errors, such as invalid characters and missing information. Using this profiling, we defined business rules with the client to automate the data quality processes. The automation enabled a faster and more accurate cleaning of the information, and we were able to reuse already developed components, which significantly reduced the time required for data cleaning.

Since the volume of data was variable throughout the project, we suggested a change in the initial strategy. By switching from a closed-loop project to a data quality factory approach, the client was able to make on-demand data cleaning payments, which represented greater efficiency and cost savings. This allowed us to manage the volatility in the data volumes and ensure a continuous and efficient service.

Benefits

Thanks to this project, we were able to improve the quality of the data by 90%. The automation of cleaning processes not only allowed us to save time, but also reduced costs by adopting a factory model. 

In addition, the data cleaning development was designed in a modular and reusable way, which eliminated the dependence on an external provider and allowed the customer to perform multiple executions without additional cost.