Our federal agency customer was faced with modernizing their application and data enviroments while having to maintain upwards of 800 database instances across multiple open source and legacy database technologies.
As part of the customer’s data and application modernization undertaking, its business lines mandated it modernize its mainframe-based systems. The scope included not only legacy data for these systems, it, the legacy data also was maintained in its data centers in other relational database technologies.
The goal was to provide rapid data migration pipelines for bulk as well as CDC based data priming of Cloud Database Services (AWS RDS) for consumption by the modernized applications. The pipelines also needed to be low code for easy maintenance while incorporating transformation logic for data quality and validation of time and date while using business logic to deliver it into a modernized set of data models.
Data Migration Service Solution
To solve this challenge, Karsun developed our Data Migration Service (DMS). This open-source solution is lightweight and flexible enough to enable immediate synchronization needs while supporting future construction as systems are modernized.
The solution implemented uses pipeline automation and leveraging the AWS cloud. This service has the added benefit of having a zero-client footprint which eliminates the need for heavy development client maintenance. The system has APIs for further integration, thereby future-proofing the service and supporting any desired DataOps workflows and methods desired. Utilizing our automation pipelines and its orchestration tools, we configure, deploy, test and promote our pipelines in a heavily automated manner.
Utilizing open-source libraries and configuration artifacts, we automated the enforcement of masking of customer desired sensitive and PII elements to satisfy security requirements. The configurable masking capabilities provided strong production quality data for non-production environments. This accelerated end to end testing for applications, pipelines and data. This minimized data and application defects and outliers when deploying into production.
The Result
- 13 application data migrations in 8 months
- Handling of tables with multiple clob and blob columns
- Pre and post migration assessment reduced manual data validation and testing dramatically and accelerated focus on application refactoring and migration
- Format preserving encryption that ensured all application testing worked without issues
- 100+ pipelines orchestrated and controlled across multiple data collector nodes with per pipeline performance metrics available in an automated manner
Your Data Modernization Partner
Karsun combines capabilities in data analytics consulting, to provide insights to solve the most challenging business problems and to make data-driven decisions. Our proven IT experience implementing and operating the underlying data platforms, tools, technologies and infrastructure allows us to repeatedly execute effective modernization solutions at an enterprise level. This rare combination of experience with data management, data operations, and data and analytics application, combined with the understanding of critical Information Technology needed to manage it at scale, reduces risk. Work with a partner that ensures that your agency will have the support for both current and future data and analytics needs, however they evolve.
Modern Data Solutions
Solutions for every step in your agency’s data strategy roadmap
