Paving the way from legacy databases to flexible data models

Lakshmi H Shastry • January 08, 2020
Share This Article


Majority of the Enterprises are embracing digital transformation strategy to accelerate impact, drive profound outcomes and remain competitive. New age digital natives, technology giants, start-ups, peer companies are accessing and penetrating growth markets, by embracing digital technologies at speed & scale. To keep pace, Enterprises are forced to embark and strategize in building strong innovation pipeline to enable new business, services, revenue scenarios and rich connected ecosystems and propel future business initiatives.

Digital increasingly revolves around delivering value to customers. Customer centric initiatives helps to engage and elevate experience with reimagined interactions and contextual hyper personalization. Demands of timely information is fulfilled by adoption and integration of disruptive data-driven modern technologies, data & insights platforms for process of growing data volumes, with guaranteed reliability and performance.

Our Enterprise customer, a trusted leader among global relocation companies, catering to relocation of companies and people, as part of digital transformation agenda, focused transitioning to Amazon cloud platform with design-first development approach for accelerating new age web scale applications. Business lever is to improve experience and build stronger relationships, by opening up a plethora of opportunities for service providers with niche capabilities, driven by deeper insights in establishing baseline ratings, mapping personas to journeys, aggregating details at an individual level ,be it client, customer, agent or supplier etc. to engage with strategic vendor partners. Business mandate is to leverage existing legacy portfolio, as the cost lever to fund some of the new initiatives, to avoid investment to the unknown without clear ROI.

To support business, strategy is aimed to get more value from data consolidation with modern purpose-built database solutions for agility, performance, scalability with co-existence of legacy on-premise database to maintain status-quo of core applications. Current enterprise data management foundation is relational data store of over 30 years. These legacies served as lifeline of critical Business Accounting, ERP, CRM, Data warehousing applications. But to compete in market, breaking out of relational only mindset and considering new paradigms was crucial to fulfil vision of comprehensive single source of truth data powering their digital age feature-rich applications. Migration of legacy data from on-premise fixed schema relational model (RDBMS) to cloud based NoSQL flexible schema model was necessary to boost ability to evolve and adapt to needs of altogether different kind of applications quickly and efficiently. Flexible, semi-structured, hierarchical, fast read/writes adjusting to evolving application, is useful for variety of use cases, to provide access to data with milliseconds latency and data persistence with ease.

Business continuity required zero downtime migration of historical financial transaction data from multiple on-premise MS SQL databases, with incremental replication of continuous data capture (CDC) into MongoDB Atlas, a Database As A Service(DBaaS) offering cloud scalability, faster I/O operations, lower cost to handle rapidly growing data volume and variety.

To get the most out of using right tool for right job, as new applications were AWS native, natural consideration was to use AWS Database Migration Service (DMS) Service. AWS DMS is engineered to be more of a migration tool than for complex continuous replication scenarios. It required multi-step manual process for moving data, mandated pre-creation of schema without indexes, CDC enablement and then creation of all indexes. In addition, AWS RDS lack of CDC support, limited the flexibility of using AWS DMS. With the intent to leverage Organization expertise, fast forwarded with native migration and replication tools of MS SQL database with continuous replication and CDC for incremental changes on AWS EC2 Windows Server, helped for initial data loading and synchronization with on premise legacy data store. Services were designed for continuously tracking changes, handling complex business rules mapping, executing relevant stored procedures, transforming SQL hierarchical structure to dynamic JSON model, performing necessary data enrichment and retrying for any failure scenarios for complete reliability.

Data consolidation is based on business rules-based mapping applied on legacy data. Business SLA for changed data to be available in MongoDB Collections from legacy data sources was within 180 seconds as this served as input for dynamic content. Poor data quality and standards due to inherited inherent problems over a period, needed instant cleansing with data quality rules & policies, before publishing for downstream applications. Configurations to control data flow provided flexibility to adapt into multiple environments extending initial data load and incremental change data capture.

MongoDB data structures were designed with embedded and reference design patterns, to achieve best results of data retrieval in terms of time efficiency. Flat structure made it easy to accommodate schema for deriving variety of insights and provided flexibility of fetching data with many to many, one to many and many to one relationship for very high growth of data with different lifecycle. Microservices with fan-out pattern to scale for parallel orchestration, state management & chaining for applying business rules in specific order of priority to convert data consistently to MongoDB data representations. Monitoring & alerting metrics, fail-overs and recovery are in place for addressing majority of the uncertainties.

Flexible data model helped to break away from restrictive one-size-fits-all monolithic data model and focus on new enterprise class application ideas. Supported business prioritization to uncover user pain points and identify opportunities helped drive greater value with competitive differentiations. Data-driven digital technologies AI, ML, RPA further enhanced business outcomes while also helping Enterprises streamline processes and achieve efficiencies in digital transformation journey.

Let’s create something amazing together!

Contact us Next
Latest Blog
LinkedIn Instagram Facebook Twitter