Building a Scalable and Highly Available Data Management Infrastructure

About Microsoft

Microsoft is a tech leader with 44 years of presence in tech innovation space and revenue of more than $120 Billion supported by 0.2 Million strong workforces. Microsoft launched payment services under their finance program in 2016, MSFT’s Worldwide Payment Solutions (WPS) Group facilitates the process of various finance options for its products to the retail consumers through tie-ups with large banks. And the industry was quick to adopt the solution, followed by quick growth and adoption across industries and geographies which led to rapid growth in data and hence the need to effectively manage their continuously increasing data of various product deals across sales region and geographies.

To overcome the challenge, a complete cloud strategy was built keeping the aspects of scalability, efficiency, and cost at the centre of strategy and was topped with data consumption and insight generation strategy.

Business Challenge

Since most of the data was coming from various Banking, retail and other industrial partners with each partner having their data in their system and following various schemas led to the challenge of data inconsistency which made migration and utilization a huge challenge. So they needed a mechanism where after identifying the relevant factors they could be pulled from various sources, harmonized and then could be bestowed to advanced analytics algorithms build by Brillio’s data science team to address the problem and perform reporting.

A scalable and highly available data management infrastructure with a self-service reporting framework for the democratization of data was required. MSFT further needed an integrated data platform to embark on its analytics journey to understand top lenders, top customers, aging applications and explore other data monetization opportunities.

Approach

The solution was aimed at migrating applications and storage to MS Azure IAAS to reduce the bottleneck, cost and resource dependency associated with on-premise infrastructure and application management and to achieve high availability, better disaster recovery, and scalability. And then democratizing the reports on top of it.

To help the customer to not only derive insights out of data but also to deal with data and infrastructure challenges Brillio adopted a zoned approach where each zone was handling a particular part of the problem and individual applications were utilized for each zone to handle the specific challenge. Not only that, Brillio automated the functioning of all zones to improve the overall accuracy and achieve cost efficiency.

Solution

Since the problem was complex in nature and volume, various zones were utilized as below:

Zone 1 – Source

This zone was aimed at bringing together various source system to ensure connectivity and availability of various systems

Zone 2 – Ingestion 

The data from various source systems were pulled using Azure Data Factory and then was converted into a canonical schema.

Zone 3 – Data Systems

The data was classified through various classification algorithms as Hot and Cold data based on business relevancy, within the zone the hot data was fed to Azure SQL DB and also archived to azure data lake while cold data was simply archived to Azure data lake. Ensuring operational and cost-efficiency.

Zone 4 – Computing 

The hot data was fed to various analytics algorithms to derive insights.

Zone 5 – Consumption 

The output of the computing zone was converted to consumable format through Power BI based on specific usability (Business use, analytics use, etc.)

Business Impact and Benefits

• Increased Availability
• Disaster Recovery using Azure Backup
• Improved performance with the usage of fewer resources
• In-built VM Diagnostics which can be used for detailed analysis
• Track and monitor deal data quality and quickly respond to critical errors
• Proactive actions and improved decision making with better insights on top lenders, top customers, aging applications
• Ensure high availability of data warehouse and 24/7 support
• Measure impact on sales hierarchy and build a robust incentivizing mechanism for deals
• KPI and metrics to track deals, loan workflow, payments and disbursements