The consequences were predictable. Even minor changes to data flows required manual intervention and carried disproportionate risk. Scaling the pipeline to accommodate new product attributes, new regions, or higher data volumes meant re-engineering logic that no one had intended to be permanent. The architecture that once connected systems was now slowing them down.
This is a pattern that shows up across enterprises carrying technical debt in integration layers. When transformation logic lives inside a platform rather than in a governed, portable data pipeline, the platform becomes a constraint. The telecom client had reached that point. What they needed was not a patch or an incremental upgrade. They needed a new architectural foundation, one that was cloud-native, automated, and designed to scale globally without accumulating the same brittleness over time.
Solution
At Brillio, we designed and implemented a cloud-native ingestion, transformation, and orchestration pipeline built on Microsoft Azure, replacing the Salesforce-bound integration end-to-end.
The architecture begins at the source. Product master data originates from the client’s enterprise PLM system and is ingested through a centralized enterprise integration layer into Azure Data Lake Storage (ADLS). From there, data moves through a structured three-layer architecture: a landing layer captures incoming data, a staging layer applies initial transformations, and a final layer holds curated, publication-ready datasets. Transformations across both active layers are executed using Databricks Spark, with Azure Data Factory (ADF) handling orchestration across the pipeline.
The most consequential design decision was the adoption of a metadata-driven orchestration framework. All transformation logic previously embedded in Salesforce Apex batch jobs was reverse-engineered, externalized, and rebuilt in Azure using Spark-based processing. Rather than encoding rules in hard-coded logic that requires developer intervention to change, transformation rules are now configurable through metadata. Adding a new product attribute or onboarding a new data object no longer requires pipeline surgery.
The platform is fully integrated with Azure DevOps for pipeline management, Azure Monitor for observability, a dedicated data quality framework for automated checks, and Unity Catalog to support governance across the environment. Transformed data is pushed back into Salesforce in its final, curated form, ready for consumption by the business systems that depend on it. The result is a governed, automated, and extensible foundation designed to support future analytics and machine learning use cases.
