Lead Azure Data Engineer

Rackspace

↗

IT Services and IT Consulting 👥 5K-10K employees employees 📍 San Antonio, Texas

Job Overview

We are seeking an experienced Lead Azure Data Engineer to lead the design, development, deployment, and optimization of enterprise data ecosystems on the Microsoft Cloud. This is a strategic role focused on building scalable, reliable, and high-performing data platforms and pipelines for both batch and real-time data processing, analytics, and reporting.

Key Responsibilities

• Lead end-to-end development of scalable data pipelines and orchestration frameworks using Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, and Microsoft Fabric


• Build robust real-time and batch data pipelines, including integration with streaming sources (e.g., Event Hubs, Kafka) and structured streaming engines


• Design and implement Structured Streaming applications in Spark for near-real-time processing of streaming data


• Develop and maintain ETL/ELT pipelines and transformations leveraging Spark, PySpark, SQL, and fabric orchestration capabilities


• Architect and implement data solutions using Microsoft Fabric, including OneLake, Dataflows, warehouses, and Fabric capacity planning


• Collaborate on data governance, cataloging, and asset organization using Unity Catalog within Databricks and Fabric environments


• Manage Microsoft Fabric capacity and resource utilization to optimize performance and cost efficiency for analytics workloads


• Design, deploy, and optimize Databricks dashboards and reporting artifacts for business stakeholders


• Apply best practices for data modeling, caching, file sizing, and performance tuning of Spark and Delta Lake jobs


• Oversee governance, access controls, metadata management, and lineage using Unity Catalog


• Lead and mentor a team of data engineers, fostering best practices in development, operations, documentation, and quality


• Work with cross-functional teams (architecture, BI, data science, DevOps) to translate business requirements into scalable data solutions


• Partner with stakeholders to define data strategy, standards, and architectural roadmaps


• Establish and enforce standards for data quality, testing, monitoring, operational observability, and governance


• Implement secure, compliant data access and lineage frameworks across cloud data platforms


• Implement CI/CD pipelines, infrastructure-as-code for data platform artifacts, and automated testing frameworks for data jobs and workflows

Required Qualifications

10+ years of hands-on experience in data engineering on Azure with deep expertise in ADF, Synapse, Databricks, and Microsoft Fabric. Proven experience with real-time data processing, streaming architectures, and Spark Structured Streaming. Strong proficiency in Azure Data Factory, Spark (PySpark), SQL, Azure Synapse Analytics, Databricks Runtime, and cloud storage. Solid knowledge of Unity Catalog for data governance, security, and access management. Experience designing and managing Databricks Dashboards, performance optimization, cost controls, and data platform resource tuning. Expertise in building scalable, fault-tolerant, and high-throughput batch & streaming data solutions. Excellent leadership, cross-team collaboration, and communication skills.