Lead Engineer, Big Data - Databricks - Remote

JOB DESCRIPTION Job Summary
Provides lead level engineering support for information technology (IT) big data architecture activities on Microsoft Azure cloud infrastructure. Representing as a Forward Deployed Engineer, responsible for leading the collection, storage, processing, and analysis of large health care payor datasets supporting Medicare/Medicaid operations. Designs enterprise-scale data solutions on Azure Databricks, Azure Data Factory, and Azure Synapse with emphasis on performance, scalability, and high-reliability. Champions AI-first practices, mentors team members, and drives adoption of machine learning and advanced analytics to improve member health outcomes. Partners strategically with business leaders to identify data-driven opportunities and delivers end-to-end solutions.
Essential Job Duties

- Operates as a lead Forward Deployed Engineer, driving end-to-end ownership culture and mentoring team members on business problem-solving.
• Provides subject matter expertise in Azure data platform architecture (Azure Databricks, Azure Synapse, Azure Data Factory, Azure Data Lake).
• Champions AI-first development practices, implementing GitHub Copilot workflows and Azure Machine Learning solutions for predictive health care analytics.
• Designs and implements business intelligence (BI) solutions using Power BI, Tableau integrated with Azure data platform.
• Analyzes current business practices and identifies opportunities for leveraging Azure data and AI PaaS services to improve Medicare/Medicaid operations.
• Builds reusable data quality frameworks and automated data validation using Azure DevOps and quality assurance automation tools.
• Leads integration initiatives connecting Azure data platform with low-code platforms (Pega, Salesforce, Sitecore).
• Converts business requirements into technical architecture, design and implementation for health care analytics use cases.
• Designs analytical data models supporting self-service BI for clinical, financial, and operational stakeholders.
• Implements batch and real-time data pipelines for claims processing, quality metrics, and population health analytics.
• Partners with business leaders across the organization to identify high-value data opportunities and prioritize initiatives.
• Leads technical issue resolution and collaborates with Azure support, infrastructure, and vendor teams.
• Supports project planning, scheduling and execution using Agile methodologies.
• Ensures HIPAA compliance and health care data governance standards are maintained.
• Provides training, mentoring and support to new and existing big data team members on Azure data technologies and Forward Deployed Engineer principles.

Required Qualifications

- At least 6 years of data management experience on cloud data platforms, or equivalent combination of relevant education and experience.
• Bachelor's degree in IT, computer science, statistics, business, health care, engineering, etc.
• Experience with Microsoft Azure data services (Azure Databricks, Azure Data Factory, Azure Synapse, Azure Data Lake Storage, Azure Machine Learning).
• Proven experience implementing AI-assisted development and machine learning solutions.
• Proficiency in Python, PySpark, Scala, and SQL with quality assurance automation experience.
• Experience with real-time streaming architectures (Kafka, Azure Event Hubs, Spark Streaming).
• Experience integrating data platforms with low-code systems (Pega, Salesforce).
• Health care payor industry experience with Medicare/Medicaid claims, quality metrics (HEDIS, Stars), and regulatory reporting.
• Understanding of HIPAA requirements and health care data governance frameworks.
• Ability to mentor and develop team members with Forward Deployed Engineer mindset.
• Experience with Azure DevOps CI/CD pipelines and Infrastructure as Code (Terraform, ARM templates).
• Critical-thinking and problem-solving skills with business outcome orientation.
• Ability