Databricks SME

Key Responsibilities:
Architecture & Platform Design

- Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework

- Define reference architectures for batch, streaming, analytics, and ML workloads

- Select and standardize cluster, compute, and workspace architectures

- Design multi-workspace strategies (dev/test/prod, shared vs. isolated)

- Ensure architectures meet scalability, availability, and performance requirements

Well-Architected Framework Alignment

Apply Databricks best practices across all pillars, including:

- Security & Governance (Unity Catalog, IAM, data access controls)

- Reliability & Resilience (job retries, checkpointing, failure isolation)

- Performance Efficiency (cluster sizing, autoscaling, caching)

- Cost Optimization (compute policies, workload separation, monitoring)

- Operational Excellence (monitoring, automation, CI/CD, runbooks)

Implementation & Engineering

- Lead Databricks workspace, cluster, and Unity Catalog implementations

- Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming

- Build and optimize ETL/ELT pipelines using Spark and SQL

- Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)

- Establish CI/CD pipelines for notebooks, jobs, and infrastructure

Security, Governance & Compliance

- Implement Unity Catalog for centralized governance

- Define data classification, lineage, and audit strategies

- Enforce least-privilege access and secure networking patterns

- Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)

Operations & Optimization

- Monitor platform health, performance, and cost

- Troubleshoot production issues across jobs, clusters, and data pipelines

- Perform workload tuning and cost-performance optimization

- Define SLOs, alerts, and operational metrics

Collaboration & Leadership

- Partner with Data Engineering, Analytics, ML, Platform, and Security teams

- Translate business requirements into technical architectures

- Provide architectural guidance and technical mentorship

- Communicate risks, tradeoffs, and recommendations to leadership

Required Qualifications:
Experience

- 7+ years in data engineering, analytics, or platform architecture

- 3–5+ years hands-on Databricks experience in production environments

- Proven experience applying the Databricks Well-Architected Framework

- Experience designing cloud-native lakehouse architectures

- Experience supporting mission-critical data platforms

Technical Skills

- Databricks Lakehouse Platform

- Apache Spark (PySpark / Scala / Spark SQL)

- Delta Lake, Delta Live Tables, Structured Streaming

- Unity Catalog (governance, lineage, access controls)

- Cloud platforms: AWS, Azure, or GCP

- Infrastructure as Code (Terraform strongly preferred)

- CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)

- Data formats and protocols (Parquet, JSON, Avro)

Certifications Required:

- Databricks Certified Data Engineer Professional

- Databricks Certified Professional Architect (or equivalent advanced certification)

Preferred / Additional Certifications

- AWS Certified Solutions Architect (Associate or Professional)

- Azure Solutions Architect Expert

- Google Professional Data Engineer

- Databricks Machine Learning Professional

- Snowflake or other cloud data platform certifications

Soft Skills

- Strong architectural decision-making and documentation skills

- Excellent communication with technical and non-technical stakeholders

- Ability to lead design reviews and architecture governance forums

- Strong troubleshooting and performance-tuning mindset

Nice-to-Have Experience

- MLflow and MLOps architectures

- Real-time analytics and streaming pipelines

- Multi-region or cross-account data architectures

- Consulting or MSP delivery experience

Originally posted on Himalayas