Databricks Data Engineer and Scientist

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Databricks Data Engineer and Scientist based in United States.

In this role, you will design, develop, and optimize enterprise-scale data pipelines within a Databricks and AWS-based Lakehouse environment. You will transform complex data into reliable, analytics-ready assets that support informed business decisions and advanced analytical initiatives. Working across data engineering, data modeling, and AI-assisted development, you will modernize legacy systems while ensuring data accuracy, performance, and scalability. You will contribute to a Bronze, Silver, and Gold medallion architecture and build robust ETL processes with comprehensive logging and error handling. The role combines hands-on technical delivery with collaboration across cross-functional teams in support of a critical federal data environment. This is a full-time, remote opportunity for an experienced professional who enjoys solving complex data challenges and improving enterprise data infrastructure.

Accountabilities
-

Design, build, deploy, and maintain end-to-end data pipelines for an enterprise Lakehouse using Databricks and AWS.

-

Develop and manage reliable ETL processes with comprehensive logging, error handling, troubleshooting, and issue resolution.

-

Design and maintain data models within a Bronze, Silver, and Gold medallion architecture, selecting appropriate dimensional or relational modeling approaches.

-

Analyze and decompose complex legacy SQL, rebuild pipelines and data models, and validate results for accuracy and performance.

-

Optimize data pipeline performance, scalability, and query execution using partitioning, bucketing, caching, and other relevant techniques.

-

Use AI-assisted coding tools to accelerate pipeline analysis, data engineering, debugging, and data modeling tasks.

-

Support data security and compliance requirements while collaborating with cross-functional teams to deliver high-quality, analytics-ready data.

Requirements

-

Bachelor's degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, or Information Systems and Technology.

-

At least 8 years of experience designing, developing, deploying, and managing ETL processes and data pipelines.

-

At least 3 years of experience building, debugging, and optimizing data pipelines using Databricks notebooks, clusters, and job orchestration tools, including Delta Lake and Unity Catalog.

-

At least 3 years of experience developing Python and PySpark code for ETL scripts and custom data transformations.

-

Practical experience using AI-assisted coding tools, such as Claude Code or Cursor, to support data engineering and data modeling activities.

-

Experience designing data models using dimensional and relational approaches within a Bronze, Silver, and Gold medallion architecture.

-

Hands-on experience with AWS services, including AWS Glue, Amazon S3, and Amazon RDS.

-

Knowledge of ETL and query performance optimization techniques, including partitioning, bucketing, and caching.

-

Demonstrated ability to analyze complex legacy SQL, reconstruct data pipelines and models, and compare outputs to validate accuracy and performance.

-

Ability to obtain and maintain a Public Trust or other required Suitability/Fitness determination, depending on client requirements.

-

U.S. citizenship is required to support the federal contract.

Benefits

-

Work arrangement: Remote position based in the United States.

-

Employment type: Full-time.

-

Technical environment: Opportunity to work with Databricks, AWS, Lakehouse architecture, Python, PySpark, and AI-assisted development tools.

-

Compensation: Salary, bonuses, and other financial benefits were not specified in the provided job description and should be confirmed during the hiring process.

-

Additional benefits: Healthcare, paid time off, retirement plans, and other employee perks were not detailed in the provided description and should be confirmed with the hiring team.