Description
**About the role**
We are looking for an experienced **Senior Data Engineer** to help modernize a 15\-year\-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers. This role also uses AI tools like Claude and GitHub Copilot to speed up development.
**What you will do**
* Design, build, and operate batch and streaming data pipelines on Databricks using **PySpark**, Delta Lake, and Databricks Workflows.
* Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and machine learning consumers.
* Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption.
* Use Claude or Github Copilot as a development accelerator, generating code scaffolding, writing and reviewing tests, creating documentation and prototyping solutions.
* Write clean, well\-tested Python and SQL; maintain high standards through code review and documentation.
* Optimize Spark jobs and Delta tables for performance and cost, including partitioning, clustering, caching, and cluster sizing.
* Implement data quality, lineage, and governance controls using Unity Catalog and automated validation checks.
* Debug, troubleshoot, and resolve pipeline failures, data defects, and production incidents.
* Participate in Agile or product\-centric delivery practices including sprint planning and retrospectives.
* Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance best practices.
**Must haves**
* **4\+ years of professional experience** in data engineering, featuring direct expertise with Apache Spark and cloud\-based data architectures.
* Strong hands\-on experience building data pipelines with **Databricks**, Apache Spark (PySpark), and Delta Lake.
* Advanced **SQL and Python**, with strong data modeling skills across dimensional and Lakehouse patterns.
* Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
* Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
* Experience with legacy platform migrations, ETL modernization, or managing data hygiene when porting old systems.
* Strong problem\-solving, collaboration, and communication skills, including mentoring junior engineers and explaining data concepts to non\-technical stakeholders.
* Familiarity with **Unity Catalog**, data governance, access control, and PII handling.
* Experience with dbt or an equivalent transformation framework.
* Familiarity with secure coding standards and industry security best practices.
* Experience delivering production data platforms at scale.
* Upper\-intermediate English level.
**Nice to haves**
* Experience with Infrastructure as Code (IaC) using Terraform and CI/CD using Azure Devops.
* Experience working with relational databases (specifically PostgreSQL) and data persistence concepts.
* Familiarity with logging and monitoring tools (e.g., Dynatrace, CloudWatch, Databricks system tables).
* Experience working in Agile or team\-based development environments preferred.
**Perks and benefits**
* **Growth without limits:** build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
* **Competitive compensation:** get recognition that reflects your skills and impact, with regular performance and compensation reviews
* **Flexibility:** work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
* **Meaningful, modern projects:** build impactful products using modern technologies alongside global teams and leading brands
* **Collaborative culture:** join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
* **Well\-being \& support:** access local well\-being programs and people\-focused support tailored to your location
Job Type: Full\-time
Work Location: Remote