Faster chat, better deals — Get the App

Senior Data Engineer ID86297

Indeed

Company

Job typeFull-time
Workplace typeRemote
Experience levelMore than 10 years
Education levelNo degree limit

Description

**About the role** We are looking for an experienced **Senior Data Engineer** to help modernize a 15\-year\-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers. This role also uses AI tools like Claude and GitHub Copilot to speed up development. **What you will do** * Design, build, and operate batch and streaming data pipelines on Databricks using **PySpark**, Delta Lake, and Databricks Workflows. * Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and machine learning consumers. * Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption. * Use Claude or Github Copilot as a development accelerator, generating code scaffolding, writing and reviewing tests, creating documentation and prototyping solutions. * Write clean, well\-tested Python and SQL; maintain high standards through code review and documentation. * Optimize Spark jobs and Delta tables for performance and cost, including partitioning, clustering, caching, and cluster sizing. * Implement data quality, lineage, and governance controls using Unity Catalog and automated validation checks. * Debug, troubleshoot, and resolve pipeline failures, data defects, and production incidents. * Participate in Agile or product\-centric delivery practices including sprint planning and retrospectives. * Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance best practices. **Must haves** * **4\+ years of professional experience** in data engineering, featuring direct expertise with Apache Spark and cloud\-based data architectures. * Strong hands\-on experience building data pipelines with **Databricks**, Apache Spark (PySpark), and Delta Lake. * Advanced **SQL and Python**, with strong data modeling skills across dimensional and Lakehouse patterns. * Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs. * Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory). * Experience with legacy platform migrations, ETL modernization, or managing data hygiene when porting old systems. * Strong problem\-solving, collaboration, and communication skills, including mentoring junior engineers and explaining data concepts to non\-technical stakeholders. * Familiarity with **Unity Catalog**, data governance, access control, and PII handling. * Experience with dbt or an equivalent transformation framework. * Familiarity with secure coding standards and industry security best practices. * Experience delivering production data platforms at scale. * Upper\-intermediate English level. **Nice to haves** * Experience with Infrastructure as Code (IaC) using Terraform and CI/CD using Azure Devops. * Experience working with relational databases (specifically PostgreSQL) and data persistence concepts. * Familiarity with logging and monitoring tools (e.g., Dynatrace, CloudWatch, Databricks system tables). * Experience working in Agile or team\-based development environments preferred. **Perks and benefits** * **Growth without limits:** build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget * **Competitive compensation:** get recognition that reflects your skills and impact, with regular performance and compensation reviews * **Flexibility:** work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm * **Meaningful, modern projects:** build impactful products using modern technologies alongside global teams and leading brands * **Collaborative culture:** join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized * **Well\-being \& support:** access local well\-being programs and people\-focused support tailored to your location Job Type: Full\-time Work Location: Remote

Posted by

João Silva

Indeed · HR

Location

João Silva

Indeed · HR

Similar jobs

Senior Data Engineer ID86297 job by Indeed in 2026 | ok.com