Databricks Data Engineer | Senior | (Remote)

Description

. **Responsibilities and Assignments** * Reconstruct legacy DW pipelines in Databricks (PySpark / Spark SQL), based on specifications derived from reverse engineering; * Implement bronze, silver, and gold layers following the medallion template, ingestion standards (CDC/batch), and project-defined reconstruction standards; * Execute domain-based reconstruction waves, coexisting with the legacy DW until cutover; * Implement business rules and transformations with automated tests in pipelines; * Perform reconciliation and parity validation of data between the legacy DW and the Lakehouse; * Optimize pipeline performance and cost (partitioning, OPTIMIZE/Z\-ORDER, job sizing); * Contribute to technical documentation of migrated rules and support wave prioritization. **Requirements and Qualifications** * Minimum 5 years of experience in Data Engineering; * Advanced PySpark and Python: development of large-scale batch pipelines with testing and sound engineering practices; * Advanced SQL and data modeling: complex transformations, performance tuning, and relational/dimensional modeling in DW contexts; * Databricks and Delta Lake: production jobs, workflows, and medallion architecture (bronze, silver, gold); Delta Live Tables / Lakeflow and Asset Bundles; * ETL pipeline migration/reconstruction: translation of business rules from legacy tools to Spark (no lift\-and\-shift), with CDC/batch ingestion from relational databases; * AWS ecosystem: S3, Glue, EMR, Athena, Lambda, DMS, and Step Functions; * Azure Synapse: experience with legacy environment (SQL pools and pipelines) to support reverse engineering; * Data quality and reconciliation: automated testing, DQ gates, and legacy × new comparison for parity acceptance; * Git and CI/CD applied to data pipelines. Desirable: * Knowledge of DataStage (job reading for reverse engineering) — a significant differentiator; * Prior experience in DW-to-lakehouse migration programs — a significant differentiator; * Unity Catalog (permissions, lineage) and data contracts; * Experience in financial services or credit markets; * Databricks certifications (Data Engineer Associate / Professional). Compass UOL is a global firm and part of the AI Revolution Company, together transforming organizations using Artificial Intelligence, Generative AI, and other of today’s most advanced technologies. We equip our team with proprietary and external AI\-driven tools to design and build digital\-native platforms, integrating cutting\-edge technologies and enabling companies to innovate, transform their businesses, and drive success in their markets. To achieve this, we attract and develop the best talent, creating opportunities that enhance people’s lives and highlight the positive impact of disruptive technologies. We empower borderless talent and promote knowledge and opportunities in the latest market trends, driving significant personal and professional growth. **Join us and be part of the AI\-driven revolution.**