Data Engineering | Remote

Description

Job Summary: A professional to work on a strategic project migrating the legacy Data Warehouse to an Enterprise Lakehouse on Databricks, with a focus on pipeline reconstruction. Key Highlights: 1. Rebuild legacy pipelines in Databricks using PySpark and Spark SQL 2. Implement bronze, silver, and gold layers (medallion architecture) 3. Optimize pipeline performance and costs **What We’re Looking For**: A professional to work on a strategic project migrating the legacy Data Warehouse to an Enterprise Lakehouse on Databricks, focusing on reconstructing pipelines currently implemented in DataStage and Azure Synapse. The candidate will be responsible for reimplementing pipelines natively in PySpark and Spark SQL, supporting data platform modernization and ensuring parity between the legacy environment and the new Lakehouse ecosystem. **Responsibilities**: * Rebuild legacy Data Warehouse pipelines in Databricks using PySpark and Spark SQL; * Implement bronze, silver, and gold layers following medallion architecture standards; * Apply CDC and batch ingestion patterns per project guidelines; * Execute domain-based reconstruction waves in coexistence with the legacy DW until cutover; * Implement business rules and transformations with automated tests; * Perform reconciliation and validation of data parity between the legacy environment and the Lakehouse; * Optimize pipeline performance and costs through partitioning, OPTIMIZE, Z\-ORDER, and workload sizing; * Contribute to technical documentation of migrated rules; * Support prioritization and execution of migration waves; **Requirements and Qualifications**: * Advanced knowledge of PySpark and Python; * Experience developing large-scale batch pipelines; * Advanced SQL knowledge; * Experience in relational and dimensional data modeling; * Experience with Databricks and Delta Lake in production environments; * Knowledge of medallion architecture (bronze, silver, gold); * Experience with Delta Live Tables, Lakeflow, and Asset Bundles; * Hands-on experience in ETL pipeline migration or reconstruction; * Experience translating business rules from legacy platforms to Spark; * Knowledge of CDC and batch ingestion from relational databases; * Experience with AWS (S3, Glue, EMR, Athena, Lambda, DMS, and Step Functions); * Knowledge of Azure Synapse (SQL Pools and Pipelines); * Experience in data quality and reconciliation; * Knowledge of automated testing and Data Quality controls; * Experience with Git and CI/CD applied to data engineering; **Nice-to-Haves**: * Knowledge of DataStage for job reading and reverse-engineering support; * Experience in Data Warehouse-to-Lakehouse migration programs; * Knowledge of Unity Catalog (permissions, lineage, and data contracts); * Experience in the financial or credit industry; * Databricks Data Engineer Associate or Professional certifications; **Behavioral Profile**: We are looking for someone who: * Enjoys working in teams and is **collaborative** in their assignments; * Has the **courage** to challenge themselves and go beyond, embracing new growth opportunities; * Transforms ideas into **creative solutions** and pursues quality in all daily activities; * Possesses strong **problem-solving skills**; * Has the ability and feels comfortable working independently and **managing their own time**; * Is interested in handling technologically **adversarial and innovative** situations. Big enough to deliver – small enough to care. **\#VempraGFT** **\#LetsGoBeyond** **\#ProudToBeGFT** .buttontext616d4705bd22aece a{ border: 1px solid transparent; } .buttontext616d4705bd22aece a:focus{ border: 1px dashed \#972875 !important; outline: none !important; } With over 35 years in the market and operations in 16 countries, GFT employs more than 12\.000 people worldwide, including 4\.000 in Brazil. Here, diversity, equality, and inclusion are at our core, and ensuring a safe workplace for everyone is part of who we are. We foster a culture of development and growth anchored in our Continuous Scaled Learning pillar because we believe intelligent use of technology is the key to success in this digital world. .buttontext51d9fe5a33d0d42c a{ border: 1px solid transparent; } .buttontext51d9fe5a33d0d42c a:focus{ border: 1px dashed \#972875 !important; outline: none !important; } What We Offer Multi-benefit card – you choose how and where to use it. Scholarships for undergraduate, graduate, MBA, and language courses. Certification incentive programs. Flexible working hours. Competitive salaries. Annual performance reviews with structured career plans. Opportunities for international careers. Wellhub and TotalPass. Private pension plan. Childcare allowance. Medical assistance. Dental assistance. Life insurance. \#VemPraGFT .jobColumnTwo .displayDTM h2 { margin\-top: 0px !important; } .jobColumnTwo .displayDT ul{ padding\-inline\-start: 20px !important; list\-style\-position: outside !important; } .buttontextb636f3111304c10b a{ border: 1px solid transparent; } .buttontextb636f3111304c10b a:focus{ border: 1px dashed \#972875 !important; outline: none !important; }