Senior Data Engineer

Description

**Mission:** Your mission will be to work on the front line of our **new data projects**, ensuring end-to-end delivery of solutions—from ingestion to final visualization—while guaranteeing that new business requirements are met with agility and precision. Using **Databricks and dbt**, you will be responsible for transforming needs from various business areas into accurate data assets, ensuring that new demands are addressed with agility and technical excellence. You will work with one of our largest clients in the **financial sector**. You will join an ecosystem presenting highly complex technical challenges, operating within a dynamic environment that requires resilient solutions for large-scale data processing on **Azure** infrastructure. **Required Knowledge / Essential Requirements:** * Experience with Python, SQL, PySpark, and especially DBT; * Experience with Delta Lake, Databricks, and Unity Catalog; * Hands-on experience with Azure infrastructure; * Experience with code versioning (Git) and CI/CD processes (preferably Azure DevOps); * Solid knowledge of Lakehouse architecture with Medallion design. **Desirable (but not mandatory) Qualifications:** * Prior experience in **financial institutions**. * LGPD and GDPR. **Responsibilities:** * **Data Pipeline Development:** Build, optimize, and maintain scalable data pipelines using **Python**, **SQL**, and **PySpark**, ensuring efficient processing of large volumes of information. * **Modeling and Transformation with DBT:** Implement the data transformation layer using **dbt**, applying business rules and ensuring data code quality, documentation, and testability. * **Data Lifecycle Management (Medallion):** Structure and organize data within the **Lakehouse** architecture, ensuring smooth and governed transitions across the **Bronze, Silver, and Gold** layers. * **Governance and Security:** Configure and manage permissions, data lineage, and catalogs via **Unity Catalog**, ensuring compliance with regulatory standards. * **Working with Delta Lake:** Ensure storage reliability and performance using **Delta Lake**. * **DevOps Culture and Automation:** Support the development lifecycle through code versioning in **Git** and automated deployments via **CI/CD** pipelines (Azure DevOps), ensuring secure and traceable production releases. * **Collaboration:** Support the team by promoting best practices and sharing experiences. **What You’ll Find Here:** * Fully remote work (anywhere office). * Performance evaluation, individual development plan, and the "Investing in You" Program. * Health and wellness program, including medical and dental plans. * Gamified challenges and an internal store where you can purchase products using your points. * Birthday Day Off. * Equipment/Home Office allowance. * Flexible allowance for daily expenses. * Referral Rewards Program. * Life insurance. Privacy Notice – Recruitment & Selection: https://drive.google.com/file/d/1LV2tTa\_PeSbyiXuqrG\_6WJw0jrEqRxnA/view?usp\=sharing