Data Engineer Pl

Description

We are looking for professionals who are eager to grow, get their hands dirty, and make things happen. We are living a unique moment, full of challenges and opportunities to add even more value to our clients. If you have this profile and this ambition, apply for this position and join us to build the future of the financial market. **\#ComeBeGenial** **Responsibilities and attributes** * Develop and evolve **batch and streaming data pipelines** using Python, Spark, and Databricks. * Build integrations with **databases, APIs, and AWS services**. * Model and transform data following the platform's architecture and quality standards. * Monitor pipelines in production, acting on **failure investigation, performance, and root cause**. * Participate in **code reviews** and contribute to the evolution of the team's engineering standards. * Work closely with business areas to **understand requirements and transform needs into data solutions**. * Ensure good practices of **data quality, security, governance, and observability**. **Requirements and qualifications** * Solid **Python** for data engineering, with good development, testing, and error handling practices. * **SQL**, including CTEs, window functions, and query optimization. * **Apache Spark / PySpark**, with an understanding of partitioning, shuffle, joins, cache, and execution plans. * **Delta Lake**, including MERGE, OPTIMIZE, VACUUM, schema evolution, and reprocessing strategies. * **Databricks**, with experience in Jobs/Workflows and knowledge of Unity Catalog. * **AWS for data**, mainly S3, IAM, and integration/messaging services. * **Dimensional modeling and layered architecture**, including SCD types 1 and 2\. * **Git and development via pull request**. * Experience with **ingestion of relational databases and REST APIs**. **Differentiators** * Experience in the **financial market**. * **Structured Streaming**. * **Databricks Asset Bundles and Terraform.** * **CI/CD** for data pipelines. * **Delta Lake**, including MERGE, OPTIMIZE, VACUUM, schema evolution, and reprocessing strategies. **Additional information** Medical Assistance Dental Assistance Meal Voucher Food Voucher Life Insurance Total Pass GymPass Daycare Allowance PPR São Paulo, Remote model **What does it mean to be Genial?** It is to seek excellence, to make things happen. It is to be talented, intelligent, curious, and to accept any challenge. It is to present and defend your ideas. It is to put the client first and to be passionate about what you do. **Why Genial?** Because we are digital, but we are also human. If you, like us, believe that investment is not synonymous with a suit and tie, we want to get to know you! **About Us** The company that invests in you! We simplify the financial market so that our clients can make good decisions when it comes to investing. **Our Values** Seek the best results for clients; Put excellence into everything we do; Be a responsible, transparent, and sustainable company; Innovate to inspire; Be passionate about what you do; Always learn and improve.