Senior Data Engineer (Databricks / Spark / Python)

Description

Job Summary: Keyrus is seeking a passionate Data Engineer to develop and maintain data ingestion pipelines, process large-scale data, and optimize performance on Databricks. Key Highlights: 1. Opportunity to grow professionally and build a career path 2. Develop and maintain data ingestion pipelines with Autoloader 3. Collaborate with business, analytics, and technology teams ***Keyrus believes in diversity and inclusion. We encourage everyone to participate in our hiring process, regardless of gender, age, race, or religion. We do not tolerate any form of discrimination. This principle is reinforced throughout our hiring process and lived daily within the company.*** We know great results are achieved only with a strong team; therefore, we seek talented and passionate individuals who wish to grow professionally and build a career path with us. **Roles and Responsibilities:** * Develop and maintain **data ingestion pipelines** using **Autoloader** (incremental, schema evolution, and throughput optimization). * Process large-scale data using **Spark Structured Streaming** and **Spark SQL**. * Apply best practices in **data engineering**, including partitioning and optimizations with **Delta Lake**. * Perform **performance and cost analysis and optimization** in **Databricks** environments. * Implement **Data Quality processes** to ensure data reliability. * Use **Git** for version control, including pull requests and collaborative development best practices. * Create **clear and well-organized technical documentation** for implemented solutions. * Collaborate with **business, analytics, and technology teams** to ensure data is structured, accessible, and secure. **Essential Qualifications:** * Experience in **data engineering and large-volume data processing**. * Proficiency in **Python, Apache Spark, and Databricks**. * Experience building and maintaining **scalable data pipelines**. * Knowledge of **Spark SQL and Structured Streaming**. * Experience with **code versioning using Git**. **Preferred Qualifications:** * Experience with **agile methodologies** (Scrum, Kanban, etc.). * Knowledge of **modern data architecture** and best practices in **ETL/ELT**. * **Self-managed profile**, with strong **problem-solving skills** and decision-making ability in complex environments. * Interest in **emerging technologies and innovation in data solutions**. Our mission is to help companies unlock the full potential of Data and Digital to enhance performance, drive transformation, and generate new growth levers and competitiveness. Join the Keyrus Team! If you’d like to learn more about us, we invite you to visit our website: http://www.keyrus.com/br/ ***Good luck!***