Mid-Level Data Engineer

Description

Job Summary: Your mission will be to design and evolve robust, reliable, and scalable data pipelines, handling ingestion, transformation, and orchestration of large volumes of data. Key Highlights: 1. Development and maintenance of Batch and Streaming data pipelines 2. Working with data tools and services in the Azure ecosystem 3. Advanced Python usage and experience with Apache Spark/PySpark **JOB DESCRIPTION** Your mission will be to design and evolve robust, reliable, and scalable data pipelines. On a day-to-day basis, you will handle ingestion, transformation, and orchestration of large volumes of data, ensuring availability, performance, and quality for analytics, products, and machine learning models. **DAILY RESPONSIBILITIES:** * Development and maintenance of data pipelines for Batch and Streaming scenarios. * Implementation of real-time data ingestion routines using messaging systems (Kafka, RabbitMQ, or equivalent services). * Building complex transformations (ETL/ELT) while ensuring data consistency and quality. * Modeling, optimization, and management of relational databases (especially PostgreSQL). * Creation and maintenance of DAGs in Apache Airflow for process orchestration. * Distributed processing using Apache Spark (PySpark). * Applying observability practices, version control, documentation, and automated testing. * Working with data tools and services in the Azure ecosystem. * Participation in agile methodologies (SCRUM or similar). **REQUIREMENTS AND QUALIFICATIONS:** * Programming language: **Python (advanced)**. * Relational Database: **PostgreSQL** (modeling, optimization, and basic administration). * Distributed Processing: **Apache Spark / PySpark**. * Orchestration: Proven experience with **Apache Airflow**. * Building **Batch and Streaming** pipelines. * Messaging: **Kafka**, **RabbitMQ**, or similar services. * Experience with **caching** strategies for performance. * Experience in the **Azure** ecosystem (ADF, ADLS, Synapse, or similar). * Version control with **Git**. * Best practices and automated testing applied to data pipelines. * Agile methodologies (SCRUM or Kanban). **PREFERRED QUALIFICATIONS** * Knowledge of **Java** and/or **Scala**. * Understanding of distributed architecture, reactive programming, and microservices. * Experience with messaging technologies (Kafka, RabbitMQ, Event Hubs/Service Bus). * Infrastructure as Code: **Terraform** or ARM. * Knowledge of **Docker**. OUR BENEFITS * Meal/Voucher Allowance * Health Insurance (Sulamérica) * Dental Insurance * Birthday Day Off * Life Insurance * Remote Work (Home Office) Employment Type: Full-Time CLT Benefits: * Medical Assistance * Dental Assistance * Life Insurance * Food Allowance * Meal Allowance Selection Question(s): * What is your salary expectation (CLT)? * Are you available to work remotely? Work Location: Remote