Senior Data Engineer

Description

Job Summary: You will design and enhance robust, reliable, and scalable data pipelines, handling ingestion, transformation, and orchestration of large-scale data volumes. Key Highlights: 1. Development and maintenance of data pipelines (Batch and Streaming) 2. Advanced Python experience and tools such as Apache Spark/PySpark 3. Experience with data tools and services in the Azure ecosystem **JOB DESCRIPTION** Your mission is to design and enhance robust, reliable, and scalable data pipelines. On a day-to-day basis, you will handle ingestion, transformation, and orchestration of large-scale data volumes, ensuring availability, performance, and quality for analytics, products, and machine learning models. **DAILY RESPONSIBILITIES:** * Develop and maintain data pipelines for Batch and Streaming scenarios. * Implement real-time data ingestion routines using messaging systems (Kafka, RabbitMQ, or equivalent services). * Build complex transformations (ETL/ELT) ensuring data consistency and quality. * Design, optimize, and manage relational databases (especially PostgreSQL). * Create and maintain DAGs in Apache Airflow for process orchestration. * Perform distributed processing using Apache Spark (PySpark). * Apply observability practices, version control, documentation, and automated testing. * Work with data tools and services in the Azure ecosystem. * Participate in agile methodologies (SCRUM or similar). **REQUIREMENTS AND QUALIFICATIONS:** * Programming Language: **Python (advanced)**. * Relational Database: **PostgreSQL** (modeling, optimization, and basic administration). * Distributed Processing: **Apache Spark / PySpark**. * Orchestration: Proven experience with **Apache Airflow**. * Building **Batch and Streaming** pipelines. * Messaging Systems: **Kafka**, **RabbitMQ**, or similar services. * Experience with **caching** strategies for performance. * Experience in the **Azure** ecosystem (ADF, ADLS, Synapse, or similar). * Version control with **Git**. * Best practices and automated testing applied to data pipelines. * Agile methodologies (SCRUM or Kanban). **PREFERRED QUALIFICATIONS** * Knowledge of **Java** and/or **Scala**. * Understanding of distributed architecture, reactive programming, and microservices. * Experience with messaging technologies (Kafka, RabbitMQ, Event Hubs/Service Bus). * Infrastructure as Code: **Terraform** or ARM. * Knowledge of **Docker**. OUR BENEFITS * Meal/Voucher Allowance * Health Insurance (Sulamérica) * Dental Insurance * Birthday Day Off * Life Insurance * Remote Work Employment Type: Full-time CLT Benefits: * Medical Assistance * Dental Assistance * Life Insurance * Food Allowance * Meal Allowance Selection Question(s): * What is your expected salary (CLT)? * Are you available to work remotely? Work Location: Remote