Senior Data Engineer

Description

Job Summary: You will be responsible for designing and enhancing robust, reliable, and scalable data pipelines, ensuring ingestion, transformation, and orchestration of large volumes of data. Key Highlights: 1. Big data analytics experience in the automotive market 2. Development and maintenance of robust data pipelines 3. Working with cutting-edge technologies in an agile environment **COME MEET CARBIGDATA!** We are the largest big data platform in the vehicle market. Our products focus on locating stolen or hijacked vehicles and verifying the true status of located assets. Specializing in big data analytics, we serve major players in the automotive industry—including banks, finance companies, rental agencies, and insurance providers. **JOB DESCRIPTION** Your mission is to design and enhance robust, reliable, and scalable data pipelines. Day-to-day responsibilities include ingesting, transforming, and orchestrating large volumes of data—ensuring availability, performance, and quality for analytics, products, and machine learning models. **DAILY RESPONSIBILITIES:** * Developing and maintaining data pipelines for Batch and Streaming scenarios. * Implementing real-time data ingestion routines using messaging systems (Kafka, RabbitMQ, or equivalent services). * Building complex transformations (ETL/ELT) to ensure data consistency and quality. * Modeling, optimizing, and managing relational databases (especially PostgreSQL). * Creating and maintaining DAGs in Apache Airflow for process orchestration. * Distributed processing using Apache Spark (PySpark). * Applying observability practices, version control, documentation, and automated testing. * Working with data tools and services within the Azure ecosystem. * Participating in agile methodologies (SCRUM or similar). **REQUIREMENTS AND QUALIFICATIONS:** * Programming language: **Python (advanced)**. * Relational Database: **PostgreSQL** (modeling, optimization, and basic administration). * Distributed Processing: **Apache Spark / PySpark**. * Orchestration: Proven experience with **Apache Airflow**. * Building **Batch and Streaming** pipelines. * Messaging: **Kafka**, **RabbitMQ**, or similar services. * Experience with **caching strategies** for performance optimization. * Experience with the **Azure** ecosystem (ADF, ADLS, Synapse, or similar). * Version control with **Git**. * Best practices and automated testing applied to data pipelines. * Agile methodologies (SCRUM or Kanban). **PREFERRED QUALIFICATIONS:** * Knowledge of **Java** and/or **Scala**. * Understanding of distributed architecture, reactive programming, and microservices. * Experience with messaging technologies (Kafka, RabbitMQ, Event Hubs/Service Bus). * Infrastructure as Code: **Terraform** or ARM. * Familiarity with **Docker**. OUR BENEFITS * Meal/Voucher Allowance * Health Insurance (Sulamérica) * Dental Insurance * Birthday Leave * Life Insurance * Remote Work Employment Type: Full-time CLT Benefits: * Medical Assistance * Dental Assistance * Childcare Allowance * Remote Work * Meal Voucher Selection Question(s): * What is your expected monthly salary (CLT)? * Are you available to work remotely? Work Location: Remote