AWS Data Engineer | Senior (Remote)

Description

Job Summary: Design, develop, and maintain cloud-based data pipelines, implementing robust and scalable ETL/ELT processes and monitoring data quality. Key Highlights: 1. Working with cutting-edge AI and Generative AI technologies 2. Optimizing and managing Databricks clusters 3. Developing solutions with PySpark and ML pipelines . **Responsibilities and Duties** * Design, develop, and maintain cloud-based data pipelines (Azure and/or AWS) with recurring execution (e.g., every X hours, daily, weekly). * Implement robust and scalable ETL/ELT processes handling large data volumes and complex transformations, primarily using Spark. * Develop distributed processing solutions with PySpark on Databricks. * Monitor pipelines to ensure data quality, consistency, and availability, including implementing alerts, recovery strategies, and backfill executions. * Manage, configure, and optimize Databricks clusters, focusing on performance, scalability, availability, and cost control. * Integrate multiple data sources, including APIs, relational and non-relational databases, and streaming data. * Perform data modeling and define table structures for different layers (raw, curated, analytics, feature). * Develop and maintain ingestion and orchestration pipelines using tools such as Azure Data Factory and Airflow. * Implement near real-time data ingestion using Kafka and CDC (Change Data Capture) concepts. **Requirements and Qualifications** * Python programming for data processing and automation; * Experience with Azure and/or AWS applied to data engineering; * Knowledge of Azure Data Factory (ADF) / Airflow or equivalent tools; * Solid understanding of Databricks, including jobs, clusters, and optimization strategies; * Intermediate knowledge of consuming and integrating REST APIs (Databricks, Azure, and AWS services); * Experience building feature pipelines for Machine Learning models (feature engineering and feature stores); * Basic understanding of data modeling tailored for Machine Learning. Compass UOL is a global firm and part of the AI Revolution Company, together transforming organizations using Artificial Intelligence, Generative AI, and other of today’s most advanced technologies. We equip our team with proprietary and external AI\-driven tools to design and build digital\-native platforms, integrating cutting\-edge technologies and enabling companies to innovate, transform their businesses, and drive success in their markets. To achieve this, we attract and develop the best talent, creating opportunities that enhance people’s lives and highlight the positive impact of disruptive technologies. We empower borderless talent and promote knowledge and opportunities in the latest market trends, driving significant personal and professional growth. **Join us and be part of the AI\-driven revolution.**