Scala Data Engineer | Senior

Description

Job Summary: A data professional responsible for creating, monitoring, and optimizing data pipelines, ensuring security and compliance, and integrating innovative solutions. Key Highlights: 1. Data pipeline creation and management with Airflow/Spark 2. Focus on performance tuning, monitoring, and debugging of pipelines 3. Working for a global company at the forefront of the AI Revolution . **Responsibilities and Duties** * Build data pipelines for data ingestion using Airflow/Spark; * Implement access controls and auditing to protect sensitive data and ensure regulatory compliance; * Migrate and integrate existing data onto the new platform, ensuring operational continuity and data integrity; * Performance Tuning: Ability to optimize data pipeline performance by adjusting parameters and configuring resources correctly; * Monitoring and Debugging: Ability to monitor and debug pipelines to identify and resolve bottlenecks and performance issues; * Apache Spark, Apache AirFlow, Apache Iceberg, AWS: EMR; AWS: Glue; AWS: S3, AWS: Lake Formation, Data Pipelines, Python. Proficiency in building and managing data pipelines, scheduling and monitoring ETL tasks. **Requirements and Qualifications** * Apache Spark; * Apache AirFlow; * Apache Iceberg; * AWS: EMR, Glue, S3, Lake Formation; * Data Pipelines; * Python; * Scala or Java; * Proficiency in building and managing data pipelines, scheduling and monitoring ETL tasks; * Data Governance and Security; * Monitoring and Debugging. Compass UOL is a global firm and part of the AI Revolution Company, together transforming organizations using Artificial Intelligence, Generative AI, and other of today’s most advanced technologies. We equip our team with proprietary and external AI\-driven tools to design and build digital\-native platforms, integrating cutting\-edge technologies and enabling companies to innovate, transform their businesses, and drive success in their markets. To achieve this, we attract and develop the best talent, creating opportunities that enhance people’s lives and highlight the positive impact of disruptive technologies. We empower borderless talent and promote knowledge and opportunities in the latest market trends, driving significant personal and professional growth. **Join us and be part of the AI\-driven revolution.**