Data Engineer (Azure/Databricks) | SR (Remote)

Description

. **Responsibilities and duties** * Construction and maintenance of scalable and reliable data pipelines, using modern distributed processing technologies to ensure the ingestion, transformation, and availability of data with quality and performance; * Develop data modeling, using Databricks, SAP Datasphere, Data Factory, Azure Data Lake Storage (ADLS), Python and DB (SQL Server, Oracle). * Data extraction and creation of data lake of data and tables; * Structure RAG of the contracts and other documents; * Perform ETL from the CXL APIs and transform\-them and process\-them in a database in Databricks; * Structure data tables in Databricks; * Development of ETL for analysis of structured and unstructured data, using API for file analysis and data validation; **Requirements and qualifications** **Requirements:** * Experience and advanced knowledge in Databricks; * Experience with Data Factory, Azure Data Lake Storage (ADLS); * Experience in the implementation of Data Lake and Data Lakehouse architectures; * Experience with Python, Apache Spark; * Experience with Airflow; * Experience with SQL/NoSQL databases (PostgreSQL, MongoDB, Cassandra); **Desirable:** * Knowledge in Kafka; * Knowledge in DBT; * Knowledge in AWS Glue, BigQuery; * Knowledge in AI/RAG; * Knowledge in SAP Datasphere; Compass UOL is a global firm and part of the AI Revolution Company, together transforming organizations using Artificial Intelligence, Generative AI, and other of today’s most advanced technologies. We equip our team with proprietary and external AI\-driven tools to design and build digital\-native platforms, integrating cutting\-edge technologies and enabling companies to innovate, transform their businesses, and drive success in their markets. To achieve this, we attract and develop the best talent, creating opportunities that enhance people’s lives and highlight the positive impact of disruptive technologies. We empower borderless talent and promote knowledge and opportunities in the latest market trends, driving significant personal and professional growth. **Join us and be part of the AI\-driven revolution.**