. **Responsibilities and Assignments** * Collaborate on building the Enterprise Lakehouse platform on Databricks, structured by data domains aligned with Data Mesh principles. * Implement and administer Unity Catalog as the central governance layer and enable cross-domain data sharing via Delta Sharing. * Develop data ingestion patterns (streaming and batch) and the medallion architecture template (bronze/silver/gold) for use by domains. * Develop the data contract framework and the Data Quality framework for the platform. * Structure CI/CD pipelines for the Databricks platform and infrastructure automation. * Integrate the Databricks platform with the existing AWS Data Mesh product, ensuring catalog interoperability and adherence to the data contract model. * Contribute to architectural decisions, technical documentation, and team mentoring. **Requirements and Qualifications** * Databricks: solid hands-on experience with the platform — workspace administration, clusters, jobs, and workflows; * Unity Catalog: data governance, permission model, lineage, external locations, and storage integration; * Delta Lake and medallion architecture (bronze, silver, gold) in production environments; * Data ingestion patterns: scalable streaming and batch processing; * CI/CD for Databricks (Databricks Asset Bundles, Repos, integration with pipelines such as GitHub Actions, Azure DevOps, or similar); * AWS ecosystem: S3, Glue, Lake Formation, EMR (EC2 and Serverless), Athena, Lambda, IAM, DMS, Kinesis, Step Functions, SNS, SQS, and EventBridge; * IaC (Terraform, including the Databricks provider); * Advanced Python and PySpark for pipeline development and platform automation; * Advanced SQL; * Data Contracts and metadata tools such as OpenMetadata; * Datadog: for observability controls and monitoring; * Desirable: * Experience in Data Mesh projects (domains, data products, federated governance) — a significant differentiator; * Experience in the financial or credit sector — a significant differentiator; * Observability and monitoring of data platforms (system tables, job metrics, data quality); * FinOps: monitoring, cost allocation, and optimization in Databricks (DBUs, cluster sizing, compute policies); * Delta Sharing in cross-company, cross-cloud, or cross-region scenarios; * AWS certifications (e.g., Data Engineer Associate, Solutions Architect); * Databricks certifications (Data Engineer Associate, Data Engineer Professional, Platform Administrator); * Open data contract specifications (Open Data Contract Standard, ODPS). Compass UOL is a global firm and part of the AI Revolution Company, together transforming organizations using Artificial Intelligence, Generative AI, and other of today’s most advanced technologies. We equip our team with proprietary and external AI-driven tools to design and build digital-native platforms, integrating cutting-edge technologies and enabling companies to innovate, transform their businesses, and drive success in their markets. To achieve this, we attract and develop the best talent, creating opportunities that enhance people’s lives and highlight the positive impact of disruptive technologies. We empower borderless talent and promote knowledge and opportunities in the latest market trends, driving significant personal and professional growth. **Join us and be part of the AI-driven revolution.**
João Silva
Indeed · HR
João Silva
Indeed · HR
