Data Engineering Specialist Professional

Description

The first thing you need to know is that you will not fall into a routine here. Radix develops solutions for companies in different sectors and industries. Each project has its technologies, solutions, and deadlines, and you will have the opportunity to work on different challenges in engineering, data, analytics, and digital transformation. In addition to our operations in Brazil, with offices in Rio de Janeiro, São Paulo, and Belo Horizonte, we also have branches in the United States, consolidating Radix increasingly as a global company. Do you want to be part of this story and turn ideas and dreams into reality? **About the position** ---------------- We are looking for a Data Engineering Specialist with strong experience in Data Platforms such as Databricks, Snowflake, Palantir, or similar, to work on the design, implementation, optimization, and maintenance of modern data solutions based on the Lakehouse architecture. This person will be responsible for building scalable, reliable, and high-performance pipelines using the Databricks platform, with a focus on ingestion, transformation, quality, governance, modeling, and data availability for analytical consumption, BI, corporate integrations, and advanced data initiatives. The ideal candidate has solid experience in Databricks, Apache Spark, PySpark, SQL, Delta Lake, Unity Catalog, Workflows, Delta Live Tables, data modeling, integration via APIs, and modern Data Quality practices, in addition to the ability to work closely with technical teams and business areas. **Main responsibilities:** --------------------------------- * Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, PySpark, SQL, and Delta Lake. * Implement data architectures based on Bronze, Silver, and Gold layers, ensuring data organization, traceability, quality, and governance. * Develop data ingestion, transformation, validation, and availability processes using native Databricks platform resources, such as notebooks, jobs, workflows, Delta Live Tables, Auto Loader, and Databricks SQL. * Design and implement Data Quality frameworks in the data ingestion, transformation, and consumption layers. * Develop automated data quality rules, validations, and monitoring processes to ensure data accuracy, completeness, consistency, uniqueness, timeliness, and integrity. * Implement data profiling, cleaning, standardization, enrichment, reconciliation, and exception handling processes. * Define, implement, and monitor data quality metrics, KPIs, scorecards, and dashboards. * Develop automated validation processes for structured and semi-structured data, including JSON data from REST APIs, SaaS platforms, and corporate systems. * Design canonical data models and analytical models that consolidate information from multiple source systems. * Create and maintain source-to-target mappings, transformation logic, business rules, and data harmonization processes. * Design dimensional models, including star schemas, to support analytical workloads, reports, and BI solutions. * Implement incremental loading strategies, CDC, data reconciliation, and end-to-end validation. * Work with Unity Catalog, governance, lineage, metadata, catalogs, schemas, permissions, and role-based security. * Optimize workloads in Databricks, including notebooks, jobs, SQL queries, clusters, computational resource usage, partitioning, Z\-Ordering, Spark tuning, and performance best practices. * Investigate, diagnose, and resolve data quality, performance, availability, and integrity issues, identifying root causes and implementing preventive solutions. * Support architecture decisions related to the use of Databricks, Delta Lake, Lakehouse, governance, integration, automation, and observability. * Collaborate with data architects, data engineers, analysts, software engineers, and business stakeholders to deliver reliable, scalable, and high-quality solutions. * Support modern engineering practices, including versioning with Git, CI/CD, deployment automation, code review, and technical documentation. **Mandatory qualifications:** ------------------------------- * Fluent English for conversation. * Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent practical experience. * Extensive experience as a Data Engineer, Data Specialist, Data Architect, or similar role. * Practical and advanced experience with Databricks in corporate data engineering projects. * Solid knowledge of Apache Spark, PySpark, and Spark SQL. * Advanced knowledge of SQL, including complex queries, transformations, optimization, and performance tuning. * Experience with Delta Lake, including Delta tables, versioning, optimization, partitioning, and storage best practices. * Experience in designing and implementing Raw/Bronze, Curated/Silver, and Business/Gold layers. * Experience with pipeline and workload orchestration in Databricks, including jobs, workflows, and automations. * Experience in implementing frameworks, methodologies, and Data Quality best practices in corporate environments. * Experience in developing and implementing data quality rules, validation frameworks, reconciliation processes, and exception handling. * Strong understanding of data profiling, cleaning, standardization, enrichment, and data matching. * Practical experience in integrating and validating JSON data from REST APIs, SaaS platforms, and corporate systems. * Solid knowledge of relational database modeling, normalization, and data integrity principles. * Experience in designing canonical data models that consolidate information from multiple source systems. * Experience with source-to-target data mapping, transformations, harmonization, and business rule documentation. * Strong knowledge of dimensional modeling, star schemas, and corporate Data Warehouse concepts. * Experience in implementing incremental loads, CDC, reconciliation processes, and automated data quality validations. * Familiarity with data governance, metadata management, data lineage, master data, reference data, and role-based security. * Experience with Unity Catalog or equivalent concepts of governance, catalog, access control, and data lineage. * Proficiency in Git, CI/CD practices, and deployment automation for data engineering solutions. * Ability to work autonomously, with a critical vision, clear communication, and collaboration with technical and business teams. **It is a plus if you have:** ----------------------------------- * Databricks certification or accreditation related to architecture, such as Databricks Platform Architect Accreditation/Badge, preferably in Azure or AWS. * Databricks Certified Data Engineer Professional certification or equivalent. * Experience with Lakehouse architecture in large-scale corporate environments. * Experience with Delta Live Tables, Auto Loader, Databricks Asset Bundles, Databricks CLI, or deployment automation in Databricks. * Experience with cloud platforms, especially Azure, AWS, or GCP. * Experience with observability, monitoring, and alerting tools for data pipelines. * Experience with Databricks SQL, dashboards, SQL Warehouses, and integration with BI tools. * Knowledge of FinOps applied to Databricks, cluster optimization, serverless, jobs, and workloads. * Master's or Doctorate in Data Science, Artificial Intelligence, Engineering, Computing, or disciplines requiring high analytical capacity. **Benefits:** * National Medical Assistance (for the holder and dependents, with private room) * National dental assistance (for the holder and dependents) * Flexible meal/food voucher * Home office allowance * Day off (in the month of the birthday) * Wellhub (formerly Gympass) * Extended Maternity Leave (6 months) and Paternity Leave (20 days) * Childcare allowance for children up to 3 years old (per child) * Mental health support with Wellz * Benefits Club with discounts from various partners * Agreement with educational institutions and language courses * Professional Development (Corporate University) * Partnership with a coworking company in Brazil * Quality of Life and Well\-Being Program * Consulting doctor for monitoring radixers * Incentive plans **Radix is always at the top of the Best Companies to Work For because:** * We have committed, dedicated, curious, and innovative professionals. * Team spirit is our greatest strength. We work cooperatively and know that we are together, rowing in the same direction. * We have a diverse environment that values equity and inclusion. * Our working hours are flexible, and in almost all projects, it is possible to work from anywhere in Brazil. * We value well\-being and care for our people, with mental health support programs, psychiatrist, and consulting doctor available. **Did you like it?** **\#VemPraRadix** **Job code: c48a**