Senior Data Engineer

Description

Caju is a **Brazilian technology company** that seeks to add more flavor to professional life, transforming the relationship between companies and employees through more innovative and secure solutions such as the Multi-Benefits Card, Corporate Expenses Solution, Rewards, and Caju Cycles. At Caju, **we are always learning**, and we become better and better in a collaborative and fun environment! **Applications from Black people, women, Indigenous people, LGBTQIA\+, or other marginalized groups are very welcome.** Apply and learn more about our team **Responsibilities and attributes** * Integration, Ingestion, and Multi-Source Orchestration: Design and maintain heterogeneous integration pipelines using tools such as Apache Airflow, Dagster, Airbyte, AWS DMS, DBT, Kubernetes, REST APIs, SFTP, and various connectors for continuous ingestion. * Performance, I/O, and Cost Optimization (AWS \& Databricks): Apply advanced partitioning, indexing (*Z\-Ordering/Liquid Clustering*), file layout optimization, and read/write strategies on Amazon S3, Databricks, and dbt, aiming to eliminate excessive scans (*full table scans*), reduce S3 request costs (GET/LIST), and accelerate query consumption. * Event\-Driven \& Streaming Architecture: Design and implement real-time data ingestion via topics and events (e.g., Apache Kafka, AWS Kinesis, Event Hubs), eliminating direct dependence on queries and bulk loads in relational databases (PostgreSQL, MySQL). * AI-Powered Engineering Automation: Implement tools, agents, and GenAI/LLM pipelines to automate engineering operational tasks (e.g., automatic documentation generation, pipelines, data quality checks, SQL translation/refactoring, root cause analysis of failures, PRs). * End\-to\-End Pipelines and Modeling: Design, build, and maintain robust pipelines, applying dimensional modeling and *Medallion* architecture to deliver clean, aggregated, and optimized data in both the ingestion and consumption processes. * Governance, Security, and Quality: Ensure compliance (LGPD/GDPR), masking of sensitive data, lineage, granular access control, and proactive monitoring of *data freshness* and *data quality*. * Alignment and Documentation: Conduct requirements gathering with business areas, document architectures, define *data contracts*, and ensure pipeline governance and maintainability. **Requirements and qualifications** * Experience: Solid experience in data engineering, working in high-volume and high-criticality production environments. * I/O, Partitioning, and Performance Optimization: Practical mastery of partitioning strategies, data layout optimization (*Small File Problem*, compaction, *Z\-Ordering*), and query optimization to prevent excessive reads on Amazon S3 and ensure low response times on Databricks and dbt. * Heterogeneous Orchestration and Ingestion: Practical experience in data integration and orchestration using Airflow, Dagster, Airbyte, AWS DMS, dbt, Kubernetes, REST APIs, SFTP, and custom connectors. * Event\-Driven \& Topic-Based Ingestion: Practical experience in event/topic-based ingestion architecture (Kafka, Kinesis, Pulsar, RabbitMQ) to replace dependence on relational databases. * Main Stack: Advanced mastery of Python, PySpark, and optimized SQL for massive and distributed processing. * Databricks Platform \& Lakehouse: Deep experience in the Databricks ecosystem (Delta Lake, Auto Loader, Structured Streaming, Jobs/Workflows, Medallion Architecture). * AWS Cloud: Experience with AWS data services (S3, IAM, Redshift, DynamoDB, Data Catalog, AWS DMS). * AI Automation: Practical knowledge or strong interest in using tools/LLMs for automating development and data engineering tasks. * Software Engineering: Experience with automated testing, version control (Git/GitHub), PR reviews, and CI/CD pipelines. * Modeling and Governance: Mastery in data modeling (Dimensional, Star/Snowflake) and security and privacy guidelines (LGPD/GDPR). **Differentiators:** * AI/GenAI for Data: Experience with pipelines for AI support (ingestion, *chunking*, and *embeddings* for Vector Search, Feature Stores, governance via Unity Catalog). * Advanced Governance \& IaC: Experience with Unity Catalog, Terraform, and Databricks Asset Bundles (DAB). * Data Contracts \& Data Quality: Implementation of data contracts at the source and use of quality/observability frameworks (Great Expectations, Soda, dbt tests, Datadog). * FinOps and Observability: Proactive management and anomaly alerts for cloud/Databricks costs by table and pipeline, aligned with statistical detection of *data drift*. * Financial Pipelines: Experience with idempotency, balance reconciliation, and financial reprocessing. * Regulated Sectors: Previous experience in fintech, payment methods, or healthcare environments, dealing with data compliance and audits. **Additional information** Caju Card, with more freedom to use your benefits (Meals, Food, Mobility, Health, Home Office, Culture, and Education); Health plan without copayment (Unimed, Sulamerica, or Alice); Zenklub, with online consultations with therapists and coaches to take care of your mental health; Wellhub; We also encourage language learning, in partnership with Rosetta Stone; Recharge day \- day off; Conexa Saúde \- online medical consultation; Childcare assistance; Partnership with Alura; Remote work, so you can work from anywhere you want within Brazil; We provide work equipment; Many growth opportunities \- we have a lot to grow and strongly hope you will help us with that! Caju is a Brazilian company and accepts people from all regions of the country. **We work** **in a 100% remote model** (those interested in visiting or working in the office, located in São Paulo, will find our doors open) **Interested? \#VemSerCaju** Caju is a **technology company**, **the fruit of Brazilian entrepreneurship**, created to transform the relationship between companies and employees, with innovative and flexible solutions. We develop a platform for companies to manage various solutions in one place, and for employees, a single card with multiple possibilities. **Our Products:** **Multi-Benefits Card:** A solution for managing up to 8 categories on the same Visa-branded Caju Card. It features an easy-to-use app for the employee and an intuitive platform for HR to manage the entire service. **Caju Expenses:** A faster advance flow, management of corporate expenses (travel, office supplies, outsourced services, and other day-to-day expenses), and linked Caju cards. All on the same platform.. Goodbye reimbursement, hello pocket convenience! **Caju Rewards:** It is the solution to reward employees with an extra balance on the card, in a specific wallet. **Caju Cycles:** End-to-end tracking of the employee journey, on a single platform for onboarding and data management. **Caju Mais:** For companies that want to improve the health and well\-being of their employees, in partnership with Gympass, Conexa Saúde, and Psicologia Viva. **Feirinha da Caju:** Exclusive offers and coupons at Caju partner stores for employees and HR. We have big dreams here, and we are proud to have **roots spread across various parts of Brazil** and people who nurture and harvest fruits with a unique taste that only we have. Having difficulty completing the registration or any other problem with the Gupy platform? Access support here. **How about building our cashew tree together?** **\#VemProCajueiro**