Data Engineer

Description

Job Summary: A professional to build and monitor data pipelines, model data in a lakehouse, contribute to architecture, ensure data observability and quality, and advance AI practices. Key Highlights: 1. Strong expertise in building and maintaining reliable data pipelines. 2. Active contribution to architecture decisions and tool adoption. 3. Opportunity to work with and evolve AI tools. **Your key deliverables:** ----------------------------- * Build, maintain, and monitor data pipelines (incremental ingestion/CDC, processing, publishing) reliably and idempotently. Model data across lakehouse layers (raw, curated, consumption) and write transformations in a distributed query engine. * Contribute to architecture decisions and tool adoption. * Ensure data observability, quality, and governance (tests, alerts, incident response). * Translate business team requirements into data modeling and consumption patterns. * Advance the team’s AI standards, documentation, and tooling. **Requirements:** --------------- * Strong Python and SQL (modeling, optimization, execution plans). * Pipeline orchestration (Airflow or similar) and batch processing. * Cloud platforms (AWS preferred; GCP/Azure equivalents accepted). * Relational databases (Postgres/MySQL) and data warehouse / lakehouse modeling. * Git and code review practices. * Autonomy to unblock problems with incomplete specifications, without relying on fully detailed scopes. * Willingness to work with and on AI tools within the engineering workflow (our development is heavily augmented by AI, and the team builds/evolves its own agent harness). **Nice-to-haves:** ----------------- * Lakehouse / open table formats (Iceberg, Delta, Hudi) and distributed engines (Trino, Spark). * Polars, Glue, Docker/Kubernetes. * Messaging systems and event-driven architecture / CDC (Kafka, Debezium or similar). * Data governance and observability (data quality, lineage, catalog). * Experience with agentic automations / LLMs in the workflow. **Benefits:** --------------- * Caju Card, offering greater flexibility in using your benefits. * + Wellhub, extendable to legal dependents. + Dental Plan. * SulAmérica Life Insurance, providing security for you and your family.