SR Data Engineer - Python

Description

Summary: Zallpy is seeking a curious, collaborative individual to build and optimize complex data pipelines, implement the Databricks ecosystem, and develop ingestion engines for AI agents. Highlights: 1. Opportunity to work with Databricks, Airflow, dbt, and AI development tools 2. Focus on building AI agents and transforming infrastructure data 3. Collaborative environment with emphasis on learning and growth Here at Zallpy, technology is born from curious, collaborative people with a desire to do things well. If you like environments where there's room to ask questions, test, learn, and evolve together, check out this opportunity! **Diversity, Equity, and Inclusion** At Zallpy, we value an environment where everyone can be themselves, with respect, safety, and a sense of belonging. We especially encourage applications from people with disabilities and guarantee an accessible environment with adaptations and support throughout the journey. We also encourage Black people, women, LGBTQIA\+ individuals, and professionals from diverse backgrounds to apply. Here, you can be yourself at every stage of your journey. Come join us. We are looking for someone to work in a team that thinks big, works with attention to detail, collaborates daily, and shares responsibility. We believe in autonomy, the exchange of different experiences, and solutions built together. Here, growth means sharing knowledge, respecting different ways of thinking, and building trusting relationships while delivering quality technology. If you identify with this way of working, come join us. **What you will do:** * Build and optimize complex pipelines using Databricks Workflows, Airflow, or dbt to transform infrastructure data into agent\-ready intelligence. * Implement the Databricks ecosystem (Unity Catalog, Delta Lake, MLflow) as the core foundation for AI agents. * Build ingestion engines for Qdrant, focusing on vector quantization and metadata\-rich payload enrichment. * Use AI development tools (Cline/Claude Code) to rapidly iterate on ETL logic and automate data validation. * Deliver documentation for data contracts and lineage in Confluence, providing the "instruction manual" for agents. **What we are looking for:** * Fluent in English. * Expert proficiency in Databricks (Delta Lake, Unity Catalog) and Apache Spark. * Hands\-on experience with Qdrant (indexing, filtering, and hybrid search). * Expert SQL and Python (PySpark, Pandas). * Mastery of Airflow, dbt, and stream processing frameworks (Spark Streaming/Flink). **Nice to have:** * Previous experience with AI Development. **Where you will work:** * This is a hybrid position, working from our office in Acate/Florianópolis. **Hiring models:** * CLT * Cooperative **Benefits of being a Zallper:** Meal/Food Allowance Caju multi\-benefit card (credit format), offering flexibility for everyday needs. (CLT only) \* Health Plan – Unimed No waiting period or copayment for the holder. Possibility to include dependents (children, spouse, or stepchildren) with copayment. (CLT only) \* Dental Plan – Uniodonto Affordable options to include dependents. (CLT only) \* Wellhub Access to gyms, physical activities, and wellness programs. Zenklub For CLT: Two free sessions per month and special rates for additional sessions. For Cooperative/Contractor: Special rates for sessions. Life Insurance More security for you and your family. (CLT only) \* Childcare Allowance Financial support for Zallpers with children from 4 months to 6 years old, according to internal policy. (CLT only) \* Baby Zallpy A special gift to celebrate the arrival of new Zallpy babies. Business Partner Support Close and human\-centered support from our People \& Culture team throughout your journey at Zallpy. Volunteer Internal Communities Diversity, Sports \& Movement, and Technology. Educational Partnerships Discounts on undergraduate and graduate programs, professional courses, and language schools. ✈️ Experiences \& Development Participation in events, workshops, trips, and team\-building activities. Referral Program Refer talents to Zallpy and receive bonuses from R$2,500 to R$5,000\. \* Contractor / PJ: For these hiring models, the benefits above apply except for those marked with an asterisk (\*), which are exclusive to CLT. In addition, we offer dedicated platforms to manage contractors, ensuring organization, transparency, and a structured experience throughout the entire journey. **Job ID:9020**