Senior Data Engineer (Python | AWS | Spark)

Description

Job Summary: We are seeking a Senior Data Engineer to drive the evolution of our data platform, architecting robust pipelines and ensuring availability and performance for critical operations. Key Highlights: 1. Proficiency in Python, AWS, and Spark in both modern and traditional data environments 2. Development and orchestration of scalable data pipelines 3. Modeling and optimization of cloud-based Data Warehouses using AWS Redshift ### **About SRM Asset** We are one of Brazil's largest independent asset management firms, leaders in Credit Rights Investment Funds (FIDCs), driven by innovation. With an ecosystem integrating Asset Management, Ventures, and Technology, we operate with the agility of a fintech and the stability of a major financial institution. If you seek an environment where your data directly impacts real-time investment and credit decisions, this is the place for you. ### **The Challenge** As a **Senior Data Engineer** (Python \| AWS \| Spark), you will be a key player in evolving our data platform. We seek a specialist capable of seamlessly navigating between modern data engineering (AWS/Spark) and traditional relational environments (MSSQL). You will architect robust pipelines, ensuring availability, quality, and performance to support our critical operations and analytical models. ### **Day-to-Day Responsibilities** * Develop, orchestrate, and monitor scalable data pipelines using **Python** and **Apache Airflow**. * Process large volumes of data (Batch/Streaming) using **Apache Spark**. * Model and optimize cloud-based Data Warehouses, with a focus on **AWS Redshift**. * Support and modernize legacy data loads, ensuring integration between the **MSSQL Server** environment and the Data Lake. * Write complex and optimized SQL queries for data transformation and analysis. * Collaborate with Data Scientists and Business Analysts to democratize access to information. ### **What We’re Looking For (Mandatory Requirements)** * **Python Proficiency:** Solid experience developing ETL/ELT scripts and manipulating data. * **Advanced SQL:** Ability to write complex, high-performance queries and perform query tuning. * **Big Data Ecosystem:** Practical experience with **Apache Spark** and orchestration via **Airflow**. * **AWS Cloud:** Hands-on experience with AWS data services, with deep expertise in **Redshift** (S3, Glue, and Lambda are bonuses). * **Engineering Mindset:** Knowledge of best practices (Git, CI/CD, Code Review). ### **You’ll Stand Out If You Have (Desirable)** * Strong background in the Microsoft ecosystem: **MSSQL Server**, **SSIS** (Integration Services), **SSAS** (Analysis Services), and **SSRS** (Reporting Services). + *Note: We highly value professionals familiar with the "legacy" data stack who can integrate or migrate it to the cloud.* * Prior experience in the financial, credit, or payments industry. ### **Benefits** * Health Insurance: Bradesco Saúde – QN10 Bradesco National Apartment Plan, no monthly premium for the primary insured. Dependents’ premiums vary by age group. * Dental Insurance: Bradesco, covering orthodontics; * Travel Insurance, including international coverage; * Beep Saúde – at-home vaccinations and medical exams; * Up to 65% discount at Raia Drogasil; * TEA Support Program, offering specialized therapies and personalized care; * Funeral Assistance; * Meal Allowance: BRL 48.00 per working day, loaded onto the flexible Swile card; * TotalPass, with option to include up to 3 dependents; * Partnership with Sesc; * Daycare Assistance, per union agreement; * Birthday Leave, available after 6 months of employment; * Internal learning path with courses available on the company’s internal platform. * Profit Sharing (PLR)