Senior Data Engineer

Description

**What are we looking for?** **Key Responsibilities:** Data Pipeline Development (ETL/ELT): Design, build, and maintain scalable, high-performance data pipelines using PySpark, AWS Glue, Databricks, and AWS EMR for large-scale data processing. Data Lake and Data Warehouse Architecture: Structure and manage data storage in AWS S3 (Data Lake) and build efficient analytical models in Redshift (Data Warehouse), ensuring data integrity, security, and governance. Workflow Orchestration: Create, monitor, and optimize DAGs in Apache Airflow to ensure reliable execution of data ingestion, transformation, and loading routines while managing complex dependencies. Real-Time and Event-Driven Processing: Develop data streaming architectures and automate lightweight, event-based processes. Data Migration and Replication: Implement Change Data Capture (CDC) routines and migrate relational and non-relational databases using AWS DMS. Data Provisioning for Analytics: Optimize and partition data in Delta to enable fast, efficient ad-hoc queries via AWS Athena, while supporting BI and Data Science teams. Monitoring and Performance: Identify bottlenecks, tune SQL queries and Spark jobs, and ensure the health of the data infrastructure. **Requirements:** Programming Languages: Advanced proficiency in Python and SQL. Big Data & Distributed Computing: Strong hands-on experience with PySpark and solid understanding of cluster operations and distributed processing optimization (Spark UI, memory management, shuffling). AWS Ecosystem: Proven practical experience configuring and using S3, EMR, Glue, Athena, Lambda, Kinesis, DynamoDB, and DMS. Databricks: Familiarity with the Databricks platform, including notebook usage, cluster management, and Data Lake integration. Google Cloud Platform (GCP): Advanced experience in modeling, partitioning, and optimizing cost and query performance in Google BigQuery. Orchestration: Experience building and maintaining pipelines in Airflow (Python Operators, Sensors, XComs). Software Engineering: Knowledge of code versioning (Git), CI/CD for data, and best practices in clean code. **Work Location:** This is an Anywhere Office position — you will work fully remotely. **Who is Insi?** We are a consultancy specialized in digital evolution. But above all, we are a people-first company. A company whose people listen before proposing solutions; go to the core before presenting answers; and measure success by what truly changed — not by what was implemented. With over three decades of history and a global presence, we believe true transformation begins from within: within people, teams, and organizations — and this conviction guides every choice we make, every project we deliver, and every person we develop. **Why build your career at Insi?** Because here, you think, decide, and create together. We offer autonomy, clear goals, and an environment where Insiders have opportunities to engage with diverse technologies, participate in challenging projects, bring new ideas, and work from anywhere in Brazil — and why not? — from anywhere in the world. We are ranked among the best companies to work for in Brazil by Great Place to Work and listed among the top 10 employers in Brazil by TIME magazine. **What are our values?** * We are people serving people * We think and act like owners * We strive for performance * We grow and learn together * We pursue excellence and simplicity * We act with creativity and innovation All individuals are welcome, regardless of condition, disability, ethnicity, religious belief, sexual orientation, appearance, age, or related factors. We want you to grow with us in an inclusive, opportunity-rich environment. If this resonates with you, then **\#JoinInsi!**