Senior Data Engineer

Description

**What we are looking for?** **Main responsibilities:** Support the development of data storage and sharing systems; Create and maintain mechanisms that provide insight into data and metadata; Perform data monitoring; Build and implement Data Warehouse and Datalake architectures; Lead as a reference for Data Analytics technical solutions; Support technical definitions of the Big Data ecosystem; Support the implementation of improvements in architecture and processes to enhance performance, monitoring, and evolution of data products. **Requirements:** **About the opportunity** We are looking for a **Data Engineer** to work on the construction and evolution of a mission-critical data platform, responsible for processing large volumes of data in real time, ensuring high availability, scalability, and quality. You will be part of an engineering team responsible for developing critical business pipelines, using modern technologies from the Apache Spark and Databricks ecosystem, always with a focus on operational reliability, observability, and technical excellence. **Responsibilities** * Develop and maintain high-availability batch and streaming data pipelines. * Design scalable solutions using Spark, Scala, and PySpark. * Develop distributed applications on Databricks. * Implement real-time data ingestion, transformation, and availability pipelines. * Work with Lakehouse architecture using Unity Catalog for data governance. * Integrate different data sources, including MongoDB and other transactional systems. * Work in cloud environments, with a focus on Microsoft Azure services. * Ensure high performance, resilience, and scalability of data engineering solutions. * Implement monitoring, observability, and alerting mechanisms for critical pipelines. * Define and evolve data quality controls, ensuring consistency, completeness, integrity, and reliability of information. * Act in the identification and resolution of incidents in critical environments. * Collaborate with architects, data scientists, analysts, and other technology teams in building data-driven solutions. * Promote best practices in development, testing, versioning, and CI/CD. **Mandatory Requirements** * Solid experience in Data Engineering. * Proficiency in Apache Spark. * Experience with Scala. * Experience with PySpark. * Practical experience with Databricks. * Knowledge of Unity Catalog. * Experience with MongoDB. * Development of real-time data pipelines (Streaming). * Experience with distributed processing of large volumes of data. * Experience in high-availability environments and critical systems. * Experience with **Microsoft Azure**, including services for data and analytics. * Knowledge of Lakehouse architecture. * Experience with Git and versioning practices. **Nice to have** * Experience with AWS. * Knowledge of Delta Lake. * Experience with Apache Kafka or other streaming platforms. * Knowledge of event-driven architectures. * Experience with DataOps. * Knowledge of Data Observability. * Microsoft Azure certifications. * Experience in financial or mission-critical environments. **What we expect** We are looking for a person with strong technical capability, a sense of ownership, and a focus on operational excellence. We expect someone committed to: * Data quality as a priority. * Continuous monitoring and observability of pipelines. * High availability and resilience of solutions. * Performance and scalability. * Automation and continuous improvement. * Best practices in software and data engineering. * Collaboration, proactivity, and knowledge sharing. **Technology Stack** **Mandatory** * Microsoft Azure * Databricks * Apache Spark * Scala * PySpark * Unity Catalog * MongoDB * Delta Lake * Git * Streaming (Real Time) * Lakehouse Architecture * Data Quality * Data Observability * CI/CD * Apache Kafka * Event-driven architectures **Desirable** * AWS * DataOps **Position location: remote** **Who is Insi?** We are a consultancy specialized in digital evolution. But above all, we are a people company. People who listen before proposing. Who go to the essence before presenting a solution. Who measure success by what truly changed, not by what was implemented. With more than three decades of history and global presence, we believe that true transformation starts from within: within people, teams, and organizations — and it is this conviction that guides every choice we make, every project we deliver, and every person we develop. **Why build your career at Insi?** Because here you think, decide, and create together. We offer autonomy, clear goals, and an environment where Insiders have space to interact with different technologies, participate in challenging projects, bring new ideas, and work from anywhere in Brazil and (why not?) the world. We are one of the best companies to work for in Brazil according to Great Place to Work and we are among the top 10 best employers in Brazil according to TIME magazine. **What are our values?** * We are people serving people * We think and act as owners * We have a drive for performance * We grow and learn together * We seek excellence and simplicity * We act with creativity and innovation All people are welcome regardless of their condition, disability, ethnicity, religious belief, sexual orientation, appearance, age, or similar. We want you to grow with us in a welcoming environment full of opportunities. Did you identify with this? Then, **\#ComeBeInsi!**