Job Summary: A professional to develop and maintain scalable data pipelines, handle large volumes of data, and implement integrations and workflow automation. Key Highlights: 1. Develop and maintain scalable and high-performance data pipelines. 2. Work with large volumes of data, ensuring quality and integrity. 3. Collaborate with technical and business teams to deliver efficient solutions. **Responsibilities:** * Develop and maintain scalable and high-performance data pipelines. * Work with large volumes of structured and unstructured data, ensuring quality, integrity, and availability. * Implement integrations via APIs (REST and/or messaging), including data transformation and workflow automation. * Develop distributed processing solutions and optimize queries. * Automate deployments and resource provisioning using infrastructure-as-code principles. * Orchestrate data workflows, ensuring monitoring, versioning, and observability. * Collaborate with technical and business teams to translate requirements into efficient data engineering solutions. **Technical Requirements (Hard Skills)** * Solid experience in Python for data manipulation, automation, and integrations. * Solid experience in advanced SQL, including query optimization and data modeling. * Knowledge of distributed and parallel processing, applying cluster computing concepts. * Experience with data architecture in cloud environments (preferably AWS). Data integration via APIs, authentication, error handling, and integration patterns. * Code versioning and automation practices (Git and DevOps/DataOps). * Familiarity with infrastructure-as-code and automated provisioning. * Knowledge of pipeline orchestration and governance of process execution. * Familiarity with distributed query engines and large-scale processing optimization. * Experience with data workflow orchestration using tools such as Airflow or similar (not mandatory but desirable). **Desirable Knowledge (Tools)** Not mandatory, but considered advantageous for the full-level position: * Version control and CI/CD platforms (e.g., GitLab); * Data processing and cluster tools (Databricks, Spark); * Data lake platforms and distributed SQL engines (Trino, Dremio); * Pipeline orchestrators (Airflow); * Cloud solutions (AWS) and infrastructure-as-code (Terraform); * API integration and consumption; Prior experience with these environments will be considered advantageous, but we seek professionals with a strong technical foundation and learning capability.
João Silva
Indeed · HR
João Silva
Indeed · HR
