Data Engineer - Teresina/PI

Description

Job Summary: Design, develop, and maintain efficient and scalable data pipelines, ensuring orchestration and optimization of data systems in a cloud environment. Key Highlights: 1. Expertise in ETL/ELT using Python and advanced SQL 2. Solid experience with Azure and data architecture (Data Warehouse, Lake) 3. Proficiency in process orchestration using Airflow and Git/GitHub Description: * Bachelor's degree in Computer Science, Computer Engineering, Data Science, or related fields (completed or currently pursuing) is preferred; * Required Knowledge: * ETL/ELT: Python (pandas, PySpark) and advanced SQL. * Databases: relational data modeling and query optimization. * Orchestration: hands-on experience with Airflow (DAGs). * Data Architecture: experience with Data Warehouse, Data Lake, and Lakehouse. * Cloud: solid experience with Azure; familiarity with AWS and GCP is a plus. * Version Control: Git/GitHub. * Integrations: experience handling data from various formats and REST APIs. * Write SQL queries to extract information from databases, ensuring reports are generated accurately and on-demand; * Maintain and optimize SQL queries to meet new requirements and ensure proper functioning of systems and reports, on-demand; * Design, develop, and maintain efficient and scalable data pipelines for collection, processing, transformation, and storage of large volumes of data; * Apply Python programming knowledge for ETL processes, ensuring maintenance and continuity of the existing infrastructure; * Develop, maintain, and optimize efficient and scalable data pipelines for collection, processing, transformation, and storage of large volumes of data (ETL/ELT), using Python and SQL. * Monitor the health and performance of data systems, performing adjustments and improvements as needed; * Ensure compliance with data protection regulations and information security standards; * Document data processes, workflows, and architecture clearly and accessibly; * Contribute to database modeling to optimize performance and scalability; * Apply Airflow expertise to ensure efficient orchestration of existing processes; * Extract information from diverse data sources such as databases, spreadsheets, XML, JSON, etc.; * Implement and maintain cloud-based solutions, focusing on Azure and complemented by knowledge of AWS and GCP. * And other duties relevant to this position. 2511070202301792388