Senior AWS Data Engineer

Description

Job Summary: A Senior Data Engineer will lead data architecture, quality, and optimization efforts, building scalable data infrastructures and integrating diverse data sources. Key Highlights: 1. Leadership in data architecture and governance 2. Building scalable and efficient data infrastructures 3. Developing data pipelines using AWS The Senior Data Engineer will be responsible for leading the definition and implementation of data architecture, ensuring data quality, security, and governance, and optimizing solution performance and scalability. This professional will build and maintain scalable and efficient data infrastructures, integrate multiple data sources, and automate data ingestion and transformation workflows. Technologies used include Airflow, DBT, DuckDB, Python, R, Scala, AWS data solutions (Kinesis, Spark EMR, Lambda, Redshift), data lakes, distributed data architecture, microservices architecture, messaging and streaming, scalability and elasticity concepts, SQL, databases and data modeling, and logging and monitoring tools. Requirements: \*\*Responsibilities:\*\* * Develop scalable data pipelines using AWS (S3, EMR, Redshift). * Process large-scale data using DBT, DuckDB, and Spark. * Participate in building dynamic DAGs via Airflow and YAML. * Discuss requirements with stakeholders and propose technical solutions. * Monitor and ensure pipeline and processed data quality. \*\*Technical Requirements:\*\* * Solid experience with AWS (S3, EMR, Redshift). * Proficiency in object-oriented Python and SQL. * Knowledge of DBT, DuckDB, and Airflow. * Experience with data architecture and pipeline interdependencies. * Experience with observability and pipeline monitoring is desirable. \*\*Soft Skills:\*\* * Proactivity and strong communication skills with requesting teams. * Collaborative mindset and technical leadership capability. * Understanding of cause-and-effect relationships in data flows. \*\*Preferred Qualifications:\*\* * Prior experience as a software engineer. * Experience in high-volume environments with specific regulatory requirements. * Familiarity with data quality tools such as SODA or Great Expectations.