Data Engineer | Azure & Databricks

Description

Job Summary: We are seeking a Data Engineer to build and optimize data pipelines in PySpark/Databricks and Azure, focusing on data ingestion, transformation, and governance. Key Highlights: 1. Building and optimizing data pipelines in PySpark/Databricks and Azure 2. Focus on data ingestion, transformation, governance, performance, quality, and security 3. Collaboration with business and analytics teams to deliver reliable datasets We are seeking a Data Engineer to build and optimize data pipelines in PySpark/Databricks, leveraging Azure services. The role involves projects related to data ingestion, transformation, and governance, with emphasis on performance, quality, and security. Hybrid work location.\* Develop and maintain data pipelines in PySpark/Databricks * Write and optimize SQL for large-scale datasets * Work on ingestion, transformation, and governance projects using Azure Data Lake and Data Factory * Ensure data performance, quality, and security * Collaborate with business and analytics teams to deliver reliable datasets\* Practical experience with SQL, PySpark, Databricks, and Azure * Knowledge of ADF, Data Lake, and data modeling best practices * Experience with code versioning (Git) and agile methodologies * Nice-to-have: Delta Lake, Unity Catalog, CI/CD, and data architecture (Lakehouse, Data Mesh) Refer to Job description section for qualification required