Senior Data Engineer – Machine Learning

Description

Job Summary: The Machine Learning Engineer will be responsible for developing, implementing, and managing ML models in production, with a focus on pipeline productionization and optimization. Key Highlights: 1. Focus on ML pipeline productionization and optimization. 2. Use of MLflow for model lifecycle management. 3. Experience with Databricks for building large-scale pipelines. Job Description **Responsibilities:** The Machine Learning (ML) Engineer will be responsible for developing, implementing, and managing machine learning models in production environments, ensuring scalability, reliability, and performance of solutions. This professional will serve as the focal point for model productionization, focusing on creating and optimizing ML pipelines on the Databricks platform, using MLflow as the primary tool for tracking, managing, and deploying models. Additionally, the ML Engineer will maintain the DevOpsML pipeline, ensuring agile practices and efficient continuous integration between ML models and the production environment. Key Responsibilities: Collaborate with data scientists to transform experimental models into productive and scalable machine learning solutions. Use MLflow to track experiments, manage models, and facilitate deployment in production environments on Databricks. Implement and optimize machine learning pipelines within a DevOps environment, ensuring continuous integration and automation of the ML workflow. Maintain and enhance the DevOpsML pipeline to support the full development lifecycle—from experimentation to production monitoring. Establish and implement monitoring and evaluation metrics for deployed models, aiming to maintain performance and detect potential drifts. Diagnose and resolve issues related to ML model operations in production, adjusting processes to ensure model robustness and efficiency. Collaborate with data engineering, data science, and operations teams to ensure interoperability and integrity of machine learning solutions. Define and apply best practices for governance, security, and compliance regarding the ML pipeline, aligning with the company’s DevOps and MLOps standards. Requirements: Experience with MLflow for managing the machine learning model lifecycle. Experience with Databricks for building large-scale ML pipelines. Knowledge of DevOps applied to machine learning (DevOpsML), with emphasis on creating continuous integration and continuous delivery (CI/CD) pipelines. Experience in monitoring models in production and in techniques for maintaining model performance. Proficiency in Python, SQL, and other data manipulation and machine learning tools and languages. Strong communication and teamwork skills, with a focus on collaboration among data engineers, data scientists, and operational teams. Bachelor’s degree or higher in a relevant field. **Preferred Qualifications** Certifications in Azure or other cloud platforms such as: AI-900, DP-900, DP-100, AI-102. Databricks certifications. Experience collaborating with cross-functional teams to deliver complex data and AI solutions. **Additional Information:** 2 days per week at the company’s headquarters located in Cidade Monções, São Paulo – SP. 3 days remote work.