Senior Data Engineer (Python, Spark, Databricks)

Description

Summary: Join Solvd Inc. as a Senior Data Engineer to build, maintain, and optimize data ingestion pipelines and develop data processing workflows for a scalable backend platform. Highlights: 1. Shape real-world AI-driven projects across key industries. 2. Be part of a global team with equal opportunities for collaboration. 3. Thrive in an inclusive environment that prioritizes continuous learning. Application *Solvd Inc. is a rapidly growing AI\-native consulting and technology services firm delivering enterprise transformation across* ***cloud, data, software engineering, and artificial intelligence****. We work with industry\-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.**Following the acquisition of* ***Tooploox****, a premier AI and product development company, Solvd now offers true* ***end\-to\-end delivery****—from strategic advisory and solution design to custom AI development and enterprise\-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector\-specific knowledge to address complex business challenges quickly and effectively.* We are looking for a **Senior Data Engineer (Python, Spark, Databricks)** to join our team. **Project description** The project consists of building a scalable backend platform that runs data\-driven analytical models in a controlled, automated, and auditable way. The backend must orchestrate model execution, enforce data validation, manage metadata and artefacts, and ensure full traceability of every processing step. The architecture is modular and cloud\-oriented, with a strong focus on automation, versioning, and operational reliability. **What you'll do** * Build, maintain, and optimize data ingestion pipelines using Python, pandas, Spark, and Databricks * Parse and transform complex Excel\-based financial models into structured Delta Lake tables * Implement data processing workflows and calculation logic in Python * Develop and manage pipelines using Databricks Jobs, clusters, and SQL Warehouse * Improve performance, scalability, and reliability of existing data pipelines * Write clean, maintainable, and production\-quality Python code * Use SQL for data validation, transformation, and pipeline integration **What you bring** * Senior\-level engineer with 5–8\+ years of experience * Strong Python skills, with proven experience building production data pipelines * Hands\-on experience with pandas for data transformation and ingestion * Experience working with Spark (PySpark or Scala) and Databricks * Solid SQL skills and experience writing complex queries * Experience designing, implementing, and maintaining data processing pipelines * Experience working with medium to large datasets * Strong ownership and ability to work hands\-on in a data platform environment * English fluency for daily communication with actuarial and engineering teams. **Nice to have** * Experience optimizing Python or Spark workloads for performance and scalability * Experience working with Delta Lake * Experience ingesting and transforming large Excel\-based datasets * Experience working with financial or structured modeling data **Tech stack** * Core: Python 3\.x (pandas, numpy, pydantic, mlflow, databricks\-sdk) * Data Layer: PostgreSQL, Azure Blob Storage, Parquet * Computation: Databricks Jobs / MLflow Projects * Integration: NestJS API * CI/CD \& Infra: GitHub Actions, Terraform **When you join Solvd, you'll…** * Shape real\-world AI\-driven projects across key industries, working with clients from startup innovation to enterprise transformation. * Be part of a global team with equal opportunities for collaboration across continents and cultures. * Thrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards. **Ready to make an impact?** If you're excited to build things that matter, champion responsible AI, and grow with some of the industry’s sharpest minds. Apply today and let’s innovate together.