Data Extraction Engineer

Description

**Specification\-Driven Extraction Engineering:** * Design and maintain declarative extraction specifications—using Pydantic models, JSON schemas, or domain\-specific languages—that describe exactly which fields to capture, their types, and validation rules. * Implement pipelines that translate these specifications into executable extraction plans, leveraging both classical (Scrapy, Playwright) and AI\-augmented (LLM\-based semantic parsing) backends. * Build reusable specification libraries for recurring data types (product prices, tariff codes, regulatory texts) to accelerate onboarding of new sources. * Design and implement autonomous data extraction **agents** that can make decisions about source selection, retry logic, and parsing strategies **Autonomous \& Self\-Healing Systems:** * Deploy self\-healing spiders that automatically detect website layout changes and repair themselves using Model Context Protocol (MCP) servers (e.g., Scrapy MCP Server, Playwright MCP). * Integrate semantic extraction (Scrapy\-LLM, custom LLM pipelines) to eliminate selector brittleness—spiders rely on field descriptions, not fragile XPaths. * Hands\-on experience building AI agents and orchestration systems. * Orchestrate complex, multi\-step browsing workflows with agentic frameworks (BMAD/TEA, AutoGPT\-like agents) that reason about page state, adapt to anti\-bot measures, and correct their own behaviour in real time. **Platform Thinking \& Reusability:** * Move beyond one\-off scrapers: build a component\-based extraction platform where selectors, login handlers, and pagination logic are shared, versioned, and tested. * Implement monitoring, alerting, and automatic rollback for failed extraction runs. * Champion ethical crawling by design—rate limiting, robots.txt respect, and compliance with GDPR/CCPA are built into the specification layer, not retrofitted. **Collaboration \& Continuous Innovation:** * Partner with data scientists and domain experts to refine extraction specifications for complex, unstructured domains (e.g., legal texts, tariff classifications). * Evaluate and pilot emerging tools to push automation coverage beyond 90%. * Document and evangelise specification\-driven best practices across the engineering organisation. **Qualification:** * Bachelor’s degree in Computer Science * 3\+ years of experience in web scraping or data extraction **Required Skills:** * Proficiency with Python * Experience with specification\-Driven Extraction * Hands‑on use of **Scrapy‑LLM, Scrapy** **MCP Server**, or similar systems that decouple field definitions from page structure * Experience with LangChain, LangGraph, LlamaIndex, AutoGen * Familiarity with frameworks that give LLMs browser control (Playwright \+ MCP, BMAD/TEA) to handle complex, non‑deterministic crawling tasks. * Classical Scraping Fundamentals * Data Validation \& Storage – Ability to define validation rules within specifications and land clean data into SQL/NoSQL databases or data lake * Basic API integration and authentication flows. * HTTP, DOM, XPath, CSS. **Nice to Haves:** * Contributions to open\-source scraping or AI\-automation projects. * Contributions to open\-source scraping or AI\-automation projects. * Familiarity with data privacy engineering (GDPR, CCPA) baked into specification design. * DevOps light – Docker, CI/CD for testing extraction specifications. **Mindset \& Approach (Non\-Negotiable):** * Strong belief that the future of scraping is declarative, not imperative. You’d rather write a schema that says “extract the price” than debug an XPath when a website redesigns. * Looking to shift from “code that scrapes” to “systems that understand extraction” Job Types: Full\-time, Permanent Pay: R$1\.00 \- R$10\.00 per year Experience: * Data Extraction: 3 years (Preferred) * Pydantic models: 3 years (Preferred) * JSON schemas: 3 years (Preferred) * Model Context Protocol (MCP) server: 3 years (Preferred) * Scrapy\-LLM: 2 years (Preferred) * Agentic AI: 1 year (Preferred) * LLMs Browser Control: 3 years (Preferred) * Large Language Model ( LLM ) : 2 years (Preferred) * Agentic Frameworks: 1 year (Preferred) Work Location: Remote