Senior Web Scraping and Web Crawler Developer

Description

Job Summary: Responsible for the development and maintenance of energy invoice-crawling web scrapers, ensuring robustness and scalability in a complex environment. Key Highlights: 1. Development and maintenance of invoice-crawling web scrapers 2. Architectural evolution strategies and crawler optimization 3. Handling critical production failures Responsible for developing and maintaining web scrapers that capture electricity invoice data — a critical component of automation systems. The professional will support the team in developing architectural evolution strategies and ensure the robustness and scalability of data-capture processes within a highly complex environment involving multiple utility distributors, access restrictions, CAPTCHAs, and anti-fraud controls. * Collaborate with the web crawler development team, guiding best practices and reviewing code; * Suggest improvements and preventive/corrective maintenance strategies; * Continuously evaluate crawler and pipeline architectures (Docker/Airflow), proposing optimizations; * Monitor operational and performance metrics (success rate, average capture time, stability, etc.); * Research and validate new tools and frameworks to enhance crawler efficiency and resilience; * Directly handle critical production failures, network blocks, or changes on distributor websites; * Document processes, standards, and domain best practices; * Review and approve team members' code.