Web Scraping and Web Crawler Tech Lead

Description

Job Summary: The Tech Lead will coordinate the crawler development team, providing technical guidance, defining architecture strategies, and ensuring robustness and scalability in a complex environment. Key Highlights: 1. Lead and guide the webcrawler developer team 2. Define strategies for architecture evolution and optimizations 3. Handle critical production failures and blockers The Tech Lead will be responsible for coordinating the team responsible for developing and maintaining crawlers that capture electricity bill data—a critical component of automation systems. The professional will guide the technical team, define architecture evolution strategies, and ensure the robustness and scalability of data capture processes in a highly complex environment involving multiple utility providers, access restrictions, CAPTCHAs, and anti-fraud controls. * Lead the webcrawler developer team, guiding best practices and reviewing code; * Define priorities for improvements and strategies for preventive and corrective maintenance; * Continuously evaluate crawler and pipeline architectures (Docker/Airflow), proposing optimizations; * Define and monitor operational and performance metrics (success rate, average capture time, stability, etc.); * Research and validate new tools and frameworks to enhance crawler efficiency and resilience; * Directly handle critical production failures, network blocks, or changes on utility provider websites; * Document processes, standards, and best practices; * Review and approve team code.