Senior Python Engineer — Web Scraping & Data Extraction

Description

Job Summary: We are seeking a Senior Python Developer with strong experience in web scraping to develop and enhance large-scale data collection solutions, focusing on complex scraping challenges. Key Highlights: 1. Strong experience in web scraping for large-scale data collection 2. Development of complex spiders and anti-blocking solutions 3. Improvement of robustness and efficiency of data collection systems We are looking for a Senior Python Developer with strong web scraping expertise to work on developing and evolving large-scale data collection solutions. This professional will contribute advanced technical knowledge and engineering best practices to improve the robustness, reliability, and efficiency of our data collection systems. This is a hands-on role focused on developing technical solutions for complex scraping challenges. **Responsibilities** * Develop and enhance large-scale crawlers and data collection systems. * Build spiders using Scrapy, Requests, Selenium, and BeautifulSoup. * Work with asynchronous programming (async/await) to optimize collection performance. * Implement and improve proxy rotation strategies and VPN usage. * Develop mechanisms to bypass blocking and anti-bot systems. * Integrate CAPTCHA solvers into scraping pipelines. * Build asynchronous processing pipelines using Celery, Redis, and RabbitMQ. * Optimize storage and queries in PostgreSQL. * Develop APIs and integrations using Django. * Diagnose and resolve blocking-related issues such as rate limiting, fingerprinting, and WAF protections. * Contribute technical improvements to data collection architecture and tooling. * Collaborate with the engineering team following agile practices and code reviews. **Requirements** * Strong Python experience. * Solid production-level web scraping experience. * Experience with Scrapy, Requests, Selenium, and BeautifulSoup. * Experience with asynchronous programming using asyncio. * Experience with Celery, Redis, and RabbitMQ. * Experience with PostgreSQL. * Advanced knowledge of HTTP, cookies, headers, and authentication. * Experience with proxy rotation and VPN usage. * Experience with CAPTCHA solvers. * Experience bypassing anti-bot systems. * Experience with Git and version control workflows. * Experience working in agile teams (Scrum or Kanban). **Nice-to-Have** * Experience with large-scale scraping infrastructure. * Experience with browser automation and user behavior simulation. * Experience with Docker and containerization. * Experience with distributed data collection systems. * Experience with crawler monitoring and observability. * Experience with CI/CD. * Experience with GitHub Actions.