Senior Python Engineer — Web Scraping & Data Extraction

Description

Job Summary: We are seeking a Senior Python Developer with strong experience in web scraping to develop and enhance large-scale data collection solutions, contributing advanced technical expertise and engineering best practices. Key Highlights: 1. Focus on developing technical solutions for complex scraping challenges 2. Contribution to the robustness, reliability, and efficiency of data collection systems 3. Work with advanced technologies such as Scrapy, Celery, Redis, and PostgreSQL We are looking for a Senior Python Developer with strong web scraping experience to work on the development and evolution of large-scale data collection solutions. This professional will contribute advanced technical knowledge and engineering best practices to improve the robustness, reliability, and efficiency of the company's data collection systems. The role is hands-on and focused on developing technical solutions for complex scraping challenges. **Responsibilities** * Develop and evolve crawlers and large-scale data collection systems. * Create spiders using Scrapy, Requests, Selenium, and BeautifulSoup. * Work with asynchronous programming (async/await) to optimize collection performance. * Implement and improve proxy rotation and VPN usage strategies. * Develop mechanisms to bypass blocks and anti-bot systems. * Integrate CAPTCHA solvers into scraping pipelines. * Build asynchronous processing pipelines using Celery, Redis, and RabbitMQ. * Optimize storage and queries in PostgreSQL. * Develop APIs and integrations using Django. * Diagnose and resolve issues related to blocks such as rate limiting, fingerprinting, and WAF protections. * Contribute technical improvements to the data collection architecture and tooling. * Collaborate with the engineering team following agile practices and code reviews. **Requirements** * Strong experience with Python. * Solid production-level experience with web scraping. * Experience with Scrapy, Requests, Selenium, and BeautifulSoup. * Experience with asynchronous programming using asyncio. * Experience with Celery, Redis, and RabbitMQ. * Experience with PostgreSQL. * Advanced knowledge of HTTP, cookies, headers, and authentication. * Experience with proxy rotation and VPN usage. * Experience with CAPTCHA solvers. * Experience with anti-bot system bypass techniques. * Experience with Git and version control workflows. * Experience working in agile teams (Scrum or Kanban). **Nice-to-Have** * Experience with large-scale scraping infrastructure. * Experience with browser automation and user behavior simulation. * Experience with Docker and containerization. * Experience with distributed data collection systems. * Experience with crawler monitoring and observability. * Experience with CI/CD. * Experience with GitHub Actions.