Summary: Seeking a Senior Web Scraping Engineer to build resilient, automated data collection systems, focusing on anti-bot techniques and ethical data extraction. Highlights: 1. Lead technical design for complex evasion challenges in web scraping 2. Deep hands-on work with headless browsers and proxy solutions 3. Focus on ethical and compliant web scraping practices We are seeking a senior Web Scraping Engineer focused on automation and anti-bot techniques to build resilient data-collection systems. This role is critical for reliably extracting data from dynamic web applications while navigating anti-bot defenses and maintaining ethical, compliant practices. The position offers deep hands-on work with headless browsers, proxy and CAPTCHA solutions, and cloud-deployed scraping architectures using JavaScript/Node.js, Python, and .NET. You will lead technical design, solve complex evasion challenges, and transfer knowledge to internal teams. **Responsibilities** -------------------- * Design and implement web scraping and browser automation solutions using headless browsers. * Develop strategies to handle dynamic content (AJAX, infinite scroll) and complex DOM interactions. * Implement anti-bot evasion techniques, including CAPTCHA integration, fingerprint spoofing, IP rotation, and proxy management. * Emulate human-like behavior (mouse movement, delays) to reduce detection. * Build reliable authentication flows and manage secure credential storage and sessions (OAuth, SAML, MFA). * Integrate and manage third-party CAPTCHA solvers and proxy services. * Deploy and containerize scraping services for scalable execution in cloud functions or containers. * Analyze network traffic and reverse-engineer web app interactions to identify stable data sources. * Collaborate with stakeholders to transfer knowledge and ensure ethical, compliant scraping practices. **Requirements** ---------------- * Senior-level experience in web scraping, browser automation, and anti-bot strategies. * Proficiency with headless browsers and automation libraries: Puppeteer, Playwright, or Selenium. * **Strong skills in JavaScript/Node.js and Python; experience with .NET also required.** * Experience parsing and extracting content using BeautifulSoup, lxml, or Scrapy. * Experience performing direct HTTP requests using requests or httpx. * Practical experience with CAPTCHA solving workflows and services such as 2Captcha, Anti-Captcha, or CapMonster. * Experience with proxy services and IP rotation strategies using providers like Bright Data, Oxylabs, or Smartproxy. * Familiarity with deploying scraping workloads to cloud functions (AWS Lambda, Azure Functions) and containerizing with Docker. * Deep understanding of authentication flows (OAuth, SAML, MFA), secure credential handling, and session management. * Awareness of legal and ethical boundaries for scraping and familiarity with data privacy considerations (e.g., GDPR, CCPA). **Nice to Have** ---------------- * Experience reverse-engineering web applications and analyzing API calls via browser dev tools. * Experience with browser fingerprinting tools such as Multilogin. * Familiarity with bot detection platforms like Cloudflare Bot Management or PerimeterX.