Python Developer — Web Scraping & Data Extraction (Junior)

Description

Job Summary: We are seeking a Junior Python Developer interested in automation, data collection, and web scraping to develop and maintain bots, working alongside engineering teams to learn best practices in development and data manipulation. Key Highlights: 1. Development and maintenance of scripts and crawlers for data collection 2. Collaborating with the engineering team to learn best practices 3. Focus on automation, data collection, and web scraping We are seeking a Junior Python Developer interested in automation, data collection, and web scraping. The candidate will be responsible for developing and maintaining bots that collect, process, and integrate data from various systems and portals. The professional will work closely with an engineering team to learn best practices in software development, system architecture, and large-scale data manipulation. **Key Responsibilities:** * Develop and maintain scripts and crawlers for automated data collection. * Create spiders using Scrapy, Requests, and BeautifulSoup (bs4\). * Automate navigation on dynamic pages using Selenium. * Assist in configuring and using proxies and VPNs for data collection. * Work with message queues and asynchronous processing using Celery, Redis, and RabbitMQ. * Integrate collected data with internal systems built on Django. * Store and manipulate data using PostgreSQL. * Monitor crawler executions and identify failures or inconsistencies in data collection. * Identify and handle basic scraping blocks such as rate limiting, incorrect headers, or session issues. **Requirements** * Proficiency in Python. * Understanding of HTTP, HTML, and REST APIs. * Basic knowledge of web scraping. * Familiarity with Git and version control workflows. * Experience working in teams using agile methodologies. * Basic knowledge of SQL. **Nice-to-Have** * Experience with Scrapy, Selenium, or Requests. * Experience with BeautifulSoup (bs4\). * Experience with Django. * Basic knowledge of proxy rotation. * Experience with CAPTCHA solvers (e.g., 2Captcha, AntiCaptcha). * Knowledge of asynchronous programming (async/await). * Familiarity with Redis, RabbitMQ, or Celery. * Experience with Docker. * Knowledge of agile methodologies (Scrum or Kanban). * Experience with CI/CD pipelines. * Experience with GitHub Actions.