Senior Web Scraping / Data Collection Engineer (Python / Crawl4AI / LLM)

📍 Location: Remote

🤝 Engagement: Full-time, long-term project for an international product client

💬 English: B2+

🎯 Level: Senior

About the Opportunity

We are looking for a Senior Web Scraping / Data Collection Engineer to join an international product project.

The project is already in development. The client is currently strengthening the data collection / web scraping direction and is looking for an engineer who can take ownership of the existing scraping work and continue developing it.

The main goal is to build a modern and scalable data collection system capable of extracting structured data from various community, hobbyist, and other public websites.

The system should primarily collect structured information about locations, including:

Name

Address

Additional information available on the source

Sentiment and user opinions

The client plans to use Crawl4AI together with LLMs to handle complex and dynamic content and enable intelligent data extraction.

What You'll Be Doing

Develop custom scrapers using Crawl4AI.

Extract structured data from various websites.

Work with complex and dynamic web content.

Implement advanced extraction strategies using Pydantic schemas, CSS selectors, and LLM-based parsing.

Produce structured JSON / Markdown data optimized for further LLM processing.

Extract basic location data such as names and addresses.

Extract all available information about locations when additional data is provided by the source.

Develop approaches for extracting sentiment from user opinions and discussions.

Transform different opinions about a location into structured data that can be used for further processing and analytics.

Work with different types of sources, including:Community websites

Hobbyist websites

Blogs

Social media communities and groups

Work with websites using sophisticated access restrictions, including age gates, CAPTCHAs, and IP rate limits.

Use session persistence and browser fingerprinting to improve scraping success rates on challenging sources.

Deliver clean and deduplicated data.

Produce results in CSV or database-ready formats.

Optionally contribute to a web interface that allows users to:Enter URLs

Configure scraping settings

Adjust stealth mode

Set crawl depth

Add extraction instructions

Monitor scraping results

Data Structure

At a basic level, the system should be able to return location data in a simple structured format:

[

{

"name": "place1",

"address": "address1"

},

{

"name": "place2",

"address": "address2"

}

]

If a source contains more information, the system should be able to return additional fields in a consistent JSON format.

The system should also support extracting sentiment-related information.

For example, if 6 people mention that they loved a location, 2 consider it average, and 2 disliked it, this information should be structured in a way that can be used for further processing and analytics.

The exact sentiment format can be flexible, but consistent JSON is preferred.

What We're Looking For

Commercial experience in Web Scraping / Web Crawling / Data Extraction.

Strong Python skills.

Hands-on experience working with dynamic and complex websites.

Experience with browser automation.

Good understanding of HTML / DOM and CSS selectors.

Experience developing custom scrapers.

Experience working with structured data and JSON.

Experience with Pydantic or similar schema-based approaches.

Understanding of LLM-based data extraction.

Experience with session persistence and browser fingerprinting.

Understanding of CAPTCHA, age gates, and IP rate limits.

Ability to independently understand an existing scraping solution and continue developing it.

English B2+.

Nice to Have

Practical experience with Crawl4AI.

Experience integrating Crawl4AI with LLMs.

Experience with Playwright, Selenium, or similar tools.

Experience with LLM-based parsing / information extraction.

Experience with sentiment analysis / NLP.

Experience collecting data from social media communities, blogs, and other user-generated sources.

Experience building lightweight web interfaces for managing scraping jobs.

Who You Are

We are looking for an engineer who can independently take ownership of the scraping direction, understand the existing solution, and continue developing it.

It is important to be able not only to crawl pages, but to extract high-quality, structured, and usable data from them.

Strong practical experience with dynamic websites, browser automation, complex scraping scenarios, and structured data extraction is especially important.

What We Offer

🌍 Remote work.

📄 Full-time, long-term project.

🚀 Work on an existing international product.

🧠 Opportunity to take ownership of a dedicated technical direction.

💼 Work with Crawl4AI, LLMs, and modern data extraction approaches.

🤝 Direct collaboration with the client team.

📈 Opportunity to influence the technical solution and further development of the data collection system.

Hiring Process

1. Recruiter Screening

2. Client Interview (A technical interview with the client team focused on your practical experience, approach to complex scraping challenges, work with dynamic websites, and structured data extraction)

3. Offer 🎉