Python Data Engineer

We're hiring full-time developer to build and maintain our Python/Flask platforms that collects data from a growing list of websites. We're looking for an engineer to own that data end to end: collecting it reliably, structuring it well, analyzing it with AI, and exposing it through APIs that our other applications depend on.

What you'll do

1. Scraping

Add new sites to the scraper. Analyze each target site's structure, write parsers with BeautifulSoup4, and map the extracted data into our database models.

Maintain existing scrapers. Diagnose and fix breakages caused by layout changes, new anti-bot measures, or changed endpoints.

Work with our adaptive HTTP client. Our client falls back through curl_cffi → httpx → requests → hrequests. You'll tune it per site, handling headers, cookies, sessions, TLS/browser impersonation, and retries.

Keep daily runs dependable. Add logging, alerting, and failure reporting to our scheduled jobs so problems surface quickly.

2. Database design

Design clean, scalable schemas. Model scraped data with proper normalization, keys, constraints, and indexes, balancing data integrity against query performance.

Manage schema changes safely. Use migrations (Alembic / Flask-Migrate) across SQLite in development and PostgreSQL in production.

Protect data quality. Validate, normalize, and deduplicate data so that everything built downstream can rely on it.

3. AI and data analysis

Turn raw data into insight. Analyze scraped data to find trends, anomalies, and patterns that matter to the business.

Apply LLMs where they add value. Use models such as Claude or GPT for classification, summarization, entity extraction, and structured output from unstructured text.

Improve how our AI reasons. Design and iterate on prompts, define structured output schemas, build evaluation sets, and measure accuracy so AI results get better over time, not just different.

4. APIs and integration

Build APIs for our other applications. Design and implement well-documented REST APIs in Flask with consistent request and response schemas, versioning, pagination, and error handling.

Secure and scale them. Implement authentication, rate limiting (Flask-Limiter), and input validation, and document endpoints with OpenAPI/Swagger.