Python/Spark/AI Developer_TT

We're looking for a mid-to-senior Python data engineer to join the data team of a fast-growing, privacy-focused fintech (Series A). The company is backed by a global credit bureau and a major retailer, and has teams across South Africa, the UK, the US, the UAE and Brazil.

Its platform lets banks and lenders work with retailer and telco data without raw personal data ever changing hands. The main goal is to help people with no credit history get access to credit.

Your purpose in this role is to lead the move of legacy SQL Server / Azure Synapse pipelines onto open-source Apache Spark and Delta Lake. You'll write production software, not notebook scripts.

Logistics

Location: fully remote, Europe

Working hours: South Africa time zone (UTC+2, the same as Kyiv and one hour ahead of Warsaw). Senior leadership is on US Eastern time.

Employment: full-time. Contract type (B2B or permanent) and salary band to be confirmed.

Level: Mid-Senior, about 4–8 years of experience

What you'll do

Rebuild legacy T-SQL stored-procedure logic as typed, tested PySpark pipelines

Prove that the new pipelines match the legacy outputs

Build FastAPI services that run and report on pipeline jobs

Add provider-neutral LLM features (local and hosted models) to the data workflows

Work in a spec-first codebase: design docs and ADRs, with tests and documentation shipped alongside every change

Core stack

Python, PySpark, Spark SQL, Delta Lake, FastAPI, pytest, Docker, T-SQL, LLM integration

Requirements

Python as your primary production language: modern typed Python (3.10+, ideally 3.12), with type hints that pass pyright or mypy, and Pydantic models

2+ years running production Apache Spark / PySpark pipelines, using the DataFrame API and Spark SQL, partitioning and performance tuning, and with a solid grasp of the driver/executor architecture

Maintainable, tested code (pytest), with tooling such as uv or Poetry and a CLI framework such as Click

Based in Europe or South Africa and happy to work UTC+2 hours

Strong advantages

Experience migrating SQL Server / Synapse / stored-procedure logic to Spark and proving output parity with the legacy system

Open-source or self-managed Spark (Spark on Kubernetes, EMR, Dataproc, Spark Connect), not only Databricks notebooks; Hive Metastore, Trino, Apache Ranger

FastAPI services with request validation, HMAC-signed auth, idempotency and job-status semantics

LLM features: prompt construction, structured-output validation, local models (Ollama, vLLM, llama.cpp), evaluation and guardrails

Privacy engineering: PII tokenisation and hashing, k-anonymity, GDPR/POPIA, multi-tenant isolation

Fintech, banking, credit-bureau, adtech-measurement or retail-data background

Observability: structlog, OpenTelemetry, OpenLineage

Azure (ADLS, Synapse) and a Databricks Certified Spark Developer credential