The False Positive Problem in Fraud Detection

The performance metrics on my fraud detection engine look like this: 89% recall, 0.98 ROC-AUC, 0.08% false positive rate. Most people reading those numbers focus on the recall. 89% fraud caught sounds good. But the number that actually shaped how I built the system is the 0.08%. That is 8 legitimate transactions flagged for every 10,000 that go through.

Why the false positive rate is the real design constraint

At the volume payments companies operate at, 0.08% is not a small number. Run 10 million transactions a day and that is 8,000 customers whose legitimate purchases got blocked. Each one is a support ticket, a potential chargeback dispute, a customer who might not come back. The model that catches 99% of fraud but flags 5% of good transactions is probably worse for the business than one that catches 89% and flags 0.08%.

My system uses a score threshold of 0.4 (not the default 0.5). Anything above 0.4 gets flagged for review. That threshold was chosen by looking at the precision-recall curve and finding the point where the false positive rate crossed 0.1%. The recall at that point is 89%. That tradeoff is the whole design.

What actually makes fraud detectable: the features

The model is XGBoost trained on 284,807 transactions, oversampled with SMOTE to handle the class imbalance. The interesting features are not in the raw transaction data. They are the features you have to compute in real time: velocity_1h (how many transactions this user made in the last hour), velocity_24h, geo_distance_km (haversine distance from the last known location), is_new_location, and amount_log. A card being used in two countries within four hours is not something you see in a single row. You need the history.

Why Redis is load-bearing

Those velocity and geo features require querying recent transaction history on every incoming event. I store them in Redis sorted sets keyed by user ID, with auto-expiring TTL. The score query is O(log n). That is what gets the warm latency to 2ms. Cold start (first request, no cache) is 27ms. Without the Redis layer, you are hitting PostgreSQL on every score request and the latency number looks very different.

The streaming pipeline

Transactions come in through Kafka, partitioned by user_id so all events for the same user land on the same consumer and velocity calculations stay consistent. I used Kafka in KRaft mode, which removes the Zookeeper dependency. The consumer calls the FastAPI scoring endpoint, which hits Redis for features, runs XGBoost inference, and routes the result: score below 0.4 goes to the ledger, score at or above 0.4 goes to the fraud_alerts table and surfaces in the reviewer dashboard for a human decision.

I built this solo over 8 weeks. The thing I kept running into: every interesting engineering decision in this system exists because of the false positive constraint. The Redis feature cache, the threshold tuning, the reviewer dashboard for human-in-the-loop review. If you just optimized for recall and did not care about flagging good customers, you would build a much simpler system. The constraint is what makes it interesting.