Real-Time Fraud Detection

I wanted to build the kind of system that sits inside a payments company. Transactions arrive through Kafka, partitioned by user so all of one person's events land on the same consumer. A FastAPI scoring service looks up that user's recent history in Redis: how many transactions in the last hour and last day, and how far this one is from their last known location.

Anything scoring below 0.4 is approved and written to the ledger. Anything at or above 0.4 goes to a fraud alerts table and shows up in a reviewer dashboard, where a person makes the call. A warm request takes 2 to 3ms because Redis keeps the history in memory. I built it solo over 8 weeks.

The training data is the Kaggle credit card fraud dataset: 284,807 transactions, only 492 of them fraud. That is 0.17%, so a model that approves everything is 99.83% accurate and catches nothing. I balanced the training set with SMOTE, which generates synthetic fraud examples, and left the test set at its natural fraud rate so the evaluation would stay honest.

I set the threshold at 0.4 instead of the default 0.5. On the test set that caught 87 of 98 frauds and wrongly flagged 43 of 56,864 legitimate transactions. Those 43 are a tiny rate, but they also mean roughly one in three flags is a real customer. That is why the reviewer dashboard exists: the model narrows things down, and a person decides.

Going back through the code, I found a problem I had missed the whole time I was building it. The model was trained on the Kaggle dataset's 28 anonymized features, V1 through V28, plus amount and time of day. The live API builds a different set: amount and time of day, plus the velocity and location features from Redis. Any column the model expects but the API did not build gets filled with zero.

So in the deployed system, the 28 features that carry nearly all of the model's signal are always zero, and the velocity and location features I built Redis around are never seen by the model at all. Live scores really only depend on amount and time of day. Even time of day does not line up: in the dataset it counts seconds from the first transaction in the file, not the hour on a clock.

This is called training-serving skew. It is easy to miss because nothing breaks. The pipeline runs, the API returns scores, and the dashboard fills up with alerts. The only sign is that the scores do not mean what you think they mean.

The 0.98 ROC-AUC and 89% recall are real results on held-out Kaggle data. They show the model can separate fraud when it has those 28 features. They do not describe the deployed system, which is why this page labels them as offline results.

They also say less than they seem to. The dataset covers two days of European cardholder transactions, and strong models on it commonly land around 0.98. The parts of this project I am most confident in are the engineering ones: the streaming pipeline, the Redis lookups, the threshold reasoning, and the review workflow.

The model needs to train on data that looks like what the live system sees: real user IDs, real timestamps, and locations, so velocity and distance can actually be learned. That means either a dataset like IEEE-CIS fraud detection or a synthetic transaction generator.

I would also add a check at startup that compares the model's expected features to what the API builds, and refuses to run if they differ, instead of quietly filling in zeros. Then I would measure the whole system end to end by replaying held-out transactions through Kafka, rather than only scoring a test set offline.

The database already stores every reviewer verdict, with a flag for whether it has been used in training. Feeding those decisions back into retraining is the step after that.