The first time I watched a transaction fall through to stand-in processing, I did not fully understand what had happened. A card transaction came in, our fraud scoring system did not respond in time, and Visa made the approve/decline decision on our behalf. The transaction was approved. It was also fraudulent.
That was the moment the two-second window stopped being an abstraction and became a data engineering problem I thought about constantly.
What the two-second window actually means
Card schemes enforce strict response time requirements on issuers. When a cardholder taps or swipes, the transaction travels from the merchant’s terminal through the acquiring bank, through the card scheme’s network (VisaNet, Mastercard Network), and arrives at the issuing bank for an approve or decline decision.
Mastercard gives the issuer four seconds to respond. Visa allows up to ten seconds for point-of-sale transactions. Those numbers sound generous until you account for network transit time in both directions, the issuer’s internal routing, and the time the fraud decisioning system needs to score the transaction.
In practice, the fraud scoring system gets roughly one to two seconds. Sometimes less.
If the issuer does not respond within the scheme’s timeout window, stand-in processing (STIP) activates. The card scheme makes the decision using pre-configured parameters the issuer has filed. These parameters are blunt instruments: transaction amount limits, merchant category restrictions, geographic rules. They are not fraud models. They are fallback logic designed to keep the payment network moving when an issuer goes dark.
The consequence: every millisecond your data pipeline adds to the scoring path is a millisecond closer to losing control of the decision.
The data problem behind the scoring problem
A fraud scoring system is only as good as the data it can access at decision time. The model itself (XGBoost is a common choice for performant scoring on tabular financial data) is the part that gets the attention. The data pipeline that feeds it is the part that determines whether the model can actually run within the window.
The scoring system needs three categories of data for each incoming transaction:
Transaction attributes. The raw fields from the authorisation message: amount, currency, merchant category code, terminal type (card-present, card-not-present, contactless), acquirer identifier, timestamp. These arrive with the transaction itself. No pipeline latency here.
Cardholder profile. Historical behaviour for this specific card: average transaction amount over the last 30 days, typical merchant categories, usual geographic regions, transaction frequency patterns. This data does not arrive with the transaction. It has to be pre-computed and available for lookup in sub-millisecond time. Redis is a decent choice for this, keyed by card hash.
Derived velocity features. Number of transactions on this card in the last 5 minutes, 30 minutes, 24 hours. Total spend in the last hour. Number of distinct merchants in the last 6 hours. Geographic distance between this transaction and the previous one. These features are the ones that catch the burst patterns (a stolen card used at four different petrol stations in twenty minutes), and they have to be computed from a sliding window over the transaction stream.
The first category is free. The second requires a fast lookup store. The third requires a streaming computation that updates in real time as transactions flow through.
Each of these has a different data pipeline behind it, and each has a different failure mode that can push you past the two-second window.
Streaming architecture: Kafka and Spark DStreams
The streaming layer for this kind of system is Apache Kafka feeding into Spark Streaming (DStreams). Kafka is stable and handles the ingestion side well. Spark Structured Streaming is still marked as experimental, so DStreams is the production choice.
The pipeline looks roughly like this:
- Transaction authorisation messages land in a Kafka topic, partitioned by card BIN range
- A Spark Streaming job consumes the topic with micro-batch intervals of one to two seconds
- The streaming job computes velocity features (windowed aggregations per card hash) and writes them to Redis
- The fraud scoring service receives the raw transaction, enriches it with the cardholder profile and velocity features from Redis, runs the model, and returns an approve or decline decision
The one-to-two-second micro-batch interval in Spark Streaming is the constraint that shapes everything else. DStreams processes data in discrete micro-batches, not true event-at-a-time streaming. This means velocity features can be up to two seconds stale when the scoring service reads them. For most fraud patterns, two seconds of staleness was acceptable. For the fastest burst patterns (multiple transactions within seconds on a cloned card), it was a gap.
Apache Flink offers true event-time processing, but adoption in Australian financial services is minimal. The operational maturity, the available talent pool, and the integration story with existing Hadoop infrastructure all favoured Spark.
The storage problem: hot, warm, and cold
Transaction data accumulates fast. A mid-tier card issuer processing a few hundred transactions per second generates tens of millions of records per day. Over a year, that is billions of rows. Australian regulatory requirements (APRA prudential standards, Anti-Money Laundering Act) mandate seven-year retention for transaction records.
The storage challenge is not just volume. It is the competing access patterns that the same data has to serve.
Hot tier. The fraud scoring system needs sub-millisecond access to the last 24 to 48 hours of transaction data for velocity feature computation. This lived in Redis, keyed by card hash, with automatic expiry. Storage cost per gigabyte was high, but the dataset was small (recent transactions only) and the access pattern was pure key-value lookup.
Warm tier. Analysts, investigators, and reporting pipelines need query access to the last 30 to 90 days of transaction data. This lived in Hive tables on HDFS, partitioned by date and merchant category, stored in Parquet format for columnar compression. SQL access via HiveQL or Spark SQL. Query latency in seconds to minutes, which was fine for investigation workflows.
Cold tier. Regulatory retention. Seven years of transaction history, compressed, partitioned by month, stored on HDFS with replication. Rarely queried directly. When it was queried (regulatory audits, historical fraud pattern analysis), the queries ran as batch Spark jobs that could take hours.
The ETL pipeline that moved data between tiers was its own source of problems. Late-arriving transactions from batch settlement files (merchants that batch their authorisations rather than sending them individually) would arrive hours or days after the original authorisation. The warm-tier Hive tables needed to handle these late arrivals without breaking partition assumptions or creating duplicate records. We used a combination of date-based partitioning with a reconciliation job that ran daily, matching settlement records against authorisation records and flagging discrepancies.
Data quality at the boundary between tiers was a constant concern. A transaction that existed in the hot tier but failed to land in the warm tier meant an investigator looking at a flagged card would see an incomplete picture. A transaction that landed in the warm tier with a corrupted merchant category code meant reporting pipelines would misclassify it. Every tier boundary was a potential data quality break, and every break had downstream consequences for fraud detection accuracy.
Hadoop vs Spark: a reality check
The Hadoop ecosystem is mature but showing its age for this kind of workload. MapReduce is reliable for batch processing but painfully slow for the iterative, SQL-heavy analytics that fraud investigation requires. Running a complex join across two months of transaction data in MapReduce means writing Java, waiting for the job to compile and deploy, and then waiting again for the cluster to process it.
Spark changed the economics of this work. The same query that took 45 minutes in MapReduce typically ran in 3 to 5 minutes in Spark SQL, thanks to in-memory processing and the Catalyst query optimiser. For the data engineering team, Spark meant we could write ETL in Python (PySpark) instead of Java, which cut development time significantly. For the analysts, Spark SQL meant they could run ad-hoc queries against the warm tier without waiting for someone to write and deploy a MapReduce job.
But Spark does not replace Hadoop. It sits on top of it. HDFS remains the storage layer. YARN manages cluster resources. Hive metastore manages table schemas. The shift is from MapReduce as the compute engine to Spark as the compute engine, with everything else staying in place.
The practical data warehousing challenge is managing this hybrid. ETL pipelines written in PySpark read from Kafka (streaming) and HDFS (batch), transform the data, and write back to HDFS in Parquet format with Hive-compatible partitioning. Schema evolution needs careful handling because a column rename or type change in the authorisation message format can break downstream Hive queries that analysts have been running for months.
Data modelling for this kind of transactional data follows a predictable pattern: a fact table of individual transactions, dimension tables for merchants, cardholders, terminal types, and geographic regions. The star schema is not glamorous, but it is understandable by analysts who need to write their own queries. Denormalised wide tables for specific reporting use cases (monthly fraud rate by merchant category, daily transaction volume by channel) get materialised as downstream artefacts from the core model.
When the pipeline falls over
The interesting engineering is not in the happy path. It is in what happens when something breaks.
Kafka consumer lag. If the Spark Streaming job falls behind (cluster resource contention, a slow micro-batch caused by a data skew), the velocity features in Redis become stale. The fraud scoring system does not know they are stale. It scores transactions using outdated features. A burst of fraudulent transactions on a single card might not trigger the velocity rules because the Redis cluster holding pre-computed features has not caught up.
Redis failover. If the Redis cluster loses a node, the scoring service falls back to scoring without velocity features. The model still runs, but it is running blind on the most predictive feature category. The approve/decline decision is less accurate, but it is still faster than STIP.
Scoring service timeout. If the scoring service itself takes too long (model inference, feature lookup, garbage collection pause), the transaction falls through to STIP. This is the scenario that started this article. The fallback is not catastrophic for a single transaction. It is catastrophic at scale, because STIP uses static rules that cannot adapt to emerging fraud patterns.
The mitigation for all of these was the same principle: degrade gracefully and measure the degradation. Every component in the pipeline published latency metrics. Alerts fired when consumer lag exceeded thresholds, when Redis hit rates dropped, when scoring service p99 latency approached the STIP window. The goal was never zero failures. It was catching failures fast enough to intervene before the degradation compounded.
This problem is not unique to fraud
The pattern of ‘score a record in real time using historical features, make a decision within a tight window, fall back to a simpler system if the primary fails’ appears across financial services.
Credit card applications. An applicant submits an online application. The decisioning system pulls credit bureau data, checks internal records, runs a risk model (tree-based classifier, typically gradient boosted), and returns an approve, decline, or refer-to-human decision. The window is not measured in seconds (applicants will wait a few minutes for a credit decision), but the architecture is similar: streaming application events, pre-computed applicant features, real-time model inference, fallback to manual review if the automated system fails.
Personal loan origination. Same pattern, different data. Income verification, employment history, existing debt obligations fed into a scoring model. The model output determines the offered interest rate and credit limit. If the automated decisioning fails, the application queues for manual underwriting, which costs the lender time and staff.
Insurance claims triage. A new claim arrives. The system scores it for fraud risk and complexity, routes low-risk straightforward claims to automated processing and high-risk or complex claims to human adjusters. The scoring model uses claim attributes, claimant history, and derived features (claim frequency, geographic patterns). Timeout means every claim gets manual review, which collapses the efficiency gains.
In each case, the data engineering challenge is the same: build a pipeline that delivers the right features to the scoring system within the decision window, store the historical data in a way that supports both real-time feature computation and batch analytics, and design the fallback so that pipeline failures degrade the decision quality rather than halting the process entirely.
The models in all of these domains are overwhelmingly tree-based ensembles. Random forests for interpretability-first use cases (where regulators or internal audit need to explain individual decisions). Gradient boosted trees (XGBoost, or gbm in R) for accuracy-first use cases where the model output feeds into a rules engine that handles the explainability layer. I have compared these approaches on synthetic card data and found that logistic regression, framed as a probability scorer, often outperforms the tree-based models for triage-style decisioning. Neural networks get discussed in research papers but are rare in production for tabular financial data. The regulatory requirement to explain individual decisions (why was this application declined?) makes black-box models a hard sell in Australian financial services.
What I took away from this
The fraud scoring model gets the credit when catch rates improve. The data pipeline gets the blame when something falls through. That asymmetry is worth understanding early if you are building these systems.
The two-second window is a constraint that shapes every architectural decision downstream. Storage tier design, streaming framework choice, feature computation strategy, fallback architecture. None of these are independent decisions. They are all responses to the same question: can we get the right data to the right model fast enough to make the decision ourselves, or does the card scheme make it for us?
Right now, the answer is ‘usually, if nothing breaks.’ The engineering effort went into making that ‘usually’ as close to ‘always’ as the infrastructure allowed.