Writing
Data engineering, platform architecture, and practical automation. Judgement and trade-offs, not tutorials.
// subscribe via rss-
Agent Intelligence Outran Agent Reliability
Model capability leapt while production reliability lagged. The next eighteen months belong to verification loops, harness engineering, and teams that close the gap.
-
Agentic Systems Have Three Learning Layers and the Model Is the Slowest
Agentic systems improve at the model, harness, and context layers. Most teams wait for the model. The compounding advantage sits in the two you own.
-
Loop Engineering for Microsoft Fabric: AI Agents, TE2, and Building DAX at Scale
TE2 C# scripts and an AI agent built hundreds of DAX measures across twenty-five Fabric semantic models. Loop engineering, not prompt engineering.
-
What Makes a Data Platform AI-Ready Before Anyone Adds an LLM
Organisations chase AI by bolting models onto fragile pipelines. The real work is upstream: quality, lineage, contracts, and governance that make AI safe.
-
Why Your Semantic Model Does Not Belong in a PBIX File
Twenty-five Power BI reports on one Fabric lakehouse. Decoupled semantic models, medallion layers, Direct Lake, and PBIX files with nothing in them.
-
Tokens Are Currency and the Subsidy Will Not Last
AI providers are subsidising tokens to drive adoption. When that ends, engineers without optimisation habits will feel it. Is your spend big enough to care?
-
Designing Data Platform Architecture for the Team You Will Have in Two Years
Most platform architecture is designed for today's team. In two years the team is bigger, the requirements are different, and the shortcuts are load-bearing.
-
Why Your dbt Project Will Become Unmaintainable
dbt makes it easy to start a data transformation layer. It also makes it easy to build one nobody can reason about six months later. The patterns that rot.
-
Test Harnesses for Data Pipelines: Engineering Confidence at Every Layer
Testing data pipelines isn't like testing application code. Messy inputs, probabilistic outputs, expensive infrastructure. How to build harnesses that work.
-
Data Quality Gates That Actually Work in Production
Most quality gates fire so often they get ignored or sit silent while bad data flows downstream. Build gates that catch real problems without crying wolf.
-
When Your Data Warehouse Needs a Second Act
Every data warehouse I have joined was overdue for redesign. The signals are consistent, the migration path known, and the biggest risk is waiting too long.
-
Streaming Production Data Across Snowflake Accounts with Fivetran, dbt, and Secure Shares
Building a near-real-time data pipeline for a mining operation using Fivetran CDC, Snowpipe, dbt incremental models, and Snowflake Secure Data Sharing. The architecture worked — but every layer had a gotcha that wasn't in the documentation.
-
Your PI Historian's Compression Settings Are Silently Destroying Your Analytics
OSIsoft PI's swinging door compression is brilliant for storage. It's also silently eating the data points your analytics need for correct averages and trends.
-
The OPC Timestamp Gotcha That Silently Shifts Your Entire Historian Dataset
When a PLC stores local time and the OPC server assumes UTC, every historian data point is wrong by your timezone offset. Daylight saving makes it worse.
-
Ellipse to Maximo Migration: The Equipment Hierarchy Problem Nobody Warns You About
Migrating Ellipse to Maximo in a mining operation sounds like a data mapping exercise. It's actually an ontological argument about what an asset is.
-
SAP PM Counter Overflow and the Haul Truck That Got Maintained Twice
When an equipment counter overflows, SAP PM calculates a negative usage delta. The consequences for mining fleet maintenance are worse than you think.
-
The Maximo Integration That Stopped Sending Data and Nobody Noticed for Six Weeks
IBM Maximo's MIF outbound queue can die silently — no error log, no alert, no notification. Here's how it happens and how to build monitoring that catches it.
-
What Managing a Data Science Project Taught Me About Scope
Technical skill produces analysis. Scope discipline is what makes that analysis land somewhere useful. A look at why data projects break traditional project management assumptions.
-
Spark MLlib Pipelines: From Manual to Managed ML
After hand-rolling k-means and gradient descent on RDDs, MLlib's Pipeline API reads differently. The fit/transform pattern is not magic, it is the loop I already wrote, packaged.
-
Implementing K-Means and Linear Regression on Spark Without MLlib
MLlib hides the training loop behind one method call. I rebuilt k-means and gradient descent as plain map and reduce jobs over RDDs to see what that call actually does.
-
Fraud Scoring on High-Velocity Card Data: When Regression Outperformed Classification
Classification vs regression on a 200K-row synthetic card fraud dataset. Why probability scoring beat hard classification for high-velocity transactions.
-
Spark Streaming: From Batch to Micro-Batch Processing
Batch processing assumes the data is all there. Spark Streaming assumes it never will be. DStreams, micro-batches, and the mental shift between the two, worked through over TCP and HDFS.
-
Reservoir Sampling: Sampling a Stream You Cannot Store
Taking a fair random sample from a stream you cannot rewind and cannot fully store sounds impossible until you see reservoir sampling. Worked through on six million Wikipedia titles.
-
Building a Flight-Delay Graph with Spark GraphX
Airport routes are a graph, not a table. I built a property graph of US flight delays with Spark GraphX, and hit a Parquet-versus-CSV performance gap I was not expecting.
-
Spark RDD Operations: The Foundation Under DataFrames
DataFrames and Spark SQL get all the attention, but RDDs explain why Spark behaves the way it does. Lazy evaluation, lineage, and partitions, worked through with a word count.
-
HBase: Schema Design for Column-Family Stores
A column-family store rewards a different kind of schema thinking than relational tables or MongoDB documents. Access pattern drives the design, not entity structure.
-
Writing a MapReduce Word Count in Java (and Why It Matters)
Word count is the canonical MapReduce example for a reason. Writing the Mapper and Reducer by hand in Java shows exactly what Spark spends its whole design abstracting away.
-
HDFS and the Hadoop Ecosystem
A distributed filesystem physically splits and replicates data across nodes. Notes from running Hadoop 2.7 on a multi-node VM and understanding what HDFS does under the hood.
-
Why Learning Scala Before Spark Matters
Spark is written in Scala, and its execution model borrows the language's assumptions about immutability and function composition. Learning Scala first is not a detour.
-
Autoencoders: Compressing and Reconstructing with Neural Networks
PCA compresses data linearly. An autoencoder can learn nonlinear compressions instead, with no labels required, built and tested in Keras on MNIST.
-
Convolutional Neural Networks: Why Spatial Structure Matters
A dense network treats an image as 784 independent numbers. A CNN treats it as a grid where neighbours matter, and that assumption is what makes it work, demonstrated in Keras.
-
My First Neural Network with TensorFlow and Keras
Installing TensorFlow and Keras to build a feedforward network on MNIST, and finding the gap between understanding gradient descent on paper and training a real model smaller than expected.
-
EM for Document Clustering: Unsupervised Learning Without Labels
No labelled data, no known number of topics, just a pile of documents. Working through the EM algorithm for a multinomial mixture model over text, in R.
-
Building a Streaming Pipeline: Python to Spark Streaming to MongoDB
Batch assumes all the data is already there. Streaming assumes it never stops arriving. The five lines that set up a Spark Streaming context changed how I think about pipelines.
-
Logistic Regression and the Multiclass Perceptron
Predicting a number and predicting a class look similar with a linear model until you look at what happens at the boundary. Working through logistic regression and the multiclass perceptron in R.
-
MongoDB Document Modelling for Sensor Data
Embedding related data inside a single document felt like breaking first principles. For 'give me everything about this reading,' one document read beats a five-table join.
-
Deriving Gradient Descent by Hand
Every model.fit() call hides the same loop: compute a gradient, take a step, repeat. Working through the derivation and coding batch and stochastic gradient descent from scratch in R.
-
Implementing Parallel Search, Join, and Sort From Scratch
Before reaching for Spark, I built parallel search, join, and sort by hand with Python's multiprocessing module. The exercise showed exactly what Spark abstracts away.
-
Bias, Variance, and the Overfitting Trap
A model with zero training error is not a good model. It might just be a lookup table with extra steps. Working through the bias-variance tradeoff and k-fold cross-validation in R.
-
Parallel Data Processing: Why One Machine Is Not Enough
pandas.read_csv() on a 50GB file does not work. At some point data outgrows a single machine, and the fix is not a bigger machine. It is splitting the work.
-
Unsupervised Learning in R: PCA, K-Means, and Hierarchical Clustering
PCA, k-means, and hierarchical clustering worked through in base R, and why unsupervised learning's real difficulty is the missing scorecard, not the maths.
-
Trees, Forests, and Boosted Models: Ensemble Methods in R
A single decision tree in R is easy to read and easy to break. How bagging, random forests, and gbm boosting trade that readability for stability, with code.
-
Data Reshaping: Normalisation, Discretisation, and Feature Engineering
Scaling, binning and sampling get filed under preprocessing, but each choice changes what a model can learn. These are modelling decisions wearing a disguise.
-
Lasso, Ridge, and Cross-Validation in R
A 50-feature regression with a high training R2 is not a good model, it's a warning sign. Lasso and Ridge in glmnet, tuned with cv.glmnet, show why.
-
Data Integration: Merging Sources Without Losing Your Mind
Two databases both have a customer table. They disagree on names, addresses, identifiers. Merging them is the most underestimated task in data work.
-
The Taxonomy of Dirty Data and How to Cleanse It
Data cleansing eats more project time than analysis ever does. A working taxonomy of missing values, outliers and duplicates turns it into a repeatable process.
-
Linear Regression on Real Estate Data in R
Fitting lm() to a real ~80-feature property dataset: stepwise selection, VIF for multicollinearity, and why coefficient interpretation gets shaky on real data.
-
Parsing the Unparseable: CSV, JSON, XML, Excel, and PDF in One Week
A dataset arrived as three CSVs, two JSON responses, an XML export, a five-tab workbook, and a PDF report. All of it had to become one DataFrame.
-
The Two-Second Window: Building Data Pipelines for Real-Time Fraud Decisioning
When your fraud system takes too long, the card scheme decides for you. Real-time scoring, streaming storage, and what happens when the pipeline fails.
-
Text Preprocessing with NLTK: Tokenisation, Stemming, and Stop Words
A machine cannot analyse a sentence. It can analyse numbers. NLTK's tokenise, normalise, filter, and represent pipeline is how you get from one to the other.
-
Regular Expressions Are the First Data Engineering Skill
SQL cannot parse a messy string. Excel cannot either. A well-written regular expression pulls structure out of raw text in one line, and every data pipeline needs one eventually.
-
Spatial Data on a Map: Geolocation Visualisation in R
Tabular data is rows and columns. Spatial data adds geometry, and that geometry carries information a pivot table cannot show. A choropleth proves it instantly.
-
Building an Interactive Data App with R Shiny
R is not a web framework. Shiny turns it into one anyway, and the reactive programming model underneath it is the part that actually changed how I write R.
-
Three Visualisation Tools, One Dataset: Tableau, R, and D3.js
Tableau thinks in drag-and-drop, ggplot2 thinks in layers, D3.js thinks in data bound to DOM elements. Using all three on the same dataset changed what I noticed in it.
-
The Five-Sheet Design Method for Data Visualisation
I thought visualisation meant picking a chart type after the analysis was done. The Five Design-Sheet methodology and Cleveland and McGill's perceptual research taught me it is a design problem.
-
Monte Carlo, Simulated Annealing, and Optimisation Without Gradients
Gradient descent needs a smooth loss surface. When the search space is discrete or riddled with local optima, Monte Carlo methods and simulated annealing do the job calculus cannot.
-
Why MML Might Be a Better Model Selection Criterion Than p-Values
A p-value tells you how surprised to be if there is no effect. It does not tell you which of two competing models explains the data most efficiently. MML does.
-
Clustering with the EM Algorithm
K-means forces every point into one cluster. The EM algorithm lets a point be 70% one thing and 30% another, and that uncertainty is closer to the truth.
-
Linear Regression in R: From Intuition to lm()
Excel trend lines and R's lm() draw the same line through the same points. Only one of them tells you whether the line means anything.
-
Probability Distributions and the Central Limit Theorem
I have summarised data with means and percentiles for years without asking why they work. The Central Limit Theorem is the reason, and seeing it converge in R changed how I read a sample.
-
Mapping a Business Problem to a Data Pipeline: Hotel Revenue Maximisation
How should a hotel set its room prices. Running a real business question through the Standard Value Chain and picking a tool for every stage, instead of solving it by feel.
-
Descriptive, Predictive, Prescriptive: The Three Levels of Analytics
Most of my work sits in descriptive analytics. A framework for what predictive and prescriptive demand, and why the jump between them is bigger than the diagrams suggest.
-
The Standard Value Chain and Why Every Data Problem Follows It
Three competing frameworks (CSIRO, Pivotal, Miller and Mork) describe the same sequence every data project follows. Mapping the academic chain against the messy version I actually work in.
-
What Data Science Actually Means (When Nobody Agrees on a Definition)
NIST, Cleveland, and O'Reilly each defined data science differently. A working data analyst tries to work out which one describes the job he already has.