Writing
All notes
A chronological index of technical writeups, project notes, and learning logs.
Data System Design Interview Glossary
A reference of every technical concept a data engineer should be ready to name, define, or trade off in a data architecture interview. Organized to map onto your delivery framework: Functional Requirements → Non-Functional Requirements → Core Entities → Dat...
Study Guide: Data Operations Architecture at Scale
Production ML data operations is a collection of connected systems: ingestion, annotation, synthetic data, multimodal storage, enrichment, quality monitoring, agentic remediation, self-service tooling, scale patterns, and governance. This post walks through...
Agentic Systems Design Interview — Concept Glossary
Tailored to a role building agentic ML-development infrastructure: agents that work with code, data, experiments, and evaluations. Organized to your framework: FR → NFR → Core Entities → Data Flow → High-Level Design → Deep Dive. Name + one-line definition ...
Study Guide: The Nemotron 3 Super Data Engineering Pipeline
A structured reference for how NVIDIA built the 25-trillion-token pretraining corpus behind Nemotron 3 Super — a case study in large-scale data engineering for LLM pretraining, distinct from (but related to) operational ML data pipelines like feature stores.
Inside Nemotron 3 Super: The Data Engineering Behind a 25-Trillion-Token Model
I went through NVIDIA’s technical report and public material on Nemotron 3 Super, and the thing that stood out most wasn’t the model architecture. It was this: the hardest engineering problem was building the corpus, not designing the network.
The Data Operations Cycle
What Actually Feeds a Production Model
Batch, Streaming, and Merged Views in ML Data Operations
In the last few notes, I have been circling around the same lesson from different angles: production ML is mostly data operations. Models matter, but they sit on top of a system that has to acquire data, validate it, enrich it, label it, serve it, monitor i...
Explaining the Multimodal Lakehouse in Interviews
I have written six implementation deep dives about my serverless multimodal data lakehouse. Those posts cover how it works: Ray actors and the catalog trust boundary, content-addressed storage and versioning, training-ready shards and loaders, the eval feed...
Training-Ready Multimodal Data: Shards and Loaders
The last post was about what comes next for the multimodal lakehouse: better deduplication, measurable data quality, and an evaluation loop that can prove whether one dataset version is better than another.
From Search Demo to Data Infrastructure
The previous post covered the eval feedback loop: the point where a dataset version stops being “new” and has to prove whether it is actually better.
Eval Feedback Loops for Multimodal Dataset Versions
The previous post made a dataset version trainable. It covered the jump from immutable manifests to WebDataset shards and training loaders: the physical layer that lets a model consume data efficiently.
What Comes Next for the Multimodal Lakehouse
This project started as a text search demo and grew into a 12-stage multimodal pipeline. The most valuable outcome was not the finished demo. It was discovering which parts of the system are still shallow once the happy path works.
Ray Actors, Catalog Trust Boundaries, and Pipeline Battle Scars
This is the second implementation deep dive in my serverless multimodal data lakehouse project.
Scaling ETL Pipelines: From One Machine to Distributed Systems
Most ETL pipelines do not fail because the transformation logic is hard.
Multimodal Lakehouse Implementation Notes
In the first version of this project, I wrote about the gap between a notebook embedding workflow and a production-shaped multimodal data pipeline. These notes go one layer deeper into the storage and deduplication decisions: how raw media gets stable ident...
Why 8 Python Threads Can Still Use Only 1 Core
A simple, visual guide to CPU-bound work, the GIL, and why ProcessPoolExecutor sometimes beats ThreadPoolExecutor by a lot.
Serverless Multimodal Data Lakehouse
Most machine learning tutorials hide the part that becomes painful later: load a dataset, preprocess it in a notebook, embed it, train a model. That pattern works on 1,000 samples. It starts breaking when the same workflow needs to handle millions of files,...
DATA PIPELINE ORCHESTRATION
ETL data pipeline design using cloud data engineering tech stack and data models
Anomaly Detection in Electricity Consumption
Important facts: Anomaly detection is a very important and active business metric for various fields. A technique that is used to identify the unusual patterns that are not in sync with the expectations. It has many applications in business-like health...
An End-to-End Machine Learning Web App
Important Facts: Works best on 30 house amenities classes (as this model is trained on pre-defined amenities) Achieved mAP@0.5:0.95 - 0.495 – close enough to be a viable product Built on Pytorch, trained using GCP (1 GPU - P1000), developed API using...
M5 - Forecasting - Part-III
After thorough exploration of the data and time-series visualization, now I will try my hand at forecasting methods and predicting the demand for the three states and three products seperately. We will discuss various different approach present, used and de...
M5 - Forecasting - Part-II
After having explored this huge dataset, I wanted to explore with more of time-series components to understand the distribution much better. So, lets dive in. For Initial Data Exploration of this data, please read here - PART-I
M5 - Forecasting - Part-I
This is from a kaggle competition, where I wanted to participated and apply my learnings in forecasting methods. In this 1st part post, I am exploring the data with various visualization and trying to understand the dataset.