Selected data engineering work
Senior data engineer · selected work
I build data systems that stay correct while they get faster.
Production case studies across change data capture, lakehouse architecture, event streaming, and query processing—plus a personal multimodal systems project built for the joy of learning how the pieces really work. Start with the outcomes; open the technical notes for the design decisions and failure modes.
Case study index
The short version
CDC to Iceberg
Removed duplicate data movement, made replay safe, and shifted heavy processing off premium warehouse compute.
67% lower Snowflake compute 02 uShip · event ingestionKinesis event path
Built a durable route from application events to governed analytics storage with replay and failure isolation.
EventBridge → Kinesis → S3 03 Meta · analytical processingQuery-path redesign
Matched physical processing to real investigation patterns and replaced full refreshes with bounded incremental work.
~5 days to ~1 day 04 Personal project · ML data infrastructureMultimodal lakehouse
Built a 12-stage pipeline that turns text, images, video, and audio into traceable, deduplicated, training-ready features.
Ray · Modal · LanceDB · WebDataset01 · CDC + Apache Iceberg
Re-architecting the lakehouse around the work that mattered
At uShip, replication failures created duplicate events, full-history backfills, and an expensive recovery loop. I redesigned the path so AWS DMS change data landed in S3, Glue merged it into Iceberg, and Snowflake remained the analyst-facing query layer.
Durable raw changes, sequence-aware state, and a serving layer that no longer owns heavy ingestion.
What I changed
I removed the Fivetran-to-Snowflake replication and warehouse-side deduplication loop. CDC changes landed cheaply in object storage, then a single Glue writer produced atomic Iceberg snapshots. Hidden partitioning and manifest statistics let readers skip irrelevant files; compaction converted small micro-batch files into query-efficient files.
The correctness trap
CDC delivery order is not business order. When multiple updates for one shipment arrived in a micro-batch, a naïve merge could overwrite a fresh price with an older one. The fix was to rank changes per primary key by the DMS sequence and merge only the latest change. Row counts would not have caught this: the row existed, but its state was stale.
Migration safety
I used parallel runs and reconciled row counts, aggregates, key distributions, samples, and critical business queries before cutting over each table. The new path carried the burden of proof because more than 100 downstream consumers could not be disrupted.
Trade-off accepted
Decoupling storage and compute reduced cost, but introduced lakehouse operations: compaction, snapshot expiry, orphan-file cleanup, catalog monitoring, and cross-engine debugging. Snowflake external reads could also trail native-table performance. For this workload, the lower cost and safer evolution justified that operational surface.
02 · Amazon Kinesis
Separating event arrival from analytical processing
I created an AWS-native path that accepted events through EventBridge, buffered them in Kinesis, landed immutable batches in S3, and loaded governed tables through Snowpipe. The design separated producer availability from warehouse availability and kept a replayable source outside the warehouse.
A buffered event log protects producers; object storage provides durable replay before warehouse loading.
Why a stream
Kinesis absorbed bursts and prevented analytical consumers from becoming a synchronous dependency of event producers. Partition keys preserved ordering where it mattered, while independent consumers could advance at their own rate.
Why land before loading
S3 was the durable handoff. Warehouse retries did not require producers to resend events, and a transformation bug could be repaired by replaying the original payloads. Snowpipe then handled continuous ingestion from files rather than coupling directly to the live stream.
Correctness controls
Event identifiers and idempotent merges make retries safe. Schema validation and quarantine keep malformed payloads out of curated tables. Freshness, rejected-event counts, duplicates, and source-to-target reconciliation distinguish pipeline health from data correctness.
Trade-off accepted
The extra landing step adds latency and another operational boundary. In return, it isolates failures and provides deterministic replay—a better fit for analytics than making the warehouse part of the producer request path.
03 · Query optimization
Making the processing model match the questions
At Meta, a framework metadata and metrics pipeline took roughly five days to produce data engineers needed for investigations. I replaced history-scaled full refreshes with bounded incremental processing and aligned the physical layout with component- and version-centric access patterns.
The main optimization was less work—not a larger cluster.
Diagnosis before tuning
The old pipeline's cost grew with all retained history, while engineers investigated by component and framework version. More compute would only accelerate a mismatched plan. I bounded each run to changed records plus a correction window and organized processing around the dimensions people actually filtered and joined.
Practical query choices
I used approximate distinct counts where the decision required percentage deltas rather than exact cardinality. I also bounded work to the roughly six or seven live framework versions because older versions represented less than 1% of users. Both choices removed expensive computation without weakening the decision.
The silent failure mode
A mutable timestamp used as a high-water mark can move past late or corrected records. The pipeline stays green while valid data disappears. I caught that risk by running the incremental path in shadow mode against full rebuilds, then added bounded correction windows and periodic reconciliation.
Trade-off accepted
I stopped at a one-day batch SLA because the consumer workflow was daily. Streaming would have increased state, late-event, and operational complexity for freshness no one could use. The goal was the fastest trustworthy answer inside the decision window, not the lowest possible latency.
04 · Personal project
Building a lakehouse for data a model can trust
I built this project to learn what sits between “embed some files” and reusable multimodal training infrastructure. One pipeline ingests text, images, video, and audio; preserves stable identity and provenance; runs distributed preprocessing and GPU inference; then produces both searchable features and streaming-friendly training shards.
Durable identity, warm GPU workers, a governed catalog, and a separate layout for training throughput.
Identity before features
Raw assets are immutable blobs addressed by SHA256. Exact duplicates across sources are stored once, and a dataset version is a JSON manifest of content hashes rather than another copied media folder. Branching a training mix adds metadata, not terabytes of duplicated assets.
Keeping expensive state warm
Ray Data uses stateful actors so each worker loads its embedding model once and serves many batches. CPU stages decode and validate media; GPU actors focus on inference. Batch size, actor count, and memory limits are explicit controls rather than hidden magic.
The catalog as a trust boundary
My first design let raw manifests reach the LanceDB catalog before quality gates. That meant corrupt or rejected samples could look valid downstream. I changed the contract so only approved filtered artifacts enter the catalog, and every search, version, shard, training job, and evaluation reads from that governed boundary.
Search and training need different layouts
LanceDB keeps embeddings and structured metadata together for filtering, similarity search, and provenance. Training needs fewer large sequential reads, so frozen manifests materialize into WebDataset tar shards. Shards optimize delivery; the catalog and manifest remain the source of truth.
Deduplication in cost order
SHA256 removes exact duplicates before inference. FAISS then finds semantic neighbors across the shared embedding layer, while quality decisions remain soft metadata instead of destructive deletes. Near-exact perceptual and fingerprint-based passes are documented next steps, not claimed as complete.
Demo scale, production-shaped
The measured run is intentionally small: roughly 10,000 records on ephemeral L4 GPUs for about $0.15. It does not prove billion-item throughput. What it validates is the system shape—stable identity, explicit stage contracts, reusable model actors, auditable curation, versioned datasets, and separate serving layouts.
Let’s talk systems
Looking for someone who can connect architecture decisions to measurable outcomes?
I’m interested in senior data engineering, data platform, and infrastructure roles where correctness, cost, and operability all matter.
Architecture details are intentionally generalized to protect proprietary information. Published figures are limited to confirmed results or deliberately approximate values; approximate values are marked with “~”.