Adjay Mistry-Katz

Multi-platform product · Claude throughout

Vega — an AI fitness platform

A fitness platform built for friends and family: React web app, native SwiftUI iOS app, Android client, Apple Watch workout scheduling, and home-screen widgets — all thin fronts over one PHP API and MySQL database. Claude is integrated across roughly nineteen surfaces: a streaming coach that queries Apple Health through client-executed tool use, post-session debriefs, multi-week training plan generation, and vision-based schedule import.

• 4 clients · 1 backend
• Swift · Kotlin · React · PHP
• ~19 AI surfaces
• SSE streaming proxy
• HealthKit + WorkoutKit
• TestFlight via Xcode Cloud

One backend, four clients, and the mirror rule

Four clients in three languages can't share a library, but some logic has to live on every client — readiness scoring constants, run-split labelling, the debrief instructions. The answer is a documented mirror rule: every deliberately ported piece of code names its twin on the other platform, and touching one side means finding and changing the other. Convention as architecture — the cheapest consistency mechanism that survives a solo maintainer, chosen over codegen or a shared spec because the mirrored surface is small and the docs live next to the code.

Planning: arithmetic in code, structure in the model

Plan generation takes a race, a date, current weekly volume and a PB, and returns a multi-week periodised block. The division of labour is the important part: every number is computed by the app, not by the model. Daniels VDOT gives the target paces for each training zone, the app detects athlete level from logged volume, and both go into the prompt verbatim. Claude is asked for structure and progression — phase ordering, session mix, how the block tapers — because that's the part where judgement helps and arithmetic errors are the part users notice.

Closing the loop on a session

A finished session is where the platform earns its keep. The Watch supplies splits and heart-rate zones through HealthKit, the app pairs them against what was prescribed, and the debrief is generated from that comparison plus the athlete's own one-line note. The summary is then persisted as context for later features — the weekly read and the next plan revision consume the debrief, not the raw splits, which keeps the downstream prompts small and keeps the coach's memory consistent with what it previously said.

Tiered model routing

Every call defaults to Haiku. Plan generation — thousands of tokens of structured output — routes to Sonnet; screenshot import uses vision-capable Sonnet; chat exposes explicit Haiku/Sonnet escalation that resets after every message, with live token and cost readouts under each reply, so spending more is always a visible choice. Routing decisions are corrected by evidence: when Haiku kept confusing similar product model names in one feature, that single surface moved up a tier.

Centrally governed prompts

Every AI surface registers in one prompt registry: a default instruction, a per-user override, and data toggles controlling exactly which context blocks get injected — goals, schedule, training history, plan progress. The settings UI renders governance cards automatically from the registry, and a preview button shows the exact assembled prompt the model will see. Domain knowledge ships as token-efficient JSON specs — running, strength, and cycling methodology — rather than prose, so one knowledge base serves every generator.

The weekly read

The reports tab is where the deterministic layer and the AI layer are visibly separate. Fitness score, training load, plan compliance and consistency are all computed metrics with their own sparklines; the coach's weekly read is generated on top of them and is stamped with the model that wrote it and what it cost. Every AI block in the product carries that stamp, and a thumbs-up/down that lands in the eval platform.

Prompt caching, and its limitation

Chat resends its large system block every turn, so it's marked for ephemeral prompt caching. Per-message cost accounting made the limitation visible: the system prompt is assembled from live user data, so any change to the injected context invalidates the cached prefix, and infrequent surfaces fall outside the cache window entirely. The wins are real inside an active conversation and near zero elsewhere — the restructure, a stable cacheable core with volatile context trailing it, is queued behind measurement in the eval platform.

Consumer web + native iOS · AI with guardrails

Lyra — a language platform that teaches

A language-learning platform that actually teaches — grammar, conjugation, and vocabulary in pedagogical order with real spaced repetition — rather than testing random words. Four languages are live (French, Spanish, Italian, Portuguese) on a React web app and a native SwiftUI iOS companion sharing one API. The course is drawn as a galaxy: CEFR level = galaxy, unit = star, lesson = planet.

• 4 languages live
• 90+ units · ~3,000 vocab items
• FSRS-4.5 scheduler
• 20 exercise formats
• Azure + Whisper speech
• Web + native iOS

The map is the data model

Nothing about the galaxy is decorative: star positions, orbital radii and planet ordering are derived deterministically from the course structure itself — unit index, lesson count, prerequisite chain — so the same course always draws the same sky, and adding a unit moves nothing that already exists. The layout functions are the most heavily mirrored code in the platform: the Swift implementation is a named twin of the web one and is pinned by exact-value tests, because a learner who switches devices has to recognise the same constellation.

A lesson teaches before it tests

Each lesson runs as a fixed sequence of phases — grammar explanation, conjugation tables, vocabulary, then drills — rather than a shuffled quiz. That ordering is the pedagogical claim of the product: presentation is a first-class schedulable item, so a conjugation table you already know can be marked as such and demoted in the queue, and the twenty exercise formats only ever draw on vocabulary the course has already introduced.

Scheduling is the product

FSRS-4.5 spaced repetition runs server-side only: clients submit graded results, never schedules. Ratings come from grading, not self-report — wrong, almost, correct, and too-easy map to scheduler ratings — and grammar and conjugation concepts are first-class schedulable items alongside vocabulary. Deterministic grading runs client-side for instant feedback and is mirrored server-side as the authority; AI grading is a capped fallback for open answers only, never for choice formats.

Content is a build pipeline

Courses are authored as unit JSON through a validated pipeline: generated authoring briefs, machine-checked invariants (vocabulary introduced before it's used, cross-unit dedupe, per-format payload schemas), and idempotent publishing that regenerates only changed rows by content hash — a full-language publish fell from twenty minutes to under thirty seconds on shared hosting. AI-generated exercises fill demand into a shared, hash-deduped cache, validated before serving and constrained to vocabulary already taught; in-app flag reports feed a triage loop back into the source content.

AI with guardrails

Prompts are composed server-side and every provider call passes one instrumented choke point logging tokens, cost, latency, and errors. The floating tutor receives the learner's screen context as a server-resolved sentence — never client free text — closing an obvious injection surface. Speech uses two providers with defined degrade paths: Azure pronunciation assessment scores read-aloud drills, Whisper transcribes free speech for debriefs; audio is consent-gated, usage-capped, and discarded after scoring.

Two clients, one contract

The iOS app is a genuine second client, not a wrapper: ported components carry named Swift twins, and every API-contract change on the web side requires a parity-log entry before it lands. The mirror rule from the fitness platform, formalised into process.

Developer tooling · LLM-as-judge, validated

Eval platform — closing the loop

Two products and roughly twenty AI features meant every prompt change was a guess: most outputs were one-shot streams, invisible after the fact. The eval platform captures real traces, puts a reviewer in front of them with the exact context the model saw, turns the notes into a failure taxonomy, and makes LLM judges prove they agree with a human before trusting them to score at scale. It has now run end to end on the fitness coach's chat: a seven-category taxonomy, a judge whose latest version agrees with the human at κ 0.85, and a prompt experiment replayed against production requests.

• Next.js App Router + TypeScript
• Neon Postgres + Drizzle
• Vercel
• One trace schema · 3 clients
• Full prompt + completion capture
• Judges validated against human labels

Capture without touching the product

Apps push traces by fire-and-forget HTTP, flushed after the user already has their response. App-generated IDs make ingestion idempotent, malformed payloads are quarantined for inspection rather than dropped, and images are replaced by content hashes before leaving the app. The shared-hosting backend can't be reached from outside, so data flows strictly one way.

The review workstation, and where every line of the prompt came from

Reviewing to saturation means reading the schedule the coach was given while reading the answer it gave, so the viewer is three panes on one screen: the prompt, the conversation, the verdict. The prompt pane is the important one. The app's prompt builder assembles a system prompt from a dozen blocks, and the platform splits it back into them and labels each with its provenance: fixed text, the device clock, text the user wrote in settings, rows from the app's tables, arithmetic the app did before the prompt — so a wrong number there is the app's bug, not the coach's — and model output, an earlier answer fed back in as fact, whose mistakes become this turn's premises. Rules exist per feature and per client, because web, iOS and Android each build their own prompt; a block no rule matches renders as Unmapped rather than mislabelled, so a drifted prompt builder announces itself. Historical turns that predate capture get a reconstructed prompt under an amber banner, each block marked exact, as stored today, inferred, or not recoverable.

Notes anchored to text, tags on the turn a judge will score

Select a span in the prompt, a message or the response and press n: the note is saved against that span and painted back over the text on every later visit, so clusters form on what the coach actually said rather than on a paraphrase of it. A note can name the failure mode it argues for, and naming one tags the turn the note was written on — in a multi-turn chat that is the turn a judge will score, not necessarily the one on screen. Tags are the ground truth; notes are the evidence. Two counters keep the two honest: how many notes stand behind each category, and the tags with no note behind them, which are the labels you cannot defend when a judge disagrees. When what went wrong is the app or the platform rather than the coach — a turn never captured, a table rendered as raw markdown — the note goes to a separate bug queue instead. A bug report in the annotations would become a failure mode no judge can score.

The taxonomy

Open coding first: a free-text note on every trace with anything wrong, in the reviewer's own words. Then axial coding: the platform clusters the notes into candidate categories, and the reviewer renames, merges and splits them into a small set, each with a one-line definition that names the concrete behaviour and the tell-tales seen in real traces. The method is Hamel Husain and Shreya Shankar's; the discipline is that a reviewed trace with no tag is a human "pass", and the judges are measured against those as much as against the fails.

Judges that have to earn it

One judge per failure mode, bound to its category. Judges are versioned append-only, return structured verdicts with reasoning, and are validated against the human labels on a dataset: a confusion matrix, true-positive and true-negative rates, and Cohen's κ, disagreements listed first. Validation is not reading the κ; it is working through the disagreements one at a time and recording which side was wrong. Either the judge was right and the label moves — with a note saying why, so no tag is ever unexplained — or the judge was wrong and the case becomes evidence for the next version. The ratio between the two is the finding. Datasets split into a seeded dev and held-out pair: prompts are tuned on dev and the κ that gets quoted comes from held-out.

Experiments: replay production, then judge both

A prompt library with append-only versions, where v1 is prefilled from what production actually sent rather than a retyped approximation. A new version is replayed over a dataset's recorded requests — same messages, same tools, same model, only the instructions changed — in small batches driven from the browser, because each server call is capped at sixty seconds. A validated judge then scores production and the experiment on the same traces, and the difference is a pass rate, not an impression. When a version is copied into the app, marking it deployed splits fresh production traces into before and after for the judge that scores them daily.

Finding the app's bugs, not just the model's

The first thing the platform found was in the app, not the coach. With every block of the prompt labelled, a missing block is visible: three conversations that started within seconds of each other had no schedule in their system prompt while every other trace that morning did, and the coach answered questions about the week anyway. It went to the bug queue with the trace and the highlighted text attached, and out of the taxonomy so no judge would be asked to score it.

Operating under real constraints

Scoring and replay both batch under the serverless 60-second ceiling and resume where they stopped, because pending work is defined as "no result yet" rather than an offset; an explicit error verdict keeps a failed call from re-queueing forever. Every run carries its own cost — a 48-trace validation run on the largest model comes to under two dollars — with model prices verified against the live price list rather than memory, and the platform's own monthly spend sits on the dashboard next to the apps'.

Status, plainly

Ingestion, the workstation, annotation, the taxonomy, judges and experiments are all live in production, with every fitness-platform AI surface instrumented across web, iOS and Android. The coach chat has been reviewed to saturation, has a seven-category taxonomy, and has one judge validated through three versions on the dev split. Not yet done: the held-out run whose κ gets quoted, judging the v2 experiment against production, and marking a prompt version deployed so the daily judge can measure before and after in production. Two further judges are written but have not been run.

Study tooling · LLM-authored, script-gated

Exam simulator — a question bank that has to prove itself

Preparing for Anthropic's Claude Certified Architect – Foundations exam, I wanted practice sittings that behaved like the real thing rather than a quiz app. What started as one HTML file is now a small monorepo: a templated engine plus exam-agnostic Python tooling that assembles a form the way the live exam does, marks it against a modelled scale, and reports what the form did and did not test. The same engine now builds two offline simulators — this one, on a 406-item bank authored, tagged, and remediated by a model pipeline with scripts holding every gate, and a second for Anthropic's Professional-tier exam.

• One templated engine, two exams built
• Single HTML file per exam, no network calls
• 406 items · 166 distinct decisions
• 147 named distractor concepts
• Two-pass tagging via the Batches API
• 200-form draw test at every gate

Drawing a form that can't flatter you

Two items that test the same underlying decision are twins, and a sitting draws at most one of them — so the same judgement call is never marked right twice. Wrong options each lean on a named misconception from the exam guide, and a ledger caps how often any one of them can be the trap in a form, so a single blind spot can't swing the score either way. Both are soft constraints that relax rather than break the blueprint, and the report counts every relaxation. A draw test builds 200 forms and asserts the outcome: zero repeated decisions, exact domain weights, and every task statement sampled.

The bank is a pipeline, not a file

Items live in JSON; a build script injects them into the engine, and a linter refuses the result if anything is off — schema, key-length bias, longest-option-is-key rate, near-duplicate stems, per-concept caps, and whether the bank can still fill a clean sitting. Four metadata fields (the decision tested, the distractor concepts used, item type, difficulty) were retrofitted onto 220 items by two independent tagging passes through the Message Batches API, constrained to a closed vocabulary via tool-use enums so an invented tag could not exist. A diff script routed only genuine disagreements to a stronger model for adjudication, and the merge refuses to land a partial result. Haiku was measured and rejected for the job — stable on the decision, unstable on distractor concepts — and Sonnet took it for about $10 across 440 requests.

One engine, now two exams

The single-exam file only stayed single-exam for a phase. It has since split into a templated engine plus exam-agnostic tooling, with everything exam-specific — domains, scenarios, task statements, scoring scale, authoring bounds — pulled out into one YAML profile per exam. A guide-ingestion workflow turns a new certification's PDF into a reviewed profile and tag vocabulary before a single item is authored, gated the same way the item bank always was. The second exam built this way, Anthropic's Claude Certified Architect – Professional, reuses every gate, script, and report the Foundations exam already had — evidence the constraints were general, not special-cased for one guide.

Gates that were allowed to move, on evidence

Every phase ends at a measured gate. Several original targets turned out to be unreachable once the first phase measured the bank — a "no concept above two per form" line that the arithmetic of 29 items on one concept made impossible — and each was rewritten against the measurement, with the reasoning kept in the plan rather than quietly relaxed. The old text-similarity twin check repeated a decision in every single form; keying twins on the tagged decision took that to zero across 200 forms. Model routing is explicit: anything countable is a script, never a model call; Haiku for mechanical passes, Sonnet for rewrites and engine patches, Opus for the vocabulary, the hardest item types, and adjudication — with human checkpoints before tagging started and after the first ten authored items.

Integrity, plainly

Every item is written against the published task statements. The authoring spec bans anything recalled from a live sitting or lifted from a dump site, and every key must quote the objective bullet it rests on. Item ids are permanent, so exported run history stays readable when an item is reworded or retired. Nothing leaves the browser: run history is local storage with a JSON export, and the file makes no network calls.