Adjay Mistry-Katz

Consumer web + native iOS · AI with guardrails

Lyra — a language platform that teaches

A language-learning platform that actually teaches — grammar, conjugation, and vocabulary in pedagogical order with real spaced repetition — rather than testing random words. Four languages are live (French, Spanish, Italian, Portuguese) on a React web app and a native SwiftUI iOS companion sharing one API. The course is drawn as a galaxy: CEFR level = galaxy, unit = star, lesson = planet.

6 languages live
90+ units · ~3,000 vocab items
FSRS-4.5 scheduler
20 exercise formats
Azure + Whisper speech
Web + native iOS

Lyra's web course view: the whole French course drawn as three visible galaxies of stars against a starfield, labelled French A1 with 30 of 30 lessons done, French A2 with 4 of 38, and French B1 with 0 of 30, plus a 'fading' marker for items due for review — **A whole language, one screen.** CEFR level is a galaxy, a unit is a star, a lesson is a planet in orbit around it. Zoom is the navigation model — tap a galaxy to fall into its stars, tap a star to land on its planets — so a learner's position in 128 lessons is a place rather than a progress bar. The comet bottom-left is the review queue.

The map is the data model

Nothing about the galaxy is decorative: star positions, orbital radii and planet ordering are derived deterministically from the course structure itself — unit index, lesson count, prerequisite chain — so the same course always draws the same sky, and adding a unit moves nothing that already exists. The layout functions are the most heavily mirrored code in the platform: the Swift implementation is a named twin of the web one and is pinned by exact-value tests, because a learner who switches devices has to recognise the same constellation.

Lyra's iOS course map zoomed to the French A2 galaxy: a vertical chain of star systems labelled with unit names — Le conditionnel présent, Les phrases avec si, Le plus-que-parfait, Les pronoms relatifs — each with lesson planets in orbit and a lesson count, next to a comet marked 'four fading, tap to review' — **Inside a galaxy.** Each star is a unit, sized and lit by how much of it is learned, chained in pedagogical order — you can see the course's dependency structure without a syllabus. Planets carry their lesson number, and the comet is the spaced repetition queue asking for attention.

Lyra's iOS unit view for 'L'imparfait I', with a plain-English description of what the unit teaches, a zero percent progress ring, and four numbered lesson planets orbiting a sun, each labelled with its topic — the nous-stem rule, être à l'imparfait, la météo, la campagne d'autrefois — **Inside a star.** The unit states what it will teach you in plain English, then its lessons orbit as planets in the order you should take them — grammar rule, the irregular verb, then the two topics that use them. Twelve items from earlier units are fading and can be reviewed from here.

A lesson teaches before it tests

Each lesson runs as a fixed sequence of phases — grammar explanation, conjugation tables, vocabulary, then drills — rather than a shuffled quiz. That ordering is the pedagogical claim of the product: presentation is a first-class schedulable item, so a conjugation table you already know can be marked as such and demoted in the queue, and the twenty exercise formats only ever draw on vocabulary the course has already introduced.

A Lyra lesson in its conjugation phase: a header for 'Être à l'imparfait' in unit L'imparfait I, a three-step phase bar reading Grammar, Conjugation, Vocabulary, and the full imparfait conjugation of être with a control to mark the table as already solid — **The conjugation phase.** The phase bar is the lesson's contract — grammar, then conjugation, then vocabulary, then drills. The controls beside the table let the learner mark it already solid, hard, or unclear; that feeds the scheduler rather than just skipping the screen.

A 'tap the mistake' exercise: the French sentence 'Sophia avait très timide, mais maintenant elle est bavarde' with its English translation, the word 'avait' selected and marked correct, and feedback buttons reading 'too easy', a tick, a question mark, and 'bad exercise' — **One of twenty exercise formats,** graded deterministically on-device for instant feedback and re-graded server-side as the authority. The four buttons after the answer are the interesting part: *too easy* and *hard* feed the spaced-repetition rating, and *bad exercise* files a content flag that lands in a triage queue against the source unit.

Scheduling is the product

FSRS-4.5 spaced repetition runs server-side only: clients submit graded results, never schedules. Ratings come from grading, not self-report — wrong, almost, correct, and too-easy map to scheduler ratings — and grammar and conjugation concepts are first-class schedulable items alongside vocabulary. Deterministic grading runs client-side for instant feedback and is mirrored server-side as the authority; AI grading is a capped fallback for open answers only, never for choice formats.

Lyra's progress screen: a five-day streak, four items due now, twenty-four answers today, an offer to take an adaptive placement test, and a course mastery list where each unit shows a bar and counts of items learned, learning, marked known and confirmed at production level — **Mastery is reported per item state, not per lesson.** Each unit breaks down into learned, learning, marked-known and — separately — confirmed at *production* level, because recognising a word and being able to produce it are different pieces of knowledge with different schedules.

Lyra's practice tab, 'Beyond the planets': timed drills called Conjugation sprint and Gender sprint presented as meteor showers, and a 'Holiday crash course' asteroid belt of standalone topics like greetings and ordering food, each with an item count — **Content that doesn't fit the course gets its own space.** Timed drills are meteor showers; standalone survival topics are asteroid belts you can drop into with no course progress at all. Both draw from the same item pool and the same scheduler, so practice outside the syllabus still moves the review queue.

Content is a build pipeline

Courses are authored as unit JSON through a validated pipeline: generated authoring briefs, machine-checked invariants (vocabulary introduced before it's used, cross-unit dedupe, per-format payload schemas), and idempotent publishing that regenerates only changed rows by content hash — a full-language publish fell from twenty minutes to under thirty seconds on shared hosting. AI-generated exercises fill demand into a shared, hash-deduped cache, validated before serving and constrained to vocabulary already taught; in-app flag reports feed a triage loop back into the source content.

AI with guardrails

Prompts are composed server-side and every provider call passes one instrumented choke point logging tokens, cost, latency, and errors. The floating tutor receives the learner's screen context as a server-resolved sentence — never client free text — closing an obvious injection surface. Speech uses two providers with defined degrade paths: Azure pronunciation assessment scores read-aloud drills, Whisper transcribes free speech for debriefs; audio is consent-gated, usage-capped, and discarded after scoring.

Two clients, one contract

The iOS app is a genuine second client, not a wrapper: ported components carry named Swift twins, and every API-contract change on the web side requires a parity-log entry before it lands. The mirror rule from the fitness platform, formalised into process.

Multi-platform product · Claude throughout

Vega — an AI fitness platform

A fitness platform built for friends and family: React web app, native SwiftUI iOS app, Android client, Apple Watch workout scheduling, and home-screen widgets — all thin fronts over one PHP API and MySQL database. Claude is integrated across roughly nineteen surfaces: a streaming coach that queries Apple Health through client-executed tool use, post-session debriefs, multi-week training plan generation, and vision-based schedule import.

4 clients · 1 backend
Swift · Kotlin · React · PHP
~19 AI surfaces
SSE streaming proxy
HealthKit + WorkoutKit
TestFlight via Xcode Cloud

Vega's week view on iOS: a coach review verdict of 'slightly behind' with a paragraph of reasoning, a weigh-in and volume summary strip, then each day's sessions with planned and actual pace side by side — **The week, on iOS.** The coach's verdict sits at the top — read against planned-vs-actual volume, not vibes — then every session for the week: runs with planned pace next to what the Watch actually recorded, gym, spin and yoga inline. Ticking a session off here is the same write the web app and the Watch make.

Vega's trends screen: a recovery ring reading 61 percent 'moderate', HRV above baseline, a target exertion band, a sleep-stage breakdown, and 30-day charts for recovery and sleep score — **Readiness, computed not guessed.** HRV, resting heart rate and sleep stages come from HealthKit; the recovery score, the strain figure and the target-exertion band are the app's own arithmetic against the athlete's own baseline. The AI is handed the result — it never estimates readiness itself.

One backend, four clients, and the mirror rule

Four clients in three languages can't share a library, but some logic has to live on every client — readiness scoring constants, run-split labelling, the debrief instructions. The answer is a documented mirror rule: every deliberately ported piece of code names its twin on the other platform, and touching one side means finding and changing the other. Convention as architecture — the cheapest consistency mechanism that survives a solo maintainer, chosen over codegen or a shared spec because the mirrored surface is small and the docs live next to the code.

A Vega home-screen widget showing today's next session — a 5K time trial with target pace — a weather-adjusted pace note, the recovery ring, a one-paragraph coach read, sleep stages and the week's running volume; a live activity timer runs in the dynamic island above — **The widget is the whole day at a glance** — next session and its target pace, the weather adjustment applied to that pace, recovery, sleep, and week-to-date volume, with a live activity running in the Dynamic Island during the session itself. It renders from a cached snapshot the app refreshes, so it stays correct without waking the network.

Planning: arithmetic in code, structure in the model

Plan generation takes a race, a date, current weekly volume and a PB, and returns a multi-week periodised block. The division of labour is the important part: every number is computed by the app, not by the model. Daniels VDOT gives the target paces for each training zone, the app detects athlete level from logged volume, and both go into the prompt verbatim. Claude is asked for structure and progression — phase ordering, session mix, how the block tapers — because that's the part where judgement helps and arithmetic errors are the part users notice.

Vega's training plan generator: goal type and sport selectors, marathon race setup with goal time, current PB, start and race dates, current weekly volume, and a panel of target paces computed by the app from Daniels VDOT with a warning that the goal pace is faster than the current threshold pace — **Plan inputs.** The pace panel is labelled with where its numbers came from — Daniels VDOT 44.6, derived from the entered PB — and the warning underneath is a deterministic check, not model output: a goal marathon pace faster than current threshold pace is flagged in code before a plan is ever generated.

One week of a generated marathon plan expanded: four runs with distances, pace ranges, weather-adjusted pace notes and per-session fuelling detail, then gym, spin and yoga blocks where three gym sessions are marked 'reduced — key cardio session tomorrow' — **One generated week, expanded.** Week 6 of 15 in the build phase: session prescriptions with pace ranges, per-session fuelling worked out in grams and timing, and forecast temperature folded into each session's target pace for equal effort. The gym, spin and yoga blocks resolve against the running plan rather than sitting beside it — three strength sessions are auto-reduced because a key cardio session lands the next day.

Closing the loop on a session

A finished session is where the platform earns its keep. The Watch supplies splits and heart-rate zones through HealthKit, the app pairs them against what was prescribed, and the debrief is generated from that comparison plus the athlete's own one-line note. The summary is then persisted as context for later features — the weekly read and the next plan revision consume the debrief, not the raw splits, which keeps the downstream prompts small and keeps the coach's memory consistent with what it previously said.

A post-run debrief: the athlete's note 'first pace session in a while but this felt good, not too hard', then a generated paragraph analysing the marathon-pace block against its heat-adjusted target, a regenerate button, the original prescription with fuelling and conditions adjustment, and a plan-versus-actual table of splits — **A marathon-pace session, debriefed.** The generated paragraph reconciles three things the athlete shouldn't have to: actual pace against the heat-adjusted target rather than the original goal, the recovery score the session was run on, and their own comment. Underneath it, the prescription as issued and the plan-vs-actual split table it's judged against. Split labelling and expansion logic is mirrored code — web and iOS have to agree on it.

Tiered model routing

Every call defaults to Haiku. Plan generation — thousands of tokens of structured output — routes to Sonnet; screenshot import uses vision-capable Sonnet; chat exposes explicit Haiku/Sonnet/Opus escalation that resets after every message, with live token and cost readouts under each reply, so spending more is always a visible choice. Routing decisions are corrected by evidence: when Haiku kept confusing similar product model names in one feature, that single surface moved up a tier — and same-input calls were deduplicated so the sampled answer stays consistent.

Vega's coach chat on iOS: the question 'it's 32 degrees, should i really still do this run' answered with advice to shift the target to effort, time it for the coolest window, hydrate, and stop on warning signs; below the reply a readout reads 'sonnet · 21 in + 658 out · $0.0099' and a Haiku/Sonnet tier selector sits above the input — **Coach chat, with the meter running.** The answer is grounded in the athlete's actual plan — it knows this session is a protected one and that Wednesday's intervals went fine — because the coach queries Apple Health and plan state through client-executed tool use. The tier selector and the per-message token and cost line under the reply are deliberate: escalating to a bigger model is a choice the user makes and can see the price of.

Centrally governed prompts

Every AI surface registers in one prompt registry: a default instruction, a per-user override, and data toggles controlling exactly which context blocks get injected — goals, schedule, training history, plan progress. The settings UI renders governance cards automatically from the registry, and a preview button shows the exact assembled prompt the model will see. Domain knowledge ships as token-efficient JSON specs — running, strength, and cycling methodology — rather than prose, so one knowledge base serves every generator.

Vega's AI prompts settings page: governance cards for the post-exercise debrief and the gym debrief, each showing a locked admin system prompt in full, a user context box with an 'improve' button, and a collapsed 'data included' toggle list — **Prompt governance, rendered from the registry.** One card per AI surface: the base instruction is admin-owned and locked, the user gets an additive context box, and *data included* expands to the exact context blocks that will be injected. Adding a feature to the registry adds its card here — there's no second place to keep in sync.

A goals editor with three reorderable plain-text goals — run a sub 3:15 marathon in October, retain muscle mass while marathon training, keep doing yoga for recovery after previous lower back pain — above the note that these are included in every AI prompt — **Goals are plain sentences, ordered by priority**, and they go into every prompt on every surface. It's the cheapest personalisation mechanism that works: no taxonomy to maintain, and the ordering is what the coach uses to decide which goal loses when two of them conflict.

The weekly read

The reports tab is where the deterministic layer and the AI layer are visibly separate. Fitness score, training load, plan compliance and consistency are all computed metrics with their own sparklines; the coach's weekly read is generated on top of them and is stamped with the model that wrote it and what it cost. Every AI block in the product carries that stamp, and a thumbs-up/down that lands in the eval platform.

Vega's reports overview: six metric tiles with sparklines — fitness score, weight, weekly volume, training load, plan compliance and eight-week consistency — above a 'coach's weekly read' panel stamped 'haiku · $0.0049' with feedback, redo and clear controls — **Computed metrics, then the model's read of them.** Six tiles, each a deterministic calculation with its own trend; the generated weekly read sits directly beneath, labelled *haiku · $0.0049*. Attribution and cost on the surface itself, not buried in a log.

Prompt caching, and its limitation

Chat resends its large system block every turn, so it's marked for ephemeral prompt caching. Per-message cost accounting made the limitation visible: the system prompt is assembled from live user data, so any change to the injected context invalidates the cached prefix, and infrequent surfaces fall outside the cache window entirely. The wins are real inside an active conversation and near zero elsewhere — the restructure, a stable cacheable core with volatile context trailing it, is queued behind measurement in the eval platform.

Developer tooling · LLM-as-judge, validated

Eval platform — closing the loop

Two products and roughly twenty AI features meant every prompt change was a guess: most outputs were one-shot streams, invisible after the fact. The eval platform closes that loop, following the Hamel Husain / Shreya Shankar methodology — capture real traces, find the failure modes by hand, then make LLM judges prove they agree with a human before trusting them to score at scale.

Next.js App Router + TypeScript
Neon Postgres + Drizzle
Vercel
One trace schema, 3 apps
Full prompt + completion capture
Cohen's κ judge validation

Capture without touching the product

Apps push traces by fire-and-forget HTTP, flushed after the user already has their response. App-generated IDs make ingestion idempotent, malformed payloads are quarantined for inspection rather than dropped, and images are replaced by content hashes before leaving the app. The shared-hosting backend can't be reached from outside, so data flows strictly one way — and when the host turned out to lack the PHP primitive for post-response work, the fallback was a bounded flush with an accepted, measured ~1% loss rather than a redesign.

The eval loop

A keyboard-driven inbox for reviewing real traces; free-text open coding; axial coding into a small failure taxonomy — AI-suggested, human-created; then one LLM judge per failure mode. Judges are versioned append-only, return structured verdicts, and are validated against human labels with a confusion matrix, true-positive and true-negative rates, and Cohen's κ, disagreements listed first. A judge that doesn't agree with the human doesn't get to score.

Operating under real constraints

Scheduled scoring batches under the serverless 60-second ceiling and resumes where it stopped; an explicit error verdict keeps failed calls from re-queueing a run forever; every run carries its own cost accounting, with model prices verified against the live price list rather than memory.

Status, plainly

Ingestion and the trace viewer are live in production, with every fitness-platform AI surface instrumented across web, iOS, and Android. Annotation, the taxonomy tooling, and the judge stack are built and green and deploying now — the first validated judge runs over production traces are the current milestone.

Study tooling · LLM-authored, script-gated

Exam simulator — a question bank that has to prove itself

Preparing for Anthropic's Claude Certified Architect – Foundations exam, I wanted practice sittings that behaved like the real thing rather than a quiz app. What started as one hand-authored HTML file is now a small monorepo: a templated engine plus exam-agnostic Python tooling that assembles a form the way the live exam does, marks it against a modelled scale, and reports what the form did and did not test. The same engine now builds two offline simulators — this one, on a 406-item bank authored, tagged, and remediated by a model pipeline with scripts holding every gate, and a second for Anthropic's Professional-tier exam.

One templated engine, two exams built
Single HTML file per exam, no network calls
406 items · 166 distinct decisions
147 named distractor concepts
Two-pass tagging via the Batches API
200-form draw test at every gate

A marked simulator item: a Structured Data Extraction scenario brief, a question about a tool-call round limit, four options with the selected option A marked correct in green, a rationale panel citing Domain 1 task statement 1.1, a link to show why each other option fails, and a Next question button — **A marked item.** Each sitting picks four of six scenario briefs, then draws sixty items against the published domain weights. Marking on submission is the default; "exam conditions" withholds everything until the report. Every rationale names the objective it rests on, and each wrong option carries its own explanation of what it would fix instead.

Drawing a form that can't flatter you

Two items that test the same underlying decision are twins, and a sitting draws at most one of them — so the same judgement call is never marked right twice. Wrong options each lean on a named misconception from the exam guide, and a ledger caps how often any one of them can be the trap in a form, so a single blind spot can't swing the score either way. Both are soft constraints that relax rather than break the blueprint, and the report counts every relaxation. A draw test builds 200 forms and asserts the outcome: zero repeated decisions, exact domain weights, and every task statement sampled.

The simulator's score report for a twelve-item sitting: a scaled score of 390 marked FAIL against a cut of 720, a Raw margin note explaining how the scale is modelled, a percent-correct-by-domain table, and a Form composition block listing distinct decisions, the most-repeated distractor concept, the item-type mix, and the task statements the form did not sample — **The report says what the form covered.** The scaled score is modelled from the published range and cut, and the report says so rather than pretending otherwise. Below the domain breakdown, a form-composition block lists distinct decisions, the most-repeated distractor concept, the item-type mix, and the task statements this form never sampled — so a strong domain score can't be mistaken for coverage.

The bank is a pipeline, not a file

Items live in JSON; a build script injects them into the engine, and a linter refuses the result if anything is off — schema, key-length bias, longest-option-is-key rate, near-duplicate stems, per-concept caps, and whether the bank can still fill a clean sitting. Four metadata fields (the decision tested, the distractor concepts used, item type, difficulty) were retrofitted onto 220 items by two independent tagging passes through the Message Batches API, constrained to a closed vocabulary via tool-use enums so an invented tag could not exist. A diff script routed only genuine disagreements to a stronger model for adjudication, and the merge refuses to land a partial result. Haiku was measured and rejected for the job — stable on the decision, unstable on distractor concepts — and Sonnet took it for about $10 across 440 requests.

The simulator's Question bank page explaining how items are built: domains, scenarios and task statements; twins that never co-occur in a sitting; distractor concepts capped per form; item types and a 1-to-3 difficulty scale; twelve items reworded from the guide's own samples; and an items-by-domain table showing 412 items against the exam weights — **The bank explains itself.** The same page a candidate uses to sit the exam documents how items are built and what the bank holds — decisions, misconceptions, item types, difficulty, and which twelve items are reworded from the guide's own samples and so calibrated against Anthropic's writing rather than mine.

One engine, now two exams

The single-exam file only stayed single-exam for a phase. It has since split into a templated engine plus exam-agnostic tooling, with everything exam-specific — domains, scenarios, task statements, scoring scale, authoring bounds — pulled out into one YAML profile per exam. A guide-ingestion workflow turns a new certification's PDF into a reviewed profile and tag vocabulary before a single item is authored, gated the same way the item bank always was. The second exam built this way, Anthropic's Claude Certified Architect – Professional, reuses every gate, script, and report the Foundations exam already had — evidence the constraints were general, not special-cased for one guide.

Gates that were allowed to move, on evidence

Every phase ends at a measured gate. Several original targets turned out to be unreachable once the first phase measured the bank — a "no concept above two per form" line that the arithmetic of 29 items on one concept made impossible — and each was rewritten against the measurement, with the reasoning kept in the plan rather than quietly relaxed. The old text-similarity twin check repeated a decision in every single form; keying twins on the tagged decision took that to zero across 200 forms. Model routing is explicit: anything countable is a script, never a model call; Haiku for mechanical passes, Sonnet for rewrites and engine patches, Opus for the vocabulary, the hardest item types, and adjudication — with human checkpoints before tagging started and after the first ten authored items.

Integrity, plainly

Every item is written against the published task statements. The authoring spec bans anything recalled from a live sitting or lifted from a dump site, and every key must quote the objective bullet it rests on. Item ids are permanent, so exported run history stays readable when an item is reworded or retired. Nothing leaves the browser: run history is local storage with a JSON export, and the file makes no network calls.

ServiceNow internal tooling · described, not shown

Accelerator tracker

Built after spotting the team's engagement admin eating delivery time. A consultant running a portfolio of customer accelerator engagements loses a surprising share of the week to scheduling maths, reminder emails, and checking whether anything happened overnight. The tracker takes that off the plate: a single-file, zero-dependency web app (HTML and JavaScript, Python build step) that runs entirely on the consultant's own machine, integrates with ServiceNow, Outlook, Zoom, and OneDrive, and treats the ServiceNow record as the system of record rather than replacing it.

Opening it answers "what do I need to do today" before showing any table: messages due, instances provisioned but not yet announced, this week's sessions, and calendar conflicts. Every engagement stage has templated messages with a token bar and live preview, so a date can never be typed wrong, and reminder schedules are configurable per accelerator type because not every type runs on the same clock or provisions an instance at all. A date landing on a weekend moves itself; one landing on a holiday is a real decision, so it is flagged for a human rather than guessed.

What the reclaimed time is worth

Modelled at a deliberately conservative two hours per consultant per week — the team's own estimate runs nearer three — over 46 working weeks. A reclaimed hour is valued at what it bills for, so this is delivery capacity returned, not cost removed. Set the team size and the blended charge-out rate.

Consultants using it 20

Blended charge-out rate $150/h · $1,125/day

Delivery capacity returned per year

$276,000

40 hreclaimed across the team, every week
1,840 hreclaimed per year
1.07full-time consultants' worth of capacity

The whole scaling surface — annual value returned

Six shading bands, dark to bright with value and log-stepped so the low end stays readable; every cell is labelled, so nothing rests on colour alone. The outlined cell is the scenario nearest the sliders above.

Three scheduled mailbox automations on Haiku 4.5 fill the record in while nobody is looking: provisioned credentials found, encrypted, and staged behind an identity check; session bookings, reschedules, and cancellations matched to the right engagement; open support cases and provisioning incidents surfaced before a "your instance is ready" message goes out. Anything ambiguous lands in an inbox for a one-click decision instead of a guess, and the tool never sends a message on anyone's behalf. An optional drafting panel writes a first reply in the team's tone, constrained to state only what it was told and never to invent a date or a promise.

Adopted across the team, saving an estimated three hours per consultant per week, and maintained like a product: a requirements spec and roadmap, a regression suite, a set of Claude Code maintainer subagents for the edits that tend to break things, and a self-serve setup runbook that lets a colleague hand the fiddly per-machine steps to Claude Code. It's internal tooling, so there's no demo or code here — but I'm happy to talk through the architecture.