Career Profile
Career evidence · career direction · imported context
Feeds Fit Report + Application PrepRoleward · AI product case study
Building Roleward
How do you make useful AI judgments across a real workflow without letting plausible language outrun the evidence, the product's rules, the budget, or the user's control?
That question is the whole product. Roleward is an AI career workbench for a search that runs for months — so a convincing sentence is not an output, it is a liability until the system can say which record it came from, what policy allowed it, what it cost, and what it still has not proven.
Product owner · independent builderAI-assisted implementationWorking product, in useAugust 2026
The short version
Five decisions, and what each one ruled out.
A decision with no cost is a preference. Each row below names the question, the position I took, and the capability I gave up to hold it — with the section that shows the mechanism.
- 01How much of the loop should the AI own?Model calls are bounded product actions with a stated budget; deterministic code owns discovery filters, identity, validation, rollups and lifecycle.Ruled outOne generic AI score per job, and any claim that the product is “an agent.”
- 02When is an output allowed to become a product claim?Only when it binds to the exact analysed source, an atomic routed criterion and an identifiable career record — and survives validation before persistence.Ruled outFluent narrative as the deliverable. Structured model output is untrusted input.
- 03What happens when a paid run fails?Every paid action reserves its maximum enforceable cost before dispatch, logs each provider attempt, and holds conservatively when the outcome is ambiguous.Ruled outAutomatic retries, provider fallback, and cost as an after-the-fact dashboard.
- 04What do I actually claim to have proven?Four grades of evidence, each matched to one claim. Outcome claims stay Unknown, in writing, on the public page.Ruled outA passing test standing in for product value — and the marketing that comes with it.
- 05Who holds authority when agents write the code?Product decision → explicit contract → bounded package → verification → acceptance. Agents choose within the boundary, never over it.Ruled outAccepting a plausible technical argument without checking its assumptions.
01 · Product model
The product starts before the prompt and continues after it.
General-purpose AI can already compare a résumé with a job description. The product question was what happens before that prompt, after it, and across dozens of opportunities over several months.
So different kinds of context stay attached to different decisions: career evidence and direction in Career Profile, current intent in Search Settings, exact job context in Jobs, application state in Applications. They feed each other without collapsing into one generic profile or one generic match score.
Context — different meanings, kept apart
Search Settings
What counts as a strong opportunity right now
Feeds Scan + scoped Fit checksWorkflow — one loop, five surfaces
ScanJobsFit ReportApplication PrepApplications
Judgments — four questions, never one score
Search relevance
Does this match the current search?
Personal Fit
What does the evidence actually support?
Company & Role Research
An independent external judgment
Lifecycle state
What the user did, recorded
Research does not redefine Personal Fit. Lifecycle state is not a model score. Applications records what you did; it never converts progress into evidence of fit.
Recurring discovery belongs in v1. Automatic submission does not.
Without scheduled scanning this is a reactive “paste a JD into AI” tool. But Roleward may discover, judge, prepare and track — what gets sent externally stays an explicit human action.
Search relevance and Personal Fit answer different questions.
One asks whether a role matches today's search. The other asks what a person's recorded evidence supports. Collapsing them is how a match percentage starts meaning nothing.
Evidence, direction and current intent are three data meanings.
What someone has done, where they want to go, and what they are searching for this month overlap without being the same field — so each one is stored, edited and cited separately.
AI execution map
Where the model actually runs — and where it is not allowed to.
Roleward is not one AI pipeline. The useful unit is the product action: what genuinely needs semantic judgment, what deterministic code does more safely and for free, and how many paid calls the action is allowed.
| Product action | Paid model calls | The model's job | Deterministic code's job |
|---|---|---|---|
| CV import | 1 | Extract structured career evidence from a résumé document | File, type and size validation, source handling, persistence |
| Search guidance | 1 | Draft what a strong opportunity looks like, from confirmed context | Validation, user editing, explicit save or replace |
| Scan | 0–1 | Prioritise surviving opportunities against the confirmed rubric | Controlled discovery, normalisation, dedupe, hard filters, result caps |
| Job ingestion | 0 for a supported source; 1 at most for generic URL or text | Shallow extraction only when a job cannot be resolved deterministically | ATS and source resolution, controlled fetch, source identity, snapshots |
| Fit Report | 2 for a parsed job; 3 for a new unstructured JD | Requirement extraction, shared evidence-matrix judgment, narrative synthesis | Evidence bundle, ID and state validation, rollups, recommendation rules |
| Company & Role Research | 1, with at most 1 bounded continuation | Synthesise bounded external evidence into a research judgment | Company and source identity, controlled search, caching, evidence bounds |
| Application Prep | 1 per step; 4 across the current sequence | Positioning strategy and grounded material generation | Evidence selection, context limits, provenance, revision lifecycle |
| Applications | 0 | — | User-controlled lifecycle state. Recording what happened needs no model |
02 · Grounding
“The model saw the résumé” is not grounding.
A model will happily write “your experience strongly demonstrates enterprise customer discovery.” The product question is not whether that sentence is fluent. It is whether the sentence is allowed to become a product claim.
To be allowed, it has to name which criterion of which version of the posting it answers, which career record supports it, and whether that support is direct or merely adjacent. Then it has to survive deterministic validation before anything is persisted.
01
Product data
Versioned evidence bundle
The exact analysed job source with its content identity, atomic routed criteria, and career evidence by stable record ID. A changed posting is never silently adopted into an existing report.
02
Model
Shared evidence-matrix judgment
Which evidence supports which criterion, whether that support is Direct or Adjacent, a proposed status, and the gaps nothing supports.
03
Deterministic code
Product-owned validation and decision
Evidence IDs, route legality, state consistency, rollups across Capability, Direction and Feasibility, then the recommendation matrix with downward-only caps.
04
Model
Narrative synthesis of an already-decided result
The model explains the validated recommendation and preserves its traceability. By this point it cannot change the verdict.
One traceable Fit Report — the recommendation was decided before the narrative was written
The guard, as it ran
The model may interpret evidence. It may not redefine the evidence policy.
Structured model output is untrusted input until it satisfies the contract. Here is one real rejection: adjacent evidence proposed as proof of the exact requirement.
- Requirement
- Lead enterprise customer discovery
- Career evidence cited
- Led adjacent user research evidence id resolves
- Evidence relationship
- Adjacent
Metrejected
Adjacent + Met is not a legal combination. Adjacent evidence can support part of a requirement; it cannot become proof that the exact experience is already there.
What the model proposed
Partialdowngraded before write
The correction happens before persistence and before the narrative phase, so the report the user reads and the record the product stores cannot disagree.
What the product persisted
Six deterministic guards, all of them refusals.
Each one exists because a plausible output tried to do something the product does not permit. None of them makes semantic judgment infallible — that is not what they are for.
Guardrails reduce the error space. Evaluation tells you what errors remain.
- Scores dropped for non-capability or unassessable criteria
- Invalid or stale evidence IDs rejected outright
- Unsupported Met or Partial downgraded to Missing
- Evidence-backed Missing raised to Partial
- Non-direct evidence never allowed to remain Met
- Positive evidence references stripped from a Missing status
Routing matters as much as scoring: only capability-routed, assessable criteria ever receive Met, Partial or Missing. A willingness to travel does not become “missing skill: travel”, and a work-authorisation question is a feasibility check, not a judgment about your career.
03 · Cost and reliability
A cost incident that was really a reliability failure.
The first intuition was the usual one: the model is expensive, so write better prompts and pick a cheaper model. The incident said otherwise.
What the legacy telemetry could measureCompany & Role Research
2
failed Company & Role Research runs
4
phase-level records already written
611,615
combined tokens across those records
0
usable product results
The real defect was that significant paid work could occur with no usable product result and no precise account of what had happened. That is not a prompt problem. It is an execution-boundary problem, so every paid logical action now gets its own envelope and reservation before any provider work begins.
Before
- User trigger
- Broad model-led research
- Large external context
- Model judgment
- Aggregate run telemetry
- Failure, timeout or invalid output — indistinguishable
- Cost known only after the work happened
- One logical run ≠ one provider attempt
- A failure was not the same thing as free
After — per paid logical action
- User intent or an approved schedule
- Free deterministic preparation first
- Define the bounded action envelope
- Reserve the enforceable maximum
- Controlled paid external and model work
- Per-attempt ledger: usage, cost, retry identity
- Settle actual usage, or hold pending reconciliation
No budget → no call, and it is a stable state rather than an error. Paid source ingestion is admitted separately from the Fit run that follows it.
Do the free deterministic work first
Nothing pays a model to normalise a URL, dedupe a record, apply a hard filter or compare two numbers.
Reserve before paid execution
Reporting after the fact is observability. Control means admission before dispatch, or a stable no-call state.
Account for provider attempts, not product runs
One logical action can contain several external or model attempts, and the ledger is written per attempt.
Never make an ambiguous failure look free
A timeout does not prove the provider did no work, so the hold stays until reconciliation resolves it.
Make every retry explicit
SDK automatic retries are disabled. Recovery is bounded, deliberate and recorded as its own attempt.
Reservation is deliberately not an optimistic estimate: admission holds the maximum enforceable amount before dispatch, known usage settles against the hold and releases the remainder, and an ambiguous provider outcome keeps the hold until reconciliation. That makes the budget a concurrency-safe boundary instead of a dashboard that reports overspend after the fact — which is why cost and reliability turned out to be one question: what work was authorised, attempted, retried, persisted and paid for?
04 · Evaluation
What exactly have I proven?
A passing test is not a good model judgment. A good model judgment is not a useful product output. A useful output is not a better interview outcome. Four kinds of evidence, four different claims — so the page states which grade each claim actually has.
Strong
Deterministic tests and known failure-mode fixtures. Proves a policy guard, nothing about model quality.
Technically accepted
Synthetic and bounded paid acceptance. Proves an execution path completes under its contract.
Early evidence
Reviewed real output. Narrower and more honest than a benchmark, and enough to redesign from.
Open or unknown
No sufficient evidence yet. Named rather than implied, and never upgraded by enthusiasm.
| Claim | Evidence today | Grade |
|---|---|---|
| Evidence-policy guards behave as designed | Deterministic tests plus known failure-mode fixtures | Strong |
| Paid Fit executes under its bounded contract | Synthetic paid acceptance, post-correction | Technically accepted |
| Fit judgments are consistently useful | Real Fit Reports reviewed across four roles | Early evidence |
| Company & Role Research quality and operating cost | Reliability correction released; paid acceptance not yet run | Open |
| Application Prep materials are broadly useful | Limited human review, separate from technical acceptance | Open |
| Roleward improves job-search outcomes | Insufficient outcome data — no longitudinal evidence exists | Unknown |
Post-correction synthetic Fit run
USD 0.103710
Completed and produced a source-bound 70% / Maybe report.
Proves the paid path completes under its contract. Says nothing about whether the verdict was good.
Post-correction Application Angle run
USD 0.040810
Single provider attempt, expected revision persisted, canonical résumé source selected.
Proves lifecycle and source selection. Usefulness of the writing is a separate, open review.
The reviews that went badly are the useful ones.
Real Fit Reports were preserved and read across four roles — Togal AI, AI71, Brain Co. and Pencil. Reviewing my own product's output as a user is what surfaced these, and none of them were visible from the code.
- Requirement scoring was unstable between runs
- Adjacent evidence was mishandled in both directions
- Narrative repeated itself and carried low-value lines
- Current preferences blurred into historical capability
- Different parts of one report contradicted each other
Not claimed
These need longitudinal, real-user evidence the product does not have yet. Listing them is cheaper than being asked.
- Product-market fit
- Higher application success rate
- Higher interview conversion
- Sustained improvement in user outcomes
There is also no systematic benchmark yet against strong general-purpose AI analysis. That comparison is the next piece of work, not a claim.
05 · Building with agents
Coding agents raised leverage, not authority.
Agents implemented substantial parts of this product. What did not move was who decides what it means. The working pattern is fixed, and every step of it is mine to sign.
Product decisionExplicit contractBounded packageVerificationAcceptance
I own
- Product meaning, scope and data semantics
- Product and architecture boundaries
- What the model may and may not decide
- Evaluation design and the PM labels
- Trade-offs, acceptance and release
- Dogfooding the real product
Agents own, inside an approved contract
- Implementation detail
- Code and migrations
- Tests and bounded technical verification
- Ordinary engineering choices inside the boundary
- Nothing that redefines product behaviour
Two failures that code review could not have caught.
A Fit Report round passed every implementation-level check and shipped duplicated reasoning plus a parent summary that no longer made sense once its child content changed structure. Only reading the rendered report as a user found it.
In another round an agent defended keeping two status concepts separate with several plausible implementation arguments. Checked against the actual behaviour, most of the claimed consequences did not hold, and the real trade-off was far smaller than the explanation implied.
Different validation layers answer different questions. A type check proves the code compiles; it cannot prove the product makes sense to the person reading it.
What I ask an agent's argument
- 01What assumption is this argument relying on?
- 02Is that assumption true in the current system?
- 03Is this a product requirement or an implementation convenience?
- 04What evidence would change the decision?
The transferable skill was not knowing the answer first. It was knowing which four questions make an argument testable.
Technical appendix
Architecture at a glance
Ordinary product data does not pass through an AI backend; paid AI execution always does. That is the whole reason the boundary exists — one controlled path to govern credentials, cost, retries and provenance.
Browser Next.js · React · TypeScript on Vercel
Ordinary product path
Supabase
PostgreSQL · Auth · row-level security · Storage
authenticated reads and writes, no AI backend involvedServer-side AI path
FastAPI on Railway
Workflow and deterministic logic, then Claude and controlled external sources
provider credentials server-side · one governed paid entry pointGraph orchestration is used where workflow state, branching or tool use earns it — Scan, Fit and Research. CV extraction and material generation stay bounded functions. “Could this be an agent?” is not the question; “does autonomy earn its complexity here?” is.