A running list of Jev references, useful ideas, and experiments.
TL;DR: Jev is TypeSafe’s AI for small decisions, not a chatbot. Give it context and a specific question; it returns a choice, score, or probability your code can use. Think routing tasks, ranking results, or flagging unsupported claims—with uncertain cases sent for review.
Updated September 20, 2026 · Original-post screenshots and summary previews.
A practical introduction to state, typed questions, and probability distributions. The linked playground presents six workflows and four harder cases, including ambiguous evidence and instructions embedded in the input.
Our angle: use those harder cases to test the checkpoint’s behavior. Keep exact checks and permissions in code; use model uncertainty to identify cases needing review.
Reference logged, not an integration. The linked playground has not been tested here. Noul returns a yes-probability, not a separate confidence score.
Start with a read-only review of existing projects. For each candidate, specify the required evidence, a narrow typed question, and what the application should do when the answer is wrong or uncertain. Test one repeated task against reviewed examples.
Our angle: find one useful decision inside the trace-to-eval workflow, while explicitly retaining tasks that ordinary code already handles well.
An audit method, not a demonstrated improvement. The embedded prompt has not been executed.
Interpret existing workflows, revisit archives with new questions, decide where deeper reasoning is worth spending, and attach small checks to individual items. Reported examples include ticket triage, post analysis, alert screening, and paragraph-level prose linting.
Our angle: screen archived sessions for the incidents worth reviewing, then spend deeper reasoning on those cases.
Reported examples; their accuracy and cost savings have not been independently tested here.
Explore Jev inside the agent’s surrounding software: select context, route work, evaluate progress, and check claims. The larger suggestion is to use inexpensive judgments to direct more capable reasoning.
Our angle: the ask-first checkpoint skill—bounded context checks before work and evidence checks before completion.
The checkpoint skill is installed, with 18 passing local tests and one completed live pilot. Results are advisory; automatic per-turn inference is not enabled.
Foreman experiments with a supervisor that compares worker activity and code changes with repository instructions. Jev supplies judgments; a code-defined policy controls steering or stopping.
Our angle: first test observation-only detection against real corrections. Measure false alarms before allowing interventions.
An experimental external project, not an installed or validated supervisor in our setup.
The TypeSafe skill gives coding agents integration guidance and a route to current documentation. Installing the skill is distinct from building a workflow that actually calls the model.
Our angle: use the installed TypeSafe guidance for small evidence-checking features, not another generic agent framework.
TypeSafe’s integration skill is installed. That does not activate our separate checkpoint experiment.
Adapted from Shann’s audit method for your private-company evals idea. This is a proposal, not an audit result or an instruction to execute the post’s prompt.
Map the existing workflows, read-only
Start with the known trace/eval project, checkpoint experiment, and weekly reading tool. Inspect their instructions, documentation, prompts, and relevant code. Record exact file references, unavailable material, and how each decision works today. Inventory other projects before expanding the scope.
For knowledge retrieval, document the current search method and source-link preservation before suggesting reranking. Keep authorization, revision matching, dates, and test exit codes in ordinary code.
Output: a short candidate list, including “no clear Jev use.” No files changed or integrations run during that proposed audit.
Write a decision contract for each candidate
Specify the input, authoritative references, one atomic question, allowed answers, and the next step in code. Compare usefulness, setup effort, frequency, and the consequences of a wrong answer. Leave unknowns marked unknown.
Initial shortlist: proposal evidence review; finding corrections that could become evals; reranking a retrieved session shortlist; checking whether an eval assertion follows the task contract.
Provisional recommendation: evidence review first, because prior project inspection identified a concrete review boundary—not because we have measured it outperforming alternatives.
Benchmark claim-to-evidence review first
Curate 30 claim/evidence pairs and have you review the labels before evaluation. Include supported claims, contradictions, missing context, and apparent failures followed by recovery. Target 10 development cases and 20 held-out cases, keeping each session and related cases together in one partition; adjust the counts if necessary to preserve that separation.
Compare deterministic citation checks plus manual review with the same process assisted by Jev. Measure false “supported” decisions, per-class recall, uncertainty, review time, end-to-end latency, and actual usage/cost. A small pilot establishes feasibility, not general reliability.
No reviewed 30-case set is claimed to exist yet. Private evidence must be selected, minimized, and approved before any TypeSafe call.
Keep the pilot advisory; turn useful incidents into evals
Every finding stays in human review. Keep source references, model/question versions, and human corrections. Advance only if the held-out run shows useful catches or less review work without increasing false acceptance; otherwise revise the question or keep the current process.
For the business, the deliverable is a small private evaluation suite that helps a company judge a real agent workflow. Test that value with a prospective buyer separately. Jev is a replaceable screening component, not the product’s proof.
No automatic proposal acceptance, skill rewrites, worker interruption, or public trace publishing.
The first decision, made concrete
Input
One proposal claim plus the smallest sufficient, revision-pinned evidence excerpt, including relevant recovery or contrary evidence.
Question
Does this evidence establish the whole claim at its stated scope and certainty?
Answers
supported · contradicted · insufficient, plus the returned probability distribution and confidence.
Code’s job
Validate ownership, source references, and revision; show the evidence and judgment together; retain the human decision separately.
Failure path
Missing evidence: unavailable. Uncertain or wrong judgment: manual review. API failure: unavailable, never a pass. No action is authorized by a model answer.
These starting points come from the September 19 project analysis; implementation details need rechecking during the proposed audit.
Checkpoint status: installed, with 18 local tests passing and one approved live pilot completed. That pilot is not a benchmark or a full archive audit. Recurring prompting remains undecided; this plan does not activate automatic inference.
Companion reading & resources
Addy Osmani — Brownfield Agentic Engineering Not a Jev thread, but the engineering counterpart: document constraints, preserve behavior, and verify changes in existing systems.
