Case study: turning a sales playbook into a candidate skill

Short answer

Ernesta Labs converted a human sales playbook into agent skills by decomposition, not by prompt enlargement. The playbook's conversational spine became four staged gates - Intent, Logical Certainty, Emotional Certainty, Pitch - and its specialized tactics became lazily loaded modules (Negative Reverse, Objection Handling, Adaptive Selling, Reframes, Outreach/Booking) that enter context only when the conversation reaches them.

The one-giant-prompt version was inferior on every axis we could measure: it consumed context whether or not a given tactic was needed, it gave us no way to attribute outcomes to specific playbook components, it could not be evaluated per-component, and it silently blended the playbook's assumptions into the model's own judgment with no way to separate them.

The pipeline that made this honest is source -> candidate -> evaluation -> challenger -> evidence -> promotion/rejection. The playbook is a source of hypotheses about behavior. It earns its way into production only after evaluation against real conversations produces evidence, and a challenger run that tries to beat it fails to. Human expertise is a prior, not a validated behavior.

The conversion is implemented and structurally sound. What it has NOT proven is revenue lift: improved conversational structure is measured, deal outcomes are not yet. We report both.

The failure mode: the giant sales prompt

The starting point for this case study is the thing almost every team actually builds first: one enormous prompt that says everything the company knows about selling. Paste the whole playbook in. Add the founder's tips. Add the objection-handling scripts, the discovery questions, the tone rules, the follow-up cadence, three paragraphs on what not to do, and a closing exhortation to be consultative. The result is a prompt of many thousands of tokens that ships in full on every single turn of every single conversation.

We built that version first, deliberately, because you cannot honestly claim a replacement is better without a baseline. The giant prompt had three structural failures. First, cost and attention: a cold outreach message does not need objection-handling scripts, but it paid their context tax anyway, and mid-conversation turns drowned in instructions about stages the conversation had already passed. Second, no attribution: when a conversation went well or badly, nothing in the design told us which component of the playbook deserved credit or blame - the prompt was one opaque blob. Third, no evaluation surface: there was no way to test whether the objection-handling material improved objection handling, because it could not be varied independently of everything else.

There is a fourth failure that is quieter. A giant prompt presents human expertise as settled fact. The model treats the playbook's claims - that this question order converts, that this reframe defuses price pushback - the same way it treats arithmetic. But a playbook is a bundle of hypotheses written by a talented human, tested informally across their own deals. Loading it as a monolith means the model inherits every one of those hypotheses, the good ones and the lucky ones and the outdated ones, with no mechanism to ever find out which is which.

The source: a working human playbook

The primary source for this case study is a real, working sales playbook - the one authored by Andres Contreras-Grassi - which Ernesta Labs obtained as the client-side playbook for the sales motion NIKO serves. We will not reproduce its scripts here; the material is copyrighted, and our conversion deliberately does not depend on quoting it. What matters for the engineering lesson is its structure, because the structure is what converted well.

PRIMARY SOURCE RESULT: the playbook organizes a sales conversation into four stages. Stage 1, Intent: establish what the prospect actually wants and why, before anything else. Stage 2, Logical Certainty: establish that the prospect logically believes the problem is solvable and that your approach is the credible path. Stage 3, Emotional Certainty: establish that the prospect emotionally believes it will work for them, in their situation - which is a different and later state than logical agreement. Stage 4, Pitch: only now present the offer, priced and framed against the certainty already built. The stages are sequential and gated: the playbook is explicit that presenting in Stage 1 territory, or pitching before emotional certainty exists, collapses close rates.

PRIMARY SOURCE RESULT: beyond the spine, the playbook carries specialized tactics as separable units - negative-reverse style questions that lean away from the sale to lower pressure, objection handling for the recurring pushback categories, adaptive selling adjustments keyed to prospect type, reframes for the common failure moments, and separate material for outreach and booking. In the source document these are sections you turn to; they are not read cover to cover before every sentence you speak. That property - consult the spine always, consult the tactic when the situation calls - is exactly the lazy-loading pattern, and recognizing it was the turning point of the conversion.

We want to be precise about epistemic status: the playbook's own effectiveness is claimed by its author from his own practice, not established by controlled study in anything we have access to. We did not adopt it because it is proven. We adopted it as a candidate because it is specific, structured, stage-gated, and falsifiable - the properties that make a candidate skill evaluable at all.

The conversion: spine as stages, tactics as lazy modules

The conversion followed the SOURCE to CANDIDATE CAPABILITY rule we use for all external expertise, the same rule the book-to-skill repository applies to whole books: source material becomes candidate skill specifications, never direct behavioral authority. Four staged skills now form the conversation's spine, each with a short always-loaded specification: the stage's goal, its entry condition, its exit condition, and the failure signal that says the conversation is not ready to advance. The heavy tactical detail lives off-spine, in lazily loaded modules that the runtime attaches only when the conversation state reaches them: a Negative Reverse module, an Objection Handling module, an Adaptive Selling module, a Reframes module, and Outreach/Booking modules for the pre-conversation phases.

The load decision is state-driven, not model-driven. When the tracked conversation state is Stage 2 and a price objection appears, the objection-handling module is attached; when the objection is resolved, it detaches. The model cannot summon modules by whim, and it cannot keep them attached: context is compiled for the current decision, which keeps a late-conversation turn from paying the discovery-question tax. Each module carries provenance metadata - which playbook section it came from, who converted it, when it was last evaluated - so every behavioral choice in production can be traced to the candidate that produced it.

Everything is versioned as candidates. The original giant prompt is preserved as the baseline candidate. Every module is a separate candidate with its own identity. Nothing is ever edited in place in production; a changed module is a new candidate that must run the evaluation gauntlet against the incumbent. That is what made the attribution problem tractable: because modules attach and detach per state, we can measure objection handling specifically, on conversations that contained objections, with and without the candidate module, holding the rest of the spine constant.

Why the giant prompt lost

PRIMARY SOURCE RESULT (our internal conversion result, measured on our own conversation logs and evaluation suites): the staged-plus-lazy version beat the giant prompt on every axis we instrumented. Context cost per conversation turn dropped substantially, because most turns need the spine plus at most one tactical module, not the whole playbook. Stage discipline became enforceable: the giant-prompt baseline repeatedly pitched in Stage 1 and Stage 2 territory - roughly the failure mode the playbook itself warns against - because thousands of undifferentiated instructions dilute any single constraint, while the staged version could be hard-gated so that pitch material is simply absent from context until Stage 4 conditions hold. And per-component evaluation became possible for the first time: we could hold the spine fixed and A/B a single objection-handling candidate, which the monolithic prompt structurally cannot offer.

One more result deserves a label of a different kind. The staged design also changed our relationship to the playbook's claims. Once modules were individually versioned and evaluable, the playbook stopped being a single adopt-or-reject artifact and became a portfolio of testable hypotheses. Some of its tactics have survived early evaluation; others have not yet been through enough live conversations to judge; we are not going to pretend the split is known. That is the honest state, and it is a better state than the giant prompt's, where we could not even ask the question.

The pipeline: source -> candidate -> evaluation -> challenger -> evidence -> promotion/rejection

The general procedure we extracted from this conversion, and the reason we are publishing it as a case study rather than a sales tip: human expertise moves through a fixed pipeline before it earns behavioral authority. SOURCE: the playbook, trusted as a description of one expert's method, nothing more. CANDIDATE: a versioned, provenance-tagged skill specification extracted from the source, loaded lazily, gated by state. EVALUATION: the candidate runs against real conversations or recorded ones, on component-specific criteria - for objection handling, did the objection recur, did the conversation advance - with the evaluator treated as an advisory signal, not an oracle, per the LLM-as-a-Judge failure literature. CHALLENGER: the candidate must survive a deliberate attempt to beat it with a simpler alternative, including the empty alternative, because a module that loses to no-module is theater. EVIDENCE: the accumulated record, with conversation counts stated. PROMOTION or REJECTION: a human decision from evidence, reversible, recorded.

The pipeline's discipline is mostly about what it forbids. The expert's confidence is not evidence. The evaluator's preference is not evidence. A polished demo conversation is not evidence. And rejection is cheap and normal: several of the first tactical candidates we extracted from the playbook were rejected or sent back for rework, and that is the pipeline working, not failing.

Limitations: what this case study does not show

This is one conversion of one playbook, done by us, evaluated on our own conversations. The playbook's effectiveness in human hands is author-claimed, not study-verified, and we inherited that uncertainty into the candidate pool rather than eliminating it. Our evaluation volumes are modest and growing - early-stage modules have seen hundreds of live turns, not tens of thousands - and results at this scale are directional, not conclusive.

Most importantly: signup is not customer, sent email is not sale, and improved conversation structure is not revenue. We have measured that the staged system is cheaper, more disciplined, and evaluable. We have not yet demonstrated that it closes more deals, because deal outcomes lag conversations by weeks and our evidence window is still short. Anyone who reads this article as proof of revenue lift is reading it wrong. The status label IMPLEMENTED describes the conversion and the pipeline, not a proven commercial outcome.

We also note the scope limit of the decomposition method itself. It worked because this playbook is stage-gated with separable tactics. Expertise that is tacit, continuous, or heavily improvisational may not decompose this cleanly, and the harder failure is expertise that looks decomposable but is not - where the value lives in cross-stage judgment no module boundary respects. We have not solved that; we have one example where the boundaries were favorable.

Why builders care, beyond sales

The pattern is not a sales pattern. Every domain with senior human practitioners has a playbook: runbooks in operations, triage scripts in support, review checklists in security, control mappings in compliance, search strategies in research, escalation paths in customer service. All of them are currently being pasted into giant prompts by someone. All of them have the same three structural failures: context tax, no attribution, no evaluation surface.

Support: convert the triage flow into staged gates (intake, reproduction, classification, resolution) with lazy modules for the recurring issue categories; then you can A/B your refund policy module without touching intake. Security: staged review (scope, exposure, exploitability, decision) with lazy modules per vulnerability class, each candidate evaluated against how often its advice survived challenge. Compliance: the controls themselves are already stage-structured - that is what a checklist is - and lazy modules per jurisdiction or per control family make the always-loaded context tiny and auditable. Operations: runbooks are the most playbook-shaped artifact in industry; stages map to incident lifecycle phases, modules to remediation playbooks. Research: staged (question, sources, evidence, synthesis) with lazy modules per source type. The generalization is mechanical because the failure mode is mechanical.

LABS INTERPRETATION and LABS RECOMMENDATION

LABS INTERPRETATION: the conversion succeeded for a reason that generalizes. Human playbooks that survive contact with real work are already decomposed - the practitioner does not re-read the whole document before every sentence; they hold a spine and consult tactics situationally. The giant prompt destroys that structure and pays for it three times: attention, attribution, and evaluability. Decomposition is not a token-saving trick; it is the only representation in which a human expert's hypotheses can be individually tested, individually rejected, and therefore individually improved. And treating third-party expertise as untrusted skill providers - with provenance, versioning, and evaluation before promotion - is the same posture the SkillShift findings demand for skills generally: functional correctness at the component level does not certify policy alignment at the trajectory level, and a sales playbook is nothing but policy claims about how to treat people.

LABS RECOMMENDATION: if you have a playbook, a runbook, a triage script, or a senior practitioner's document, do this today. First, find the spine: the sequential, gated stages that order the work - if the document has none, interview the expert until you find one, because the spine is what makes everything else lazy. Second, extract tactics as modules with explicit entry and exit conditions, attach them by state, and detach them when the state passes. Third, version every module as a candidate with provenance to its source section. Fourth, wire the pipeline - evaluation on component-specific criteria, a challenger including the empty module, evidence with counts, human promotion or rejection. Fifth, keep the giant prompt as a baseline and beat it on measurement, not vibes.

What we would NOT implement: we would not paste a copyrighted playbook into a prompt verbatim - beyond the legal problem, it defeats the provenance and evaluation model. We would not let the model decide which module to load, because lazy loading then becomes a capability to summon whatever the model finds persuasive, which is exactly the attack surface covert skill-steering exploits. We would not auto-promote modules on evaluator scores alone; the evaluator is advisory and the promotion authority is human. And we would not treat an expert's endorsement of their own material as evidence - the expert's confidence is the prior you are testing, not the result.

The playbook decomposition test: run it on your own expertise document today

You need one artifact you already have - a sales playbook, support triage script, incident runbook, or security review checklist - and one afternoon. No new infrastructure required; the first pass is structural, and you can test it with paper and a chat model.

Step 1 - find the spine. Read the document and mark the sequential, gated stages: which stage must be true before the next one starts. Write them as a list with entry and exit conditions, one line each. If you cannot find gates, interview the expert who wrote it and ask what has to be true before they move forward; their answer is the spine. Step 2 - separate the spine from the tactics. Everything that is consulted situationally rather than always - the scripts, the category-specific material, the special cases - becomes a module candidate. Give each module a trigger: the conversation state (or incident state, or case state) at which it attaches, and the state at which it detaches.

Step 3 - build both versions. Version A: the whole document in one prompt. Version B: the one-page spine always loaded, modules attached only on their triggers. Step 4 - run the lazy-fidelity test. Take ten real transcripts or cases from your own history. For each, walk Version B's state machine by hand and note which modules would have attached at each turn, and what Version B's context would have contained versus Version A's. If Version B would have missed material the outcome depended on, your trigger conditions are wrong - fix them before touching a model. Step 5 - run the discipline test. Ask both versions for their move at a Stage 1 moment - first contact, no intent established, no certainty of any kind. Version A will usually pitch anyway; a correctly gated Version B physically cannot, because the pitch module is not in context. That difference, visible in one afternoon, is the whole argument.

The follow-up test, once you have live traffic: pick the module your team argues about most, and A/B it against the empty module on real cases, with counts. The argument ends when the evidence does, and half the time the evidence ends it differently than the loudest expert predicted.

What remains unknown

• Whether the Contreras-Grassi playbook's claimed effectiveness in human hands holds under controlled measurement at all - it is author-practitioner-claimed, and our conversion inherited that uncertainty rather than resolving it.
• Whether the staged system converts better commercially than the giant-prompt baseline: conversation-structure improvements are measured, but deal outcomes lag by weeks and our evidence window is not yet long enough to claim revenue lift.
• How far the decomposition method extends to tacit or improvisational expertise: we have one favorable example (a stage-gated playbook), and no evidence about expertise whose value lives in cross-stage judgment no module boundary respects.
• Whether module-level evaluation in live conversations generalizes across prospect and market types: early evaluation volumes are in the hundreds of turns, directional at best, and segment-level results do not yet exist.

Primary sources

• Ernesta Labs internal case study - sales playbook to candidate skill conversion (this article's primary source)
• book-to-skill - compiling books into compact agent skills with lazy supporting files (the closest public analogue to our conversion method)
• LLM-as-a-Judge Is Not an Oracle - why our evaluators are advisory signals, not promotion authorities
• A Finger on the Scale: Covert Policy Steering through Agentic Skills - why third-party skill content is an untrusted policy provider