跳到内容
ClickHouse Loom
Esc
↑↓导航↵打开⌘J预览
本页内容

Proposed software factory pilot

This draft measures whether application-owned context retrieval changes objective resolution, total calls, tokens, cost per successful resolution and elapsed time. It compares three cold arms under the same tools, graders and resource limits. It measures controlled context utility; three tasks cannot establish adoption, savings or a general quality improvement.

The plan is not frozen or approved. Publishing or merging it authorizes no model calls, index ingestion or paid benchmark. No live result is reported here.

Implementation and tasks

The factory implementation supplies offline replay, a bounded controller, injected provider/SDK bindings, authored host behavioral checks, attempt accounting and paired analysis. A qualified live driver and deployment-specific evidence remain prerequisites. The example’s offline replay fixtures are separate from the historical pilot.

Use the three historical cases in the implementation’s historical-candidates.json: pallets__flask-4045, pallets__flask-4575 and pallets__flask-4935. Each task pins its exact public source base and dependency image digest. The imported official test selectors are diagnostic inputs: candidate Python controls its own test process, exit status and reports, so those reports cannot authorize a passing final grade. Retain all failures. Do not replace tasks or choose the best attempt after observing results.

The proposed objective grader uses narrow authored public-behavior probes: dotted blueprint names, redirect method/delegation/override, and nested subdomains. The sandbox returns bounded primitive observations; evaluator-owned assertions and the final verdict stay outside candidate mounts. This coverage is not equivalent to the imported official selectors. The final protocol must explicitly choose this coverage and freeze its probe seed, bridge and assertions.

Before running, independently verify passing visible neighbors on the base, a failing target check on the base, and passing target/neighbors with the reference on every exact image. Candidate syntax/import crashes or malformed output remain failed checks so revision can continue; infrastructure, state or cleanup faults invalidate grading. Recheck forgery, source-write and output-limit controls. Imported receipts and image identity alone do not qualify another grading path.

Arms and schedule

Arm Application behavior Agent exposure
Baseline No context client construction or retrieval No Loom schema, instructions or hooks
Orchestrated Exact intake lookup, impact before changed-source verification, failure retrieval as needed Ordinary source tools; no selected-call quota
Natural No automatic context hooks Optional exact lookup with zero minimum calls

All arms share the same provider/model, native response handling, source tools, objective graders and per-attempt limits. Only retrieved context and its resulting trajectory may differ. The treatment cannot change the source base, hidden tests, prices, schedule, provider or retained attempts.

Use 3 tasks × 3 paired repeats × 3 arms = 27 primary attempts. Add 3 tasks × 2 baseline A/A copies = 6 reference attempts, for 33 attempts total. The 15 baseline attempts allow zero Loom requests. All attempts start cold; no cross-task memory writes, warm reuse or chronological learning is part of this pilot.

The proposed schedule uses seed 741 and persists all 33 slots before calls. Its schedule_sha256 covers the canonical attempt list. Freeze this order with the final protocol; a seed reproduces the schedule and analysis, not the model’s responses. Retain failures, nonuse and budget stops without replacement attempts, automatic retries or post-hoc exclusions.

Proposed limits

Resource Per attempt Whole primary and A/A cohort
Generation requests 12 396
Input tokens 20,000 per generation request, subject to a verified bound 7,920,000
Output tokens 2,048 per generation request 811,008
Agent tools 30 990
Logical Loom reads 6 in each context arm; zero in baseline 108
Physical Loom HTTP requests 18 in each context arm; zero in baseline 324
Serialized transcript 64,000 characters per generation request Enforced on every request
Accepted context 12,000 characters total Enforced on every attempt
Visible tests 3 invocations, 30 seconds each No replacement runs
Independent final hidden grade 1 invocation, 30 seconds No model judge
Agent policy deadline 180 seconds Serial bounded attempts
Hard controller deadline 210 seconds, including hidden grading Retain interrupted attempts

The physical Loom cap assumes one retrieval and two single-request scope checks for each logical read. Auth, discovery, retries, failed dispatches and any extra checks consume the same cap. Stop before another physical request would exceed it; additional traffic requires a revised protocol and approval.

The generation request cap excludes a token-counting service. If a verified counting mechanism needs network requests, freeze a separate ceiling of at most 396 counting requests, their endpoint, latency and cost before approval. Do not silently replace an unsupported counting API or treat characters as token counts. The proposed 20,000-input-token bound remains unqualified.

After stopping a controller, reconcile at most four recorded, owned containers. Each has a 10-second inspection limit and a 10-second removal limit, for at most 80 seconds of additional cleanup. Record end-to-end elapsed time including setup, independent grading and cleanup. Incomplete cleanup halts the cohort. A timeout or unresolved in-flight request remains a failure with unknown usage.

Provider and spending freeze

Use one exact model through a direct Converse provider, with existing authorized configuration, standard inference, no temperature default and no explicit cache checkpoints. Preserve native reasoning blocks. Freeze its exact model identifier, region, endpoint, API behavior and complete dependency hashes before approval. Do not introduce an alternate gateway, provider or credential route to complete a run. No infrastructure, permission or credential changes are authorized here.

The working price assumption is $2.20 per million input tokens and $11 per million output tokens. These assumptions require account-specific verification for the exact model, region, inference class and cache semantics. They are not a promise of current pricing or a complete bill.

At the proposed token caps, generation reserves at most 7,920,000 / 1,000,000 × 2.20 + 811,008 / 1,000,000 × 11 = $26.345088. The proposed total ceiling is $30, conditional on bounding all counting, server, transport, auth, cache and infrastructure overhead within the remaining $3.654912. Unknown overhead is unknown, not zero. If verified token bounds or all-in costs do not fit, revise the plan before requesting approval.

No dollar amount in this draft authorizes spending. Freeze the final protocol digest, schedule, dependency locks, price table and whole-run reservation, then obtain actual owner approval referencing that digest and ceiling. Approval of benchmark intent or of a PR does not replace that approval.

Qualification required before the first request

  1. Pin the merged server/SDK integration and factory controller commits. Check the actual integration and its required CI evidence separately.
  2. Freeze public task inputs, the host probe seed and assertions, observation bridge, source allowlists, tool schemas, prompts, runtime locks, controller and trusted worker hashes. Keep evaluator assertions, reference fixes and credentials outside agent access. Official test reports remain diagnostic.
  3. Qualify a source-only checkout per attempt, with no Git history, future objects, sibling mounts, network or credential access. Candidate code runs in an isolated focused grader. Verify traversal, symlink, import, subprocess and egress isolation. Export only allowlisted implementation edits into a separate independent final grader; preserve evaluator tests and runner configuration.
  4. Use a private server and database owned by this pilot; never reuse a production server. Supply authorized immutable index scopes for all three exact source bases using reader credentials. Use exact retrieval only, with no embeddings, conceptual search or model enrichment. Verify repository, full commit, server build, permissions and every required index tier before and after reads. External immutability evidence is mandatory; matching watermarks alone are insufficient. Restricted zero counts mean unknown. No index creation or grants are implied.
  5. Qualify the actual controller/worker and client transport paths, hard deadlines, durable pre-dispatch reservations and cleanup. Baseline constructs no context client. Instrument auth and metadata side traffic or refuse it. Failed or interrupted attempts retain their reservations and require reconciliation.
  6. Verify the input-token bound and complete usage/pricing semantics. Join client request IDs to model and server billing evidence. Count failed and retried physical dispatches. Never record credentials or arbitrary response headers.
  7. Obtain exact protocol and total-spend approval after all evidence exists. Refuse inference when a required field or receipt is missing.

Offline contracts, stubbed transports, scripted replay and proposed schedules are preparation evidence. They establish no live model resolution, complete billing, index authorization or spending permission.

Measurement and analysis

Retain every scheduled attempt and its independent objective result, invalid-run reason, source and patch hashes, provider/index provenance, failures, nonuse, abstentions and stale, repeated or empty contexts. Separate accepted and failed calls, host-selected and agent-selected calls, logical reads, physical dispatches and retries. Avoid counting an agent-selected Loom call twice in tool totals.

Keep raw input, output and cache usage, priced and unknown cost, component latency and total elapsed time. Cost per successful resolution includes failed attempts; zero successes has no finite estimate. Unknown usage or pricing suppresses an all-in cost estimate. Whole-cohort cost includes A/A references.

Pair by task and repeat. Average repeats within each task and use seed 741 for paired bootstrap intervals that resample task clusters with arms kept together. Report A/A variation separately. Missing attempts remain visible and suppress partial-cohort intervals. Three task clusters support exploratory intervals, with no significance, percentage-gain, general adoption or savings claim.

Freeze original outputs before fixing any observed failure. Diagnose a concrete pattern, add a focused regression, fix the smallest cause and compare under a preserved equivalent protocol. Never tune against hidden solutions. Any rerun needs its own reserved budget and approval; it is not a free replacement attempt.

这个页面有帮助吗?