Finite dataset embedding contract audit

Initial decision: Prototype diagnostic. This product hypothesis helps a search or representation team examine distance-preservation claims. It does not implement the new existential dimension-reduction theorem.

Research finding

Family 094 gives subpolynomial target dimension for fixed distortion greater than one in the same real Lp geometry, while exact embeddings have quadratic worst-case dimension for p other than two. The manuscript explicitly states that it does not assert an efficient embedding algorithm. Family 099 supplies a separate warning about growing worst-case L1 distortion for edit distance. Lp manuscript, Lp formal scope, edit metric scope.

Problem and buyer

A vector-search provider, search team or representation vendor may promise that compression preserves distances. Reviewers need to know which source metric, target metric, finite dataset, scaling convention and query regime support that promise. Distance preservation and ranking accuracy are different claims.

What the finding could enable

A contract-audit framework could accompany embeddings with explicit applicability records and measured distortion. The new results sharpen what is theoretically possible or obstructed. Finite all-pairs auditing itself predates this release; the proposed value is connecting theorem scope to operational claims and repeatable review.

Technical and commercial limits

An empirical audit of supplied points does not certify the full real-space mapping or unseen query behavior. Nonlinear existential embeddings need not provide a cheap out-of-sample transform. An L2 random-projection bound does not establish the same guarantee in another Lp norm. Large datasets require quadratic pair work in the initial audit; sampling pairs weakens completeness.

Minimal architecture

Metric and dataset specification -> embedding import -> finite all-pairs diagnostic -> duplicate and collapse detection -> retrieval-task evaluation -> reviewed claim record. Keep mathematical contract, floating-point diagnostic and downstream search benchmark separately labeled.

The local research/tools/audit_embedding.py prototype reports the ratio between largest and smallest pairwise target/source distance ratios after global rescaling. A collapsed positive-distance pair or inconsistent representation of a duplicated source point prevents a bounded distortion claim. Its numeric output is empirical and uses no formal verification or interval arithmetic.

Existing alternatives and differentiation

Scikit-learn already offers random projection and a Euclidean Johnson–Lindenstrauss dimension estimate. A new compressor without a demonstrated construction is weakly differentiated. The narrower opportunity is an independent metric-contract review and reproducible integration across compression backends. Scikit-learn Euclidean projection documentation.

Monetization hypothesis

Assume an initial AUD 3,000-10,000 offline assessment for a team with a real compression rollout, followed by a supported CI integration if the assessment recurs. These are proposed pilot values, not observed contracts. Saved compute, latency and storage must be measured on the actual pipeline; no profitability is established.

Validation experiment

Check all six pairs of a four-point square under exact uniform scaling: the global-rescaling distortion should be one. Collapse two distinct vertices: the diagnostic should report no bounded distortion. Add unequal-ratio, duplicate, malformed-input and out-of-sample query fixtures before treating the prototype as a robust evaluation library. For commercial validation, compare reviewer time and detected unsupported claims with an existing manual rollout review.

Conditions to reject or defer

Reject if buyers already perform equivalent checks cheaply, if their primary objective is task accuracy rather than metric fidelity, or if a claimed theorem-derived compression improvement has no executable construction. Defer certified numeric claims until a validated arithmetic implementation exists.

Next concrete action

Run the small fixtures, add a known distorted but injective map and explicitly test unseen-query failure. Investigate a pilot workflow without outreach or service purchases.