Cosine Similarity vs Dot Product in Vector Search

Cosine similarity compares vector direction. Dot product compares direction and length together. They produce equivalent scores for unit-length vectors, but they can rank unnormalized vectors differently. Choose the metric expected by your embedding model, then verify what your index actually computes.

This distinction matters when a search upgrade changes normalization or when two services use different defaults. A request can succeed, return numbers and still rank documents incorrectly. A small numerical fixture is often more useful than staring at a large array of embeddings.

Work through a two-dimensional example

Use a query vector q = [1, 0] and two document vectors: a = [0.8, 0.6] and b = [2, 2]. The dot products are 0.8 and 2, so dot product prefers b. The cosine values are 0.8 and approximately 0.707, so cosine prefers a.

Vector b gains a dot-product advantage from its greater length. Vector a points closer to the query direction. Neither calculation is defective; they answer different mathematical questions. Real embeddings contain more dimensions, but the same relationship applies.

from math import sqrt
def dot(a, b):
    if len(a) != len(b):
        raise ValueError("dimension mismatch")
    return sum(x * y for x, y in zip(a, b))
def cosine(a, b):
    denominator = sqrt(dot(a, a) * dot(b, b))
    if denominator == 0:
        raise ValueError("zero vector")
    return dot(a, b) / denominator

This deliberately simple reference is suitable for tiny test fixtures. Use a numerical library and the model's supported implementation for production workloads.

Check the model's intended metric

Some embedding systems normalize their outputs; others intentionally preserve length. Do not normalize every model automatically because cosine sounds familiar. Length may carry information that a particular training objective expects the retrieval metric to use.

Sentence Transformers' similarity documentation exposes multiple similarity functions. The practical implication is that a function name belongs in the deployment configuration, alongside normalization and model version. A library default should not become an undocumented product decision.

Inspect a redacted configuration with JSON Viewer. Compare the search service and indexing service settings with JSON Diff. A difference such as normalize: false versus true is more consequential than whitespace or field order.

Verify what the database returns

A vector database may expose similarity, distance, negative inner product or another transformed value. Higher may be better in one API and lower may be better in another. Read the index documentation before sorting results or applying a minimum-score filter.

Insert the tiny vectors above into a temporary test collection you control. Query them and compare the reported ordering with your reference calculation. Then repeat with normalized vectors. This establishes the behavior of the actual API rather than the behavior you hoped its metric label implied.

Include a dimension mismatch and a zero vector in the test plan. Your application should reject malformed vectors with a clear error rather than quietly padding, truncating or dividing by zero. Also check how it handles non-finite values; NaN and infinity should not become normal searchable coordinates.

Why 0.8 is not a universal relevance threshold

A cosine score is not a calibrated probability that an answer is correct. Its useful range depends on the model, domain and candidate set. A threshold copied from a different model can reject good results or admit unrelated ones.

Collect real queries with reviewed relevant and irrelevant documents. Examine their score distributions and choose a threshold that meets the product's needs. A support assistant may prefer to admit uncertainty rather than return a confident answer from a weak match. A browsing interface may reasonably show more candidates.

Revisit thresholds after a model migration, normalization change or content-domain expansion. Also review the interaction with reranking. A first-stage embedding score and a reranker's score have different meanings; do not compare their raw numbers as though they share a scale.

Separate numerical correctness from useful search

Passing a vector fixture proves that the metric is wired correctly. It does not prove that the embedding model understands your documents. You still need task-level judgments, coverage checks and access-control tests.

When a query fails, first ask whether the correct document exists in the candidate index. Next inspect whether filters removed it. Only then investigate whether the model or metric ranked it poorly. This sequence prevents a team from tuning cosine thresholds to compensate for a missing document.

Keep the calculation test deterministic and small. Record vectors, expected rankings and allowed floating-point tolerance. Exact string comparison of floating-point scores is brittle across execution environments; compare within a justified tolerance while preserving the expected order.

Keep an exact-search reference for a small corpus when testing an approximate index. If approximation changes the candidate set, a ranking disagreement may come from search recall rather than the cosine or dot-product calculation itself.

Pick deliberately, then test the entire path

Use the model's recommended metric and an explicit normalization policy. Confirm database score direction with a numerical fixture, then judge relevance using your own queries. If the problem appeared during an upgrade, review the embedding migration compatibility plan before changing thresholds.