Qdrant Scaling Query Volume — Agent Skill by GitHub

From the author

Guides Qdrant query volume scaling. Use when someone asks 'query returns too many results', 'scroll performance', 'large limit values', 'paginating search results', 'fetching many vectors', or 'high cardinality results'.

— the description by anush008

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides guidance on scaling query volume in Qdrant when dealing with large result sets across multiple shards. It addresses the inefficiency of naively merging full result sets from every shard by implementing a Poisson distribution-based subsampling strategy. By requesting a smaller, statistically calculated limit from each shard, the system significantly reduces data transfer overhead while maintaining high accuracy, making it ideal for high-cardinality or large-limit search scenarios.

What it does

Explains the inefficiency of merging full result sets from multiple shards.
Introduces a Poisson distribution-based approach for calculating shard-level limits.
Defines the conditions under which subsampling is automatically triggered.
Balances query accuracy against inter-shard network traffic.
Provides a strategy for handling large limit and offset values in distributed vector searches.

When to use it

When queries return too many results or require large limit values.
When optimizing performance for paginated search results.
When fetching many vectors across a multi-shard Qdrant deployment.
When dealing with high-cardinality results in auto-sharded environments.

Advantages

Significantly reduces inter-shard data transfer volume.
Maintains high result accuracy comparable to approximate HNSW indexing.
Improves performance for high-limit queries.
Leverages statistical safety factors to minimize error rates.

Usage examples

“How can I optimize my Qdrant query when I need to fetch 1000 results across 10 shards?”: The agent explains the Poisson subsampling strategy to reduce the data transferred from each shard while maintaining result integrity.
“Why is my paginated search in Qdrant slow when using large limits?”: The agent identifies the overhead of merging large result sets and suggests applying the subsampling guidelines for distributed queries.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/github/qdrant-scaling-query-volume/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip qdrant-scaling-query-volume.zip -d .agents/skills

Documentation by anush008

Scaling for Query Volume

Problem: When a query has a large limit (e.g. 1000) and there are multiple shards (e.g. 10), naively each shard must return the full 1000 results — totaling 10,000 scored points transferred and merged. This is wasteful since data is randomly distributed across auto-shards.

Core idea

Instead of asking every shard for the full limit, ask each shard for a smaller limit computed via Poisson distribution statistics, then merge. This is safe because auto-sharding guarantees random, independent data distribution.

When it activates

More than 1 shard
Auto-sharding is in use (all queried shards share the same shard key)
The request's limit + offset >= SHARD_QUERY_SUBSAMPLING_LIMIT (128)
The query is not exact

Key tradeoff

The strategy trades a small probability of slightly incomplete results for a large reduction in inter-shard data transfer, especially for high-limit queries across many shards. The 1.2x safety factor and the 99.9% Poisson threshold keep the error rate very low — comparable to inaccuracies already introduced by approximate vector indices like HNSW.

Files

ATTRIBUTION.mdAdded by AutoGineer895 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md1.4 KiB

LICENSE

MIT License

Copyright GitHub, Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Requirements and compatibility

Written for: GitHub Copilot

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Anndata
K-Dense· Community
A Python framework for managing annotated data matrices in single-cell genomics and general research workflows.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskGitHub stars of the source repository56.4k
Liteparse
K-Dense· Community
Local document and PDF parsing with spatial text extraction and bounding box support.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
Apache-2.0Low riskHas scriptsGitHub stars of the source repository56.4k
Usfiscaldata
K-Dense· Community
Access U.S. Treasury federal financial data via the Fiscal Data REST API.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskGitHub stars of the source repository56.4k
Geopandas
K-Dense· Community
Guidance and local audit tools for Python workflows using GeoPandas 1.2.0 for spatial data analysis.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskHas scriptsGitHub stars of the source repository56.4k
Markitdown
K-Dense· Community
A lightweight utility for converting diverse document formats into structured Markdown for analysis, search, and LLM ingestion.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskHas scriptsGitHub stars of the source repository56.4k
Zarr Python
K-Dense· Community
A guide for storing and querying chunked N-D scientific arrays using Zarr-Python 3.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskGitHub stars of the source repository56.4k