Migrating to LanceDB 0.41.0: Embedding Trust Fix and Lazy Iterators

Every Python application on LanceDB 0.40.0 or earlier that opens tables it did not create should move to LanceDB 0.41.0, released on October 29 after a series of betas. The release fixes GHSA-x8v2-vj73-gfwm, rated high: a crafted table could make the process that opens and queries it run arbitrary code. The fix changes defaults, so applications that reopen tables using remote-code models or custom embedding endpoints must add an explicit approval, and several listing APIs now return iterators.

Why Opening a Table Could Run Code #

LanceDB rebuilds a table's embedding function from the table's own Arrow metadata. For the built-in sentence-transformers function that metadata included the model name and trust_remote_code, which defaulted to True. The advisory shows the result: the first query that computed embeddings called SentenceTransformer(name, trust_remote_code=True) and executed the custom Python of whatever model repository the table named. The victim only had to open and search the table, which makes shared datasets, downloaded artifacts and multi-tenant stores the risky cases. The advisory, classed as CWE-94, was validated on 0.40.0. PR #4550 adds that table metadata could also point an OpenAI base_url at another server, sending it queries and the OPENAI_API_KEY from the environment.

This is the old flow, which on 0.40.0 ran the named model's code when the table carried such metadata:

import lancedb

db = lancedb.connect(path)
tbl = db.open_table(name)
tbl.search("q").limit(1).to_list()

This is the new way: approve the trusted model, pinned to a reviewed revision, and any custom endpoint before the process reopens such tables:

from lancedb.embeddings import get_registry

registry = get_registry()
registry.allow_remote_code("your-trusted-model", revision="your-reviewed-commit")
registry.allow_endpoint("https://your-trusted-endpoint.example")

Trust is keyed by model name, optionally with a revision, so approving one model does not let a table swap in another. Functions created in the same process are approved automatically, and SentenceTransformerEmbeddings.trust_remote_code now defaults to False. The PR lists what it does not cover: TypeScript, whose built-ins have neither remote code nor configurable endpoints, Bedrock assumed_role, and third-party functions that do not implement the new hooks.

Iterators, Renames and Other Breaking Changes #

list_databases now returns an iterator (PR #4442), so callers no longer page manually; the iterator exposes num_page_results and page_token for those who want control. Table and namespace listings return lazy iterators as well (#4599, #4459). Code that indexed into the old list result needs list(...) or a loop. 0.41.0 also renames the LSM APIs to MemWAL and then to RTE, the realtime engine (#4075, #4598), and makes several Python signature changes, including the parameter order of RemoteDBConnection.create_table (#4608) and an error on bool(Expr) instead of silently dropping predicates (#4682).

The release uses Lance 14.0.2. Local SQL computed columns and materialized views stay experimental behind the experimental-functions build feature, which the published Python and Node packages leave disabled. Restoring a remote table now requires a checked-out version and fails with an input error instead of a server deserialization error (#4406).

Migration Steps #

1. Upgrade to 0.41.0 or later; the 0.42 line is still in beta.
2. Before the first search in each process that reopens tables using remote-code models, call get_registry().allow_remote_code(model, revision=...) for each trusted model, and allow_endpoint(url) for custom OpenAI, Ollama or watsonx endpoints.
3. Do not approve models named by tables from untrusted or shared sources; leave those to raise UntrustedEmbeddingConfigError.
4. Replace indexing into list_databases() and table or namespace listings with iteration or list(...).
5. Rename calls to the LSM or MemWAL APIs to the RTE names, and fix keyword arguments where Python signatures changed.

Verification Checklist #

• pip show lancedb reports 0.41.0 or later in every environment that opens tables.
• Searching a table with an approved remote-code model works in a fresh process, and an unapproved one raises UntrustedEmbeddingConfigError.
• No code relies on trust_remote_code defaulting to True.
• Listing calls are consumed as iterators.