Getting Started with bigANNOY

bigANNOY is an approximate nearest-neighbour package for bigmemory::big.matrix data. It builds a persisted Annoy index from a reference matrix, searches that index with either self-search or external queries, and returns results in a shape aligned with bigKNN.

This vignette walks through the first workflow most users need:

create a small reference matrix
build an index on disk
run self-search and external-query search
inspect the returned neighbours and distances
reopen and validate the index in a later step

The examples are intentionally small, but the same API is designed for larger file-backed big.matrix inputs.

Load the Packages

library(bigANNOY)
library(bigmemory)

Create a Small Reference Matrix

bigANNOY is built around bigmemory::big.matrix, so we will start from a dense matrix and convert it into a big.matrix.

ref_dense <- matrix(
  c(
    0.0, 0.1, 0.2, 0.3,
    0.1, 0.0, 0.1, 0.2,
    0.2, 0.1, 0.0, 0.1,
    1.0, 1.1, 1.2, 1.3,
    1.1, 1.0, 1.1, 1.2,
    1.2, 1.1, 1.0, 1.1,
    3.0, 3.1, 3.2, 3.3,
    3.1, 3.0, 3.1, 3.2
  ),
  ncol = 4,
  byrow = TRUE
)

ref_big <- as.big.matrix(ref_dense)
dim(ref_big)

#> [1] 8 4

The reference matrix has r nrow(ref_dense) rows and r ncol(ref_dense) columns. Each row is a candidate neighbour in the final search results.

Build the First Annoy Index

annoy_build_bigmatrix() streams the reference rows into a persisted Annoy index and writes a sidecar metadata file next to it.

index_path <- tempfile(fileext = ".ann")

index <- annoy_build_bigmatrix(
  ref_big,
  path = index_path,
  n_trees = 20L,
  metric = "euclidean",
  seed = 123L,
  load_mode = "lazy"
)

index

#> <bigannoy_index>
#>   path: /var/folders/h9/npmqbtmx4wlblg4wks47yj5c0000gn/T//RtmpBEyDSE/fileb1f61d3b7074.ann
#>   metadata: /var/folders/h9/npmqbtmx4wlblg4wks47yj5c0000gn/T//RtmpBEyDSE/fileb1f61d3b7074.ann.meta
#>   index_id: annoy-20260327203933-d4a614637b45
#>   metric: euclidean
#>   trees: 20
#>   items: 8
#>   dimension: 4
#>   build_seed: 123
#>   build_threads: -1
#>   build_backend: cpp
#>   load_mode: lazy
#>   loaded: FALSE
#>   file_size: 2816
#>   file_md5: d4a614637b45839a9eb126d130a96397
#>   prefault: FALSE

A few details are worth noticing:

the Annoy index lives on disk at index$path
metadata is written to index$metadata_path
load_mode = "lazy" means the object is initially metadata-only
the native handle is loaded automatically on first search

You can check the current loaded state directly.

annoy_is_loaded(index)

#> [1] FALSE

Run a Self-Search

With query = NULL, annoy_search_bigmatrix() searches the indexed reference rows against themselves. In self-search mode, the nearest neighbour for each row is another row, not the row itself.

self_result <- annoy_search_bigmatrix(
  index,
  k = 2L,
  search_k = 100L
)

self_result$index

2	3
1	3
2	1
5	6
4	6
5	4
8	4
7	4

round(self_result$distance, 3)

0.200	0.346
0.200	0.200
0.200	0.346
0.200	0.346
0.200	0.200
0.200	0.346
0.200	4.000
0.200	3.904

Because the first search loads the lazy index, the handle is now available for reuse.

annoy_is_loaded(index)

#> [1] TRUE

The result object follows the same high-level shape as bigKNN:

str(self_result, max.level = 1)

#> List of 8
#>  $ index   : int [1:8, 1:2] 2 1 2 5 4 5 8 7 3 3 ...
#>  $ distance: num [1:8, 1:2] 0.2 0.2 0.2 0.2 0.2 ...
#>  $ k       : int 2
#>  $ metric  : chr "euclidean"
#>  $ n_ref   : int 8
#>  $ n_query : int 8
#>  $ exact   : logi FALSE
#>  $ backend : chr "annoy"

In particular:

index is a 1-based integer matrix
distance is a double matrix
k, metric, n_ref, and n_query describe the search
exact is always FALSE for bigANNOY
backend is "annoy"

Search with an External Query Matrix

External queries are often the more common workflow in practice. Here we build a small dense query matrix with rows close to the first, middle, and final clusters in the reference data.

query_dense <- matrix(
  c(
    0.05, 0.05, 0.15, 0.25,
    1.05, 1.05, 1.10, 1.25,
    3.05, 3.05, 3.15, 3.25
  ),
  ncol = 4,
  byrow = TRUE
)

query_result <- annoy_search_bigmatrix(
  index,
  query = query_dense,
  k = 3L,
  search_k = 100L
)

query_result$index

1	2	3
5	4	6
7	8	4

round(query_result$distance, 3)

0.100	0.100	0.265
0.087	0.132	0.240
0.100	0.100	3.951

The three query rows each return three approximate neighbours from the indexed reference matrix. For small examples like this one, the results will typically look exact, but the important point is that the API stays the same for larger problems where approximate search is preferable.

Tune the Main Search Controls

Two arguments matter most when you begin tuning:

n_trees controls index quality and index size at build time
search_k controls search effort at query time

As a starting point:

increase search_k first if recall looks too low
rebuild with more n_trees when query-time tuning alone is not enough
keep metric = "euclidean" when you want the most direct comparison with bigKNN

The package also supports "angular", "manhattan", and "dot" metrics, but Euclidean is usually the easiest place to begin.

Stream Results into big.matrix Outputs

For larger workloads, you may not want to keep neighbour matrices in ordinary R memory. bigANNOY can write directly into destination big.matrix objects.

index_out <- big.matrix(nrow(query_dense), 2L, type = "integer")
distance_out <- big.matrix(nrow(query_dense), 2L, type = "double")

streamed <- annoy_search_bigmatrix(
  index,
  query = query_dense,
  k = 2L,
  xpIndex = index_out,
  xpDistance = distance_out
)

bigmemory::as.matrix(index_out)

1	2
5	4
7	8

round(bigmemory::as.matrix(distance_out), 3)

0.100	0.100
0.087	0.132
0.100	0.100

The returned object still reports the same metadata, but the actual neighbour matrices live in the destination big.matrix containers.

Reopen and Validate a Persisted Index

One of the main v3 improvements is explicit index lifecycle support. You can close a loaded handle, reopen the same index from disk, and validate its metadata before reuse.

annoy_close_index(index)
annoy_is_loaded(index)

#> [1] FALSE

reopened <- annoy_open_index(index$path, load_mode = "eager")
annoy_is_loaded(reopened)

#> [1] TRUE

Validation checks the recorded metadata against the current Annoy file and can also verify that the index loads successfully.

validation <- annoy_validate_index(reopened, strict = TRUE, load = TRUE)

validation$valid

#> [1] TRUE

validation$checks[, c("check", "passed", "severity")]

check	passed	severity
index_file	TRUE	error
metric	TRUE	error
dimensions	TRUE	error
items	TRUE	error
file_size	TRUE	error
file_md5	TRUE	error
file_mtime	TRUE	warning
load	TRUE	error

This is especially helpful when you want to reuse an index across sessions or share the .ann file and its .meta sidecar with someone else.

What Inputs Are Accepted?

For the quick start above we used:

a big.matrix reference
a dense matrix query
in-memory big.matrix destinations for streamed outputs

The package also accepts:

external pointers to big.matrix objects
big.matrix descriptor objects
descriptor file paths
query = NULL for self-search

That broader file-backed workflow is covered in the dedicated vignette on bigmemory persistence and descriptors.

Recap

You have now seen the full first-run workflow:

create a big.matrix reference
build a persisted Annoy index
search it in self-search and external-query modes
stream results into destination big.matrix objects when needed
reopen, validate, and reuse the index

From here, the most useful next steps are:

Persistent Indexes and Lifecycle for eager/lazy loading and explicit close and reopen workflows
File-Backed bigmemory Workflows for descriptor files and on-disk matrices
Benchmarking Recall and Latency for benchmark_annoy_bigmatrix() and benchmark_annoy_recall_suite()