Valkey Large Object - Configuration and Monitoring

Note: valkey-large-object is at version 0.x. Commands, arguments, replies, and configuration can change before version 1.0.

Set configurations

The configuration names have the largeobj. prefix, for example largeobj.segment-size. Set the configurations when the module loads, with the full names. Memory sizes can have units, such as 64mb and 1gb.

In valkey.conf, put each configuration on its own line:

loadmodule /path/to/libvalkey_large_object.so
largeobj.operating-mode Dram
largeobj.segment-size 64mb
largeobj.max-object-size 32mb

On the command line:

valkey-server --loadmodule /path/to/libvalkey_large_object.so \
  --largeobj.operating-mode Dram \
  --largeobj.segment-size 64mb \
  --largeobj.max-object-size 32mb

With MODULE LOADEX:

MODULE LOADEX /path/to/libvalkey_large_object.so CONFIG largeobj.operating-mode Dram CONFIG largeobj.segment-size 64mb CONFIG largeobj.max-object-size 32mb

The configurations with No in the Mutable column can be set only when the module loads. A lower largeobj.max-object-size does not change objects that are already stored.

Disk directory

Warning: In Tiered mode, the module deletes the largeobj.disk-dir directory and all its contents at startup and at shutdown, then creates it again, empty. Set largeobj.disk-dir to a new, empty directory that only this module uses, below the mount point (for example /mnt/nvme/largeobj). Never use a directory that has other data, or a mount point.

Configuration parameters

This table shows the configurations that you are most likely to set. For the full list, see the module repository.

NameTypeDefaultRangeMutableDescription
largeobj.operating-modeenumDramDram, TieredNoDram: objects are in DRAM only. Tiered: objects are on NVMe and DRAM is a read cache.
largeobj.disk-dirstring""-NoDirectory for object files. Necessary in Tiered mode. Must be on XFS, the only supported file system, with O_DIRECT and user extended attributes, and a writable parent. Deleted at startup and shutdown: see Disk directory.
largeobj.disk-dir-shardsinteger10241 to 65536NoTiered: number of subdirectories of largeobj.disk-dir for the object files. More subdirectories let file creates and deletes run in parallel.
largeobj.segment-sizememory1gb1mb to 1gbNoSize of each DRAM pool segment and disk staging segment. Must be at least two times largeobj.chunk-size. The pool grows in segments; in Tiered mode it also shrinks. Each pool has at most 16384 segments: the DRAM pool stops at 16384 x largeobj.segment-size, and largeobj.disk-staging-size / largeobj.segment-size must not be more than 16384.
largeobj.disk-staging-sizememory1gb1mb to 2^63-1NoTiered: total size of the DRAM staging buffers for disk I/O.
largeobj.disk-maxmemorymemory00 to 2^63-1YesMaximum disk use in largeobj.disk-dir. Each file counts as its length rounded up to 4KB, or, on a striped XFS, to the stripe unit for files of at least one stripe unit. On XFS, each file also counts its inode. A file also counts the metadata blocks that it uses outside its inode (see disk_metadata_spills). 0 is unlimited.
largeobj.max-object-sizememory512mb1 to 2^63-1YesMaximum object size for BLOB.TCP_SET and BLOB.RDMA_SET. BLOB.TCP_SET is also limited by the core proto-max-bulk-len (default 512MB).
largeobj.max-promote-sizememory256mb0 to 2^63-1YesTiered: maximum size of an object that is promoted from disk to DRAM. 0 disables promotion.
largeobj.chunk-sizememory1mb4096 to 256mbNoChunk size for streaming I/O. Must be a multiple of 4096.
largeobj.max-buffers-per-opinteger642 to 1024YesMaximum number of largeobj.chunk-size buffers that one streaming operation holds. In Tiered mode, a lower value lets more disk operations share the disk staging buffers.
largeobj.min-buffers-per-opinteger21 to 1024YesTiered: minimum number of largeobj.chunk-size buffers that a disk operation must get to start. If fewer buffers are free, the operation fails with ERR disk staging buffer pool exhausted; retry later. Must not be more than largeobj.max-buffers-per-op.
largeobj.worker-threadsinteger81 to 32NoNumber of worker threads for asynchronous I/O.
largeobj.scaling-shrink-watermarkinteger (%)9050 to 95YesServer used_memory / maxmemory, in percent, above which the DRAM pool does not expand. In Tiered mode, the pool also releases segments above this value (the objects stay on disk).
largeobj.promote-min-hitsinteger11 to 255YesTiered: number of disk reads of an object; the read that reaches this number promotes the object to DRAM. 1 promotes on the first read.
largeobj.fabric-providerenumEfaDirectEfaDirect, EmulatedNolibfabric provider for RDMA. EfaDirect: EFA hardware (see Providers). Emulated: libfabric over TCP sockets, for development and tests only.
largeobj.fabric-interfacesstring""-NoComma-separated list of fabric domains to use. Empty uses all domains that the provider finds.
largeobj.fabric-crc-pool-threadsinteger21 to 1024NoNumber of threads that calculate checksums for BLOB.RDMA_SET transfers.
largeobj.max-rdma-registered-bytesmemory380gb0 to 2^63-1YesMaximum bytes of DRAM pool and disk staging segments that are registered with the fabric. The pool does not expand above this value. 0 is unlimited. No effect when the fabric is not available.
largeobj.enabledboolyesyes, noYesno rejects all large object commands with DISABLED largeobj module is disabled. Stored keys stay, and core commands such as DEL still work on them.

Cross-configuration rules

The module checks these rules when it loads. It checks them again when CONFIG SET changes largeobj.disk-maxmemory, largeobj.max-object-size, largeobj.max-promote-size, largeobj.max-buffers-per-op, or largeobj.min-buffers-per-op. If a rule is not satisfied, the module does not load, or CONFIG SET fails with the error in the table.

RuleModeError
largeobj.disk-maxmemory is 0 or equal to or larger than largeobj.max-object-sizeTieredERR largeobj.disk-maxmemory must be >= largeobj.max-object-size in Tiered mode
largeobj.disk-staging-size is equal to or larger than largeobj.segment-sizeTieredERR largeobj.disk-staging-size must be >= largeobj.segment-size
largeobj.segment-size is equal to or larger than two times largeobj.chunk-sizeAllERR largeobj.segment-size must be >= 2 * largeobj.chunk-size
largeobj.chunk-size is a multiple of 4096AllERR largeobj.chunk-size must be a multiple of 4096
largeobj.max-buffers-per-op is equal to or larger than largeobj.min-buffers-per-opAllERR largeobj.max-buffers-per-op must be >= largeobj.min-buffers-per-op
An object of largeobj.max-object-size fits in one segmentDramERR largeobj.segment-size is insufficient for largeobj.max-object-size
An object of largeobj.max-promote-size fits in one segmentTieredERR largeobj.segment-size is insufficient for largeobj.max-promote-size

An object "fits in one segment" when its chunks, plus 4KB of overhead for each chunk, plus 4KB, are not larger than largeobj.segment-size. Thus largeobj.max-object-size must be a little smaller than largeobj.segment-size in Dram mode. For example, with 64MB segments and the default largeobj.chunk-size, the largest largeobj.max-object-size is 66842624 bytes (largeobj.segment-size minus 260KB). In Tiered mode, largeobj.max-promote-size (default 256MB) must fit in one segment, with the same overhead as largeobj.max-object-size (with 64MB segments, at most 66842624). The default values (largeobj.segment-size 1gb, largeobj.max-object-size 512mb, largeobj.chunk-size 1mb) satisfy all rules.

Memory and eviction

The DRAM limit is the core Valkey maxmemory setting, together with largeobj.scaling-shrink-watermark. When the fabric is available, largeobj.max-rdma-registered-bytes also limits the DRAM pool. The maximum number of segments also limits the DRAM pool (see largeobj.segment-size in Configuration parameters).

A write can evict other large objects to make space. The core maxmemory-policy setting controls which objects the module can evict:

  • allkeys-* policies - any large object.
  • volatile-* policies - only large object keys that have a TTL.
  • noeviction - no object. The write fails with an OOM error.

The module evicts in least recently used (LRU) order for all policies. A read and a TTL change count as a use. For the *-lfu, *-random, and volatile-ttl policies, the module also uses LRU order, and the server log shows a warning.

  • Dram mode - Eviction occurs only when maxmemory is set.
  • Tiered mode - Eviction removes object files when largeobj.disk-maxmemory is larger than 0. maxmemory is not necessary for disk eviction. When the DRAM cache is full, the module drops cached copies with the lowest LFU score (demotion). The objects stay on disk.

An evicted key stays in the keyspace until the module deletes it, within a few milliseconds when the backlog is small. Until then, a read returns nil, but EXISTS and DBSIZE still count the key. pending_evictions shows how many evicted keys wait to be deleted.

ACL

The module adds the ACL category @largeobj. All large object commands are in this category, and no other commands are.

Command@largeobj@read@write@connection@fast
BLOB.TCP_SETyyy
BLOB.TCP_GETyyy
BLOB.RDMA_HELLOyyy
BLOB.RDMA_SETyyy
BLOB.RDMA_GETyyy
BLOB.INFOyyy

For example, this user can use all large object commands on keys that start with kv::

127.0.0.1:6379> ACL SETUSER cache-client on >password +@largeobj ~kv:*
OK

DEL and UNLINK are not in @largeobj, so a user that must delete large object keys also needs these commands, for example +@largeobj +del +unlink.

To stop a user from using the module, remove the category from the user, for example ACL SETUSER <user> -@largeobj.

Security

  • The TCP commands carry the object data on the normal Valkey connection. If you use Valkey TLS, TLS also protects this data.
  • The RDMA commands carry only the command, the key, the client memory regions, and the reply on the Valkey connection. The object data moves directly between client memory and server memory over the fabric, not through the Valkey connection. Valkey TLS does not protect this data, and the module does not encrypt it.
  • In Tiered mode, the module does not encrypt the object files in largeobj.disk-dir. If you need encryption at rest, use file system or disk encryption.
  • For an RDMA transfer, the server reads or writes only the client memory regions that the command names, at the client endpoint of the session of that connection.
  • The server registers its own memory for local access only. A client cannot read or write server memory directly; the server starts every transfer.

What to watch

The field names in this table are without the largeobj_ prefix (see INFO fields).

FieldWhat it tells youAction
evictionsLarge object keys evicted to make space.If it grows fast, add memory or disk, or store fewer objects.
pending_evictionsEvicted keys that the module did not delete yet.If it stays high, the module cannot delete keys as fast as it evicts them.
dram_pool_exhausted, disk_capacity_exceededWrites that the module failed with OOM. A write that core Valkey rejects with OOM before the module runs is not counted.Raise maxmemory or largeobj.disk-maxmemory, or use a maxmemory-policy that permits eviction.
disk_staging_buffer_exhaustedTiered disk operations that failed because of too much concurrency.Raise largeobj.disk-staging-size, lower largeobj.max-buffers-per-op, or send fewer concurrent operations.
disk_write_errors, disk_read_errorsFailed disk I/O.Look at the server log and the disk health.
rdma_read_errors, rdma_write_errors, crc_mismatchFailed RDMA transfers.Look at the client: registered regions, endpoint, and checksum.

The core evicted_keys field in INFO stats does not count large objects that the module evicts. The module deletes the evicted keys itself (with an unlink), not through core eviction. Use evictions in INFO largeobj instead.

INFO fields

Use INFO largeobj to see the module metrics. When largeobj.enabled is no and no large objects are stored, INFO largeobj returns no sections. Valkey adds the largeobj_ prefix to each section name and field name, for example section largeobj_core_metrics and field largeobj_num_objects. The tables below show the names without the prefix. They show only the most important fields. For the full list, see the module repository.

core_metrics

FieldDescription
num_objectsNumber of large objects.
inflight_requestsNumber of large object commands that are in progress.

evictions

FieldDescription
evictionsNumber of objects evicted to make space for a write. Tiered cache drops are demotions and are not counted.
pending_evictionsNumber of evicted keys that the module did not finish deleting. These keys read as missing.

dram

FieldDescription
dram_segmentsNumber of DRAM pool segments.
dram_allocated_bytesBytes allocated in the DRAM pool.
dram_capacity_bytesPool capacity: (segments minus draining segments) multiplied by the segment size.
dram_utilization_pctDRAM pool use in percent.
dram_objectsNumber of objects in DRAM.
cache_hitsTiered only: reads served from the DRAM cache.
cache_missesTiered only: reads served from disk.
cache_promotionsTiered only: objects promoted to the DRAM cache.
cache_demotionsTiered only: cached copies dropped from DRAM.
dram_rdma_registered_segmentsDRAM pool segments registered with the fabric. 0 when the fabric is not available.

disk

This section is only in Tiered mode.

FieldDescription
disk_used_bytesBytes used in largeobj.disk-dir, counted as for largeobj.disk-maxmemory.
disk_utilization_pctdisk_used_bytes as a percent of largeobj.disk-maxmemory. 0.00 when largeobj.disk-maxmemory is 0.
disk_metadata_spillsNumber of object files that use disk blocks outside their inode, in addition to their data. For example, a file that has more extents than its inode can list uses one more block. These blocks count in disk_used_bytes.

rdma

FieldDescription
rdma_sessionsNumber of open RDMA sessions.
rdma_read_bytesBytes that the server read from client memory (BLOB.RDMA_SET).
rdma_write_bytesBytes that the server wrote into client memory (BLOB.RDMA_GET).

error_metrics

FieldDescription
disk_read_errorsFailed disk reads.
disk_write_errorsFailed disk writes.
rdma_read_errorsFailed RDMA reads from client memory.
rdma_write_errorsFailed RDMA writes into client memory.
dram_pool_exhaustedWrites that could not get DRAM pool memory.
disk_staging_buffer_exhaustedTiered disk reads and writes that failed with ERR disk staging buffer pool exhausted; retry later, because too many disk operations ran at the same time for the disk staging buffers.
disk_capacity_exceededWrites that failed because of largeobj.disk-maxmemory.
crc_mismatchBLOB.RDMA_SET writes rejected because the CRC value did not agree.