mmap-object
Wrap a shared memory-mapped file in a plain old object.
Shared Memory Objects
Super-fast file-based sharing of Javascript objects among multiple processes.
This module maps Javascript objects into shared memory for simultaneous access by different Node processes running on the same machine. Shared memory is loaded via mmap. Object access is mediated by Boost's unordered map class so object property access are speedy.
Data is lazily loaded piece-by-piece as needed so opening even a huge file takes no time at all.
There are two modes:
Unshared Write-only Mode
A single process creates a new file which is mapped to a Javascript object. Setting properties on this object writes those properties to the file. You can read from the object within this mode but sharing an object in write-only mode with other processes is certain to result in crashes.
Shared Read-only mode
Open an existing file for reading. Multiple processes can safely open this file. Opening is lightning fast and only a single copy remains in memory.
Faster performance with buffers
If you use lengthy data values, buffers can speed things up considerably. In rough benchmarking, a 300% speedup was see when reading 20k-byte values as buffers instead of strings. For 200k-byte values, the speedup was 2000%.
Requirements
Binaries are provided for OSX and Linux for various node versions (check the releases page to see which). If a binary is not provided for your platform, you will need Boost and and a C++11 compliant compiler (like GCC 4.8 or better) to build the module.
Installation
npm install mmap-object
Usage
const Shared = const shared_object = 'filename' shared_object'new_key' = 'a string value'shared_objectnew_property = Buffershared_object'useless key' = 0 // Erase a keydelete shared_object'useless_key'shared_object // Read a fileconst read_only_shared_object = 'filename'consoleconsole read_only_shared_object
API
new Create(path, [file_size], [initial_bucket_count], [max_file_size])
Creates a new file mapped into shared memory. Returns an object that provides access to the shared memory. Throws an exception on error.
Arguments
path
- The path of the file to createfile_size
- Optional The initial size of the file in kilobytes. If more space is needed, the file will automatically be grown to a larger size. Minimum is 500 bytes. Defaults to 5 megabytes.initial_bucket_count
- Optional The number of buckets to allocate initially. This is passed to the underlying Boost unordered_map. Defaults to 1024. Set this to the number of keys you expect to write.max_file_size
- Optional The largest the file is allowed to grow in kilobites. If data is added beyond this limit, an exception is thrown. Defaults to 5 gigabytes.
Example
// Create a 500K map for 300 objects.const obj = '/tmp/sharedmem' 500 300
new Open(path)
Maps an existing file into shared memory. Returns an object that
provides read-only access to the object contained in the file. Throws
an exception on error. Any number of processes can open the same file
but only a single copy will reside in memory. Uses mmap
under the
covers, so only those parts of the file that are actually accessed
will be loaded.
Arguments
path
- The path of the file to open
Example
// Open up that shared fileconst obj = '/tmp/sharedmem'
close()
Unmaps a previously created or opened file. If the file was most
recently opened with Create()
, close()
will first shrink the file
to remove any unneeded space that may have been allocated.
It's important to close your unused shared files in long-running processes. Not doing so keeps shared memory from being freed.
The closing of very large objects (a few gigabytes and up) may take
some time (hundreds to thousands of milliseconds). To prevent blocking
the main thread, pass a callback to close()
. The call to close()
will return immediately while the callback will be called after the
underlying munmap()
operation completes. Any error will be given as
the first argument to the callback.
Example
obj
Iteration
The iterable protocol is supported for efficient iteration over the entire contents of the object:
const Shared = const obj = 'filename' for let key value of obj console
(This ES6 syntax is supported in Node 6+, for previous versions of node a more laborious syntax is necessary.)
isData()
When iterating, use isData()
to tell if a particular key is real
data or one of the underlying methods on the shared object:
const obj = '/tmp/sharedmem' for let key in obj if obj // Only show actual data console
isOpen()
Return true if this object is currently open.
isClosed()
Return true if this object has been closed.
get_free_memory()
Number of bytes of free storage left in the shared object file.
get_size()
The size of the storage in the shared object file, in bytes.
bucket_count()
The number of buckets currently allocated in the underlying hash structure.
max_bucket_count()
The maximum number of buckets that can be allocated in the underlying hash structure.
load_factor()
The average number of elements per bucket.
max_load_factor()
The current maximum load factor.
Unit tests
npm test
Limitations
It is strongly recommended to pass in the number of keys you expect
to write when creating the object with Create
. If you don't do this,
the object will resize as you fill it up. This can be a very
time-consuming process and can result in fragmentation within the
shared memory object and a larger final file size.
Object values may be only string, buffer, or number values. Attempting to set a different type value results in an exception.
Symbols are not supported as properties.
Publishing a binary release
To make a new binary release:
- Edit package.json. Increment the
version
property. node-pre-gyp rebuild
node-pre-gyp package
node-pre-gyp-github publish
npm publish
You will need a NODE_PRE_GYP_GITHUB_TOKEN
with repo:status
,
repo_deployment
and public_repo
access to the target repo. You'll
also need write access to the npm repo.
MSVS build prerequisites
Set up Boost.
Set BOOST_ROOT environment variable.
bootstrap
b2 --build-type=complete