dets
A Disk Based Term Storage
The module dets
provides a term storage on file. The
stored terms, in this module called objects, are tuples
such that one element is defined to be the key. A Dets
table is a collection of objects with the key at the same
position stored on a file.
Dets is used by the Mnesia application, and is provided as is for users who are interested in an efficient storage of Erlang terms on disk only. Many applications just need to store some terms in a file. Mnesia adds transactions, queries, and distribution. The size of Dets files cannot exceed 2 GB. If larger tables are needed, Mnesia's table fragmentation can be used.
There are three types of Dets tables: set, bag and duplicate_bag. A table of type set has at most one object with a given key. If an object with a key already present in the table is inserted, the existing object is overwritten by the new object. A table of type bag has zero or more different objects with a given key. A table of type duplicate_bag has zero or more possibly matching objects with a given key.
Dets tables must be opened before they can be updated or read, and when finished they must be properly closed. If a table has not been properly closed, Dets will automatically repair the table. This can take a substantial time if the table is large. A Dets table is closed when the process which opened the table terminates. If several Erlang processes (users) open the same Dets table, they will share the table. The table is properly closed when all users have either terminated or closed the table. Dets tables are not properly closed if the Erlang runtime system is terminated abnormally.
Note!
A ^C command abnormally terminates an Erlang runtime system in a Unix environment with a break-handler.
Since all operations performed by Dets are disk operations, it is important to realize that a single look-up operation involves a series of disk seek and read operations. For this reason, the Dets functions are much slower than the corresponding Ets functions, although Dets exports a similar interface.
Dets organizes data as a linear hash list and the hash list
grows gracefully as more data is inserted into the table. Space
management on the file is performed by what is called a buddy
system. The current implementation keeps the entire buddy system
in RAM, which implies that if the table gets heavily fragmented,
quite some memory can be used up. The only way to defragment a
table is to close it and then open it again with the repair
option set to force
.
It is worth noting that the ordered_set type present in Ets is
not yet implemented by Dets, neither is the limited support for
concurrent updates which makes a sequence of first
and
next
calls safe to use on fixed Ets tables. Both these
features will be implemented by Dets in a future release of
Erlang/OTP. Until then, the Mnesia application (or some user
implemented method for locking) has to be used to implement safe
concurrency. Currently, no library of Erlang/OTP has support for
ordered disk based term storage.
Two versions of the format used for storing objects on file are supported by Dets. The first version, 8, is the format always used for tables created by OTP R7 and earlier. The second version, 9, is the default version of tables created by OTP R8 (and later OTP releases). OTP R8 can create version 8 tables, and convert version 8 tables to version 9, and vice versa, upon request.
All Dets functions return {error, Reason}
if an error
occurs (first/1
and next/2
are exceptions, they exit
the process with the error tuple). If given badly formed
arguments, all functions exit the process with a badarg
message.
Opaque continuation used by
bchunk/2
.
Match specifications, see the match specification documentation in the ERTS User's Guide and ms_transform(3).
Opaque continuation used by
match_object/1
and
match_object/3
.
See ets:match/2 for a description of patterns.
Functions
all/0
Returns a list of the names of all open tables on this node.
bchunk/2
Returns a list of objects stored in a table. The exact
representation of the returned objects is not public. The
lists of data can be used for initializing a table by giving
the value bchunk
to the format
option of the
init_table/3
function. The Mnesia application uses this
function for copying open tables.
Unless the table is protected using safe_fixtable/2
,
calls to bchunk/2
may not work as expected if
concurrent updates are made to the table.
The first time bchunk/2
is called, an initial
continuation, the atom start
, must be provided.
The bchunk/2
function returns a tuple
{
,
where
is a list of
objects.
is another continuation
which is
to be passed on to a subsequent call to bchunk/2
. With
a series of calls to bchunk/2
it is possible to extract
all objects of the table.
bchunk/2
returns '$end_of_table'
when all
objects have been returned, or {error,
if an error occurs.
close/1
Closes a table. Only processes that have opened a table are allowed to close it.
All open tables must be closed before the system is stopped. If an attempt is made to open a table which has not been properly closed, Dets automatically tries to repair the table.
delete/2
Deletes all objects with the key
from
the table
.
delete_all_objects/1
Deletes all objects from a table in almost constant time.
However, if the table if fixed, delete_all_objects(T)
is equivalent to match_delete(T, '_')
.
delete_object/2
Deletes all instances of a given object from a table. If a
table is of type bag
or duplicate_bag
, the
delete/2
function cannot be used to delete only some of
the objects with a given key. This function makes this
possible.
first/1
Returns the first key stored in the table
according to the table's internal order, or
'$end_of_table'
if the table is empty.
Unless the table is protected using safe_fixtable/2
,
subsequent calls to next/2
may not work as expected if
concurrent updates are made to the table.
Should an error occur, the process is exited with an error
tuple {error, Reason}
. The reason for not returning the
error tuple is that it cannot be distinguished from a key.
There are two reasons why first/1
and next/2
should not be used: they are not very efficient, and they
prevent the use of the key '$end_of_table'
since this
atom is used to indicate the end of the table. If possible,
the match
, match_object
, and select
functions should be used for traversing tables.
foldl/3
foldr/3
Calls
on successive elements of
the table
together with an extra argument
AccIn
. The
order in which the elements of the table are traversed is
unspecified.
must return a new
accumulator which is passed to the next call.
is returned if
the table is empty.
from_ets/2
Deletes all objects of the table
and then
inserts all the objects of the Ets table
.
The order in which the objects are inserted is not specified.
Since ets:safe_fixtable/2
is called the Ets table must
be public or owned by the calling process.
info/1
Returns information about the table
as a list of tuples:
-
{file_size, integer() >= 0}
, the size of the file in bytes. -
{filename, file:name()}
, the name of the file where objects are stored. -
{keypos, keypos()}
, the position of the key. -
{size, integer() >= 0}
, the number of objects stored in the table. -
{type, type()}
, the type of the table.
info/2
Returns the information associated with
for the table
.
In addition to the {
pairs defined for info/1
, the following items are
allowed:
-
{access, access()}
, the access mode. -
{auto_save, auto_save()}
, the auto save interval. -
{bchunk_format, binary()}
, an opaque binary describing the format of the objects returned bybchunk/2
. The binary can be used as argument tois_compatible_chunk_format/2
. Only available for version 9 tables. -
{hash,
Hash}
. Describes which BIF is used to calculate the hash values of the objects stored in the Dets table. Possible values of Hash arehash
, which implies that theerlang:hash/2
BIF is used,phash
, which implies that theerlang:phash/2
BIF is used, andphash2
, which implies that theerlang:phash2/1
BIF is used. -
{memory, integer() >= 0}
, the size of the file in bytes. The same value is associated with the itemfile_size
. -
{no_keys, integer >= 0()}
, the number of different keys stored in the table. Only available for version 9 tables. -
{no_objects, integer >= 0()}
, the number of objects stored in the table. -
{no_slots, {
Min,
Used,
Max}}
, the number of slots of the table.Min
is the minimum number of slots,Used
is the number of currently used slots, andMax
is the maximum number of slots. Only available for version 9 tables. -
{owner, pid()}
, the pid of the process that handles requests to the Dets table. -
{ram_file, boolean()}
, whether the table is kept in RAM. -
{safe_fixed,
SafeFixed}
. If the table is fixed, SafeFixed is a tuple{FixedAtTime, [{Pid,RefCount}]}
.FixedAtTime
is the time when the table was first fixed, andPid
is the pid of the process that fixes the tableRefCount
times. There may be any number of processes in the list. If the table is not fixed, SafeFixed is the atomfalse
. -
{version, integer()
, the version of the format of the table.
init_table/2
init_table/3
Replaces the existing objects of the table
with objects created by calling the input function
,
see below. The reason for using this function rather than
calling insert/2
is that of efficiency. It should be
noted that the input functions are called by the process that
handles requests to the Dets table, not by the calling
process.
When called with the argument read
the function
is assumed to return
end_of_input
when
there is no more input, or {Objects, Fun}
, where
Objects
is a list of objects and Fun
is a new
input function. Any other value Value is returned as an error
{error, {init_fun, Value}}
. Each input function will be
called exactly once, and should an error occur, the last
function is called with the argument close
, the reply
of which is ignored.
If the type of the table is set
and there is more
than one object with a given key, one of the objects is
chosen. This is not necessarily the last object with the given
key in the sequence of objects returned by the input
functions. Duplicate keys should be avoided, or the file
will be unnecessarily fragmented. This holds also for duplicated
objects stored in tables of type bag
.
It is important that the table has a sufficient number of
slots for the objects. If not, the hash list will start to
grow when init_table/2
returns which will significantly
slow down access to the table for a period of time. The
minimum number of slots is set by the open_file/2
option min_no_slots
and returned by the info/2
item no_slots
. See also the min_no_slots
option
below.
The
argument is a list of
{Key, Val}
tuples where the following values are allowed:
-
{min_no_slots, no_slots()}
. Specifies the estimated number of different keys that will be stored in the table. Theopen_file
option with the same name is ignored unless the table is created, and in that case performance can be enhanced by supplying an estimate when initializing the table. -
{format, Format}
. Specifies the format of the objects returned by the function
. IfInitFun Format
isterm
(the default),
is assumed to return a list of tuples. IfInitFun Format
isbchunk
,
is assumed to returnInitFun
as returned byData bchunk/2
. This option overrides themin_no_slots
option.
insert/2
Inserts one or more objects into the table
.
If there already exists an object with a key matching the key of
some of the given objects and the table type is set
,
the old object will be replaced.
insert_new/2
Inserts one or more objects into the table
.
If there already exists some object with a key matching the key
of any of the given objects the table is not updated and
false
is returned, otherwise the objects are inserted
and true
returned.
is_compatible_bchunk_format/2
Returns true
if it would be possible to initialize
the table
, using
init_table/3
with the
option {format, bchunk}
, with objects read with
bchunk/2
from some
table T
such that calling
info(T, bchunk_format)
returns
BchunkFormat
.
is_dets_file/1
Returns true
if the file
is a Dets table, false
otherwise.
lookup/2
Returns a list of all objects with the key
stored in the table
. For example:
2>dets:open_file(abc, [{type, bag}]).
{ok,abc} 3>dets:insert(abc, {1,2,3}).
ok 4>dets:insert(abc, {1,3,4}).
ok 5>dets:lookup(abc, 1).
[{1,2,3},{1,3,4}]
If the table is of type set
, the function returns
either the empty list or a list with one object, as there
cannot be more than one object with a given key. If the table
is of type bag
or duplicate_bag
, the function
returns a list of arbitrary length.
Note that the order of objects returned is unspecified. In particular, the order in which objects were inserted is not reflected.
match/1
Matches some objects stored in a table and returns a
non-empty list of the bindings that match a given pattern in
some unspecified order. The table, the pattern, and the number
of objects that are matched are all defined by
, which has been returned by a prior
call to match/1
or match/3
.
When all objects of the table have been matched,
'$end_of_table'
is returned.
match/2
Returns for each object of the table
that
matches
a list of bindings in some unspecified
order. See ets:match/2 for a
description of patterns. If the keypos'th element of
is unbound, all objects of the table are
matched. If the keypos'th element is bound, only the
objects with the right key are matched.
match/3
Matches some or all objects of the table
and
returns a non-empty list of the bindings that match
in some unspecified order.
See ets:match/2 for a
description of patterns.
A tuple of the bindings and a continuation is returned,
unless the table is empty, in which case
'$end_of_table'
is returned. The continuation is to be
used when matching further objects by calling
match/1
.
If the keypos'th element of
is bound,
all objects of the table are matched. If the keypos'th element is
unbound, all objects of the table are matched,
objects at a time, until at least one object matches or the
end of the table has been reached. The default, indicated by
giving
the value default
,
is to let the number
of objects vary depending on the sizes of the objects. If
is a version 9 table, all objects with the same
key are always matched at the same time which implies that
more than
The table should always be protected using
safe_fixtable/2
before calling match/3
, or
errors may occur when calling match/1
.
match_delete/2
Deletes all objects that match
from the
table
.
See ets:match/2 for a
description of patterns.
If the keypos'th element of Pattern
is bound,
only the objects with the right key are matched.
match_object/1
Returns a non-empty list of some objects stored in a table
that match a given pattern in some unspecified order. The
table, the pattern, and the number of objects that are matched
are all defined by
, which has been
returned by a prior call to match_object/1
or
match_object/3
.
When all objects of the table have been matched,
'$end_of_table'
is returned.
match_object/2
Returns a list of all objects of the table
that
match
in some unspecified order.
See ets:match/2 for a
description of patterns.
If the keypos'th element of
is
unbound, all objects of the table are matched. If the
keypos'th element of
is bound, only the
objects with the right key are matched.
Using the match_object
functions for traversing all
objects of a table is more efficient than calling
first/1
and next/2
or slot/2
.
match_object/3
Matches some or all objects stored in the table
and returns a non-empty list of the objects that match
in some unspecified order.
See ets:match/2 for a
description of patterns.
A list of objects and a continuation is returned, unless
the table is empty, in which case '$end_of_table'
is returned. The continuation is to be used when matching
further objects by calling match_object/1
.
If the keypos'th element of
is bound, all
objects of the table are matched. If the keypos'th element is
unbound, all objects of the table are matched,
objects at a time, until at least one object matches or the
end of the table has been reached. The default, indicated by
giving
the value default
, is to let the number
of objects vary depending on the sizes of the objects. If
is a version 9 table, all matching objects with
the same key are always returned in the same reply which
implies that more than
The table should always be protected using
safe_fixtable/2
before calling match_object/3
,
or errors may occur when calling match_object/1
.
member/2
Works like lookup/2
, but does not return the
objects. The function returns true
if one or more
elements of the table has the key
, false
otherwise.
next/2
Returns the key following
in the table
according to the table's internal order, or
'$end_of_table'
if there is no next key.
Should an error occur, the process is exited with an error
tuple {error, Reason}
.
Use first/1
to find
the first key in the table.
open_file/1
Opens an existing table. If the table has not been properly closed, it will be repaired. The returned reference is to be used as the name of the table. This function is most useful for debugging purposes.
open_file/2
Opens a table. An empty Dets table is created if no file exists.
The atom
is the name of the table. The table
name must be provided in all subsequent operations on the
table. The name can be used by other processes as well, and
several process can share one table.
If two processes open the same table by giving the same name and arguments, then the table will have two users. If one user closes the table, it still remains open until the second user closes the table.
The
argument is a list of {Key, Val}
tuples where the following values are allowed:
-
{access, access()}
. It is possible to open existing tables in read-only mode. A table which is opened in read-only mode is not subjected to the automatic file reparation algorithm if it is later opened after a crash. The default value isread_write
. -
{auto_save, auto_save()}
, the auto save interval. If the interval is an integerTime
, the table is flushed to disk whenever it is not accessed forTime
milliseconds. A table that has been flushed will require no reparation when reopened after an uncontrolled emulator halt. If the interval is the atominfinity
, auto save is disabled. The default value is 180000 (3 minutes). -
{estimated_no_objects, no_slots()}
. Equivalent to themin_no_slots
option. -
{file, file:name()}
, the name of the file to be opened. The default value is the name of the table. -
{max_no_slots, no_slots()}
, the maximum number of slots that will be used. The default value as well as the maximal value is 32 M. Note that a higher value may increase the fragmentation of the table, and conversely, that a smaller value may decrease the fragmentation, at the expense of execution time. Only available for version 9 tables. -
{min_no_slots, no_slots()}
. Application performance can be enhanced with this flag by specifying, when the table is created, the estimated number of different keys that will be stored in the table. The default value as well as the minimum value is 256. -
{keypos, keypos()}
, the position of the element of each object to be used as key. The default value is 1. The ability to explicitly state the key position is most convenient when we want to store Erlang records in which the first position of the record is the name of the record type. -
{ram_file, boolean()}
, whether the table is to be kept in RAM. Keeping the table in RAM may sound like an anomaly, but can enhance the performance of applications which open a table, insert a set of objects, and then close the table. When the table is closed, its contents are written to the disk file. The default value isfalse
. -
{repair, Value}
.Value
can be either aboolean()
or the atomforce
. The flag specifies whether the Dets server should invoke the automatic file reparation algorithm. The default istrue
. Iffalse
is specified, there is no attempt to repair the file and{error, {needs_repair, FileName}}
is returned if the table needs to be repaired.The value
force
means that a reparation will take place even if the table has been properly closed. This is how to convert tables created by older versions of STDLIB. An example is tables hashed with the deprecatederlang:hash/2
BIF. Tables created with Dets from a STDLIB version of 1.8.2 and later use theerlang:phash/2
function or theerlang:phash2/1
function, which is preferred.The
repair
option is ignored if the table is already open. -
{type, type()}
, the type of the table. The default value isset
. -
{version, version()}
, the version of the format used for the table. The default value is9
. Tables on the format used before OTP R8 can be created by giving the value8
. A version 8 table can be converted to a version 9 table by giving the options{version,9}
and{repair,force}
.
pid2name/1
Returns the name of the table given the pid of a process
that handles requests to a table, or undefined
if
there is no such table.
This function is meant to be used for debugging only.
repair_continuation/2
This function can be used to restore an opaque continuation
returned by select/3
or select/1
if the
continuation has passed through external term format (been
sent between nodes or stored on disk).
The reason for this function is that continuation terms
contain compiled match specifications and therefore will be
invalidated if converted to external term format. Given that
the original match specification is kept intact, the
continuation can be restored, meaning it can once again be
used in subsequent select/1
calls even though it has
been stored on disk or on another node.
See also ets(3)
for further explanations and
examples.
Note!
This function is very rarely needed in application code. It
is used by Mnesia to implement distributed select/3
and select/1
sequences. A normal application would
either use Mnesia or keep the continuation from being
converted to external format.
The reason for not having an external representation of compiled match specifications is performance. It may be subject to change in future releases, while this interface will remain for backward compatibility.
safe_fixtable/2
If
is true
, the table
is
fixed (once more) by the calling process, otherwise the table
is released. The table is also released when a fixing process
terminates.
If several processes fix a table, the table will remain fixed until all processes have released it or terminated. A reference counter is kept on a per process basis, and N consecutive fixes require N releases to release the table.
It is not guaranteed that calls to first/1
,
next/2
, select and match functions work as expected
even if the table has been fixed; the limited support for
concurrency implemented in Ets has not yet been implemented
in Dets. Fixing a table currently only disables resizing of
the hash list of the table.
If objects have been added while the table was fixed, the hash list will start to grow when the table is released which will significantly slow down access to the table for a period of time.
select/1
Applies a match specification to some objects stored in a
table and returns a non-empty list of the results. The
table, the match specification, and the number of objects
that are matched are all defined by
,
which has been returned by a prior call to select/1
or select/3
.
When all objects of the table have been matched,
'$end_of_table'
is returned.
select/2
Returns the results of applying the match specification
to all or some objects stored in the table
. The order of the objects is not specified. See
the ERTS User's Guide for a description of match
specifications.
If the keypos'th element of
is
unbound, the match specification is applied to all objects of
the table. If the keypos'th element is bound, the match
specification is applied to the objects with the right key(s)
only.
Using the select
functions for traversing all
objects of a table is more efficient than calling
first/1
and next/2
or slot/2
.
select/3
Returns the results of applying the match specification
to some or all objects stored in the table
. The order of the objects is not specified. See
the ERTS User's Guide for a description of match
specifications.
A tuple of the results of applying the match specification
and a continuation is returned, unless the table is empty,
in which case '$end_of_table'
is returned. The
continuation is to be used when matching further objects by
calling select/1
.
If the keypos'th element of
is bound, the
match specification is applied to all objects of the table
with the right key(s). If the keypos'th element of
is unbound, the match specification is
applied to all objects of the table,
objects at a
time, until at least one object matches or the end of the
table has been reached. The default, indicated by giving
the value default
, is to let the number of
objects vary depending on the sizes of the objects. If
is a version 9 table, all objects with the same
key are always handled at the same time which implies that the
match specification may be applied to more than
The table should always be protected using
safe_fixtable/2
before calling select/3
, or
errors may occur when calling select/1
.
select_delete/2
Deletes each object from the table
such that
applying the match specification
to the
object returns the value true
. See the ERTS
User's Guide for a description of match
specifications. Returns the number of deleted objects.
If the keypos'th element of
is
bound, the match specification is applied to the objects
with the right key(s) only.
slot/2
The objects of a table are distributed among slots,
starting with slot 0
and ending with slot n. This
function returns the list of objects associated with slot
. If
is greater than n
'$end_of_table'
is returned.
sync/1
Ensures that all updates made to the table
are
written to disk. This also applies to tables which have been
opened with the ram_file
flag set to true
. In
this case, the contents of the RAM file are flushed to
disk.
Note that the space management data structures kept in RAM, the buddy system, is also written to the disk. This may take some time if the table is fragmented.
table/1
table/2
Returns a QLC (Query List
Comprehension) query handle. The module qlc
implements a query language aimed mainly at Mnesia but Ets
tables, Dets tables, and lists are also recognized by qlc
as sources of data. Calling dets:table/1,2
is the
means to make the Dets table
usable to qlc
.
When there are only simple restrictions on the key position
qlc
uses dets:lookup/2
to look up the keys, but when
that is not possible the whole table is traversed. The
option traverse
determines how this is done:
-
first_next
. The table is traversed one key at a time by callingdets:first/1
anddets:next/2
. -
select
. The table is traversed by callingdets:select/3
anddets:select/1
. The optionn_objects
determines the number of objects returned (the third argument ofselect/3
). The match specification (the second argument ofselect/3
) is assembled byqlc
: simple filters are translated into equivalent match specifications while more complicated filters have to be applied to all objects returned byselect/3
given a match specification that matches all objects. -
{select, match_spec()}
. As forselect
the table is traversed by callingdets:select/3
anddets:select/1
. The difference is that the match specification is explicitly given. This is how to state match specifications that cannot easily be expressed within the syntax provided byqlc
.
The following example uses an explicit match specification to traverse the table:
1>dets:open_file(t, []),
ok = dets:insert(t, [{1,a},{2,b},{3,c},{4,d}]),
MS = ets:fun2ms(fun({X,Y}) when (X > 1) or (X < 5) -> {Y} end),
QH1 = dets:table(t, [{traverse, {select, MS}}]).
An example with implicit match specification:
2> QH2 = qlc:q([{Y} || {X,Y} <- dets:table(t), (X > 1) or (X < 5)]).
The latter example is in fact equivalent to the former which
can be verified using the function qlc:info/1
:
3> qlc:info(QH1) =:= qlc:info(QH2).
true
qlc:info/1
returns information about a query handle,
and in this case identical information is returned for the
two query handles.
to_ets/2
Inserts the objects of the Dets table
into the
Ets table
. The order in which the objects are
inserted is not specified. The existing objects of the Ets
table are kept unless overwritten.
traverse/2
Applies
to each object stored in the table
in some unspecified order. Different actions are
taken depending on the return value of
. The
following
return values are allowed:
continue
Continue to perform the traversal. For example, the following function can be used to print out the contents of a table:
fun(X) -> io:format("~p~n", [X]), continue end.
{continue, Val}
Continue the traversal and accumulate
. The
following function is supplied in order to collect all
objects of a table in a list:
fun(X) -> {continue, X} end.
{done, Value }
Terminate the traversal and return [
.
Any other value
returned by
terminates the
traversal and is immediately returned.
update_counter/3
Updates the object with key
stored in the
table
of type set
by adding
to the
element at the
:th position.
The new counter value
is returned. If no position is specified, the element directly
following the key is updated.
This functions provides a way of updating a counter, without having to look up an object, update the object by incrementing an element and insert the resulting object into the table again.
- all
- bchunk
- close
- delete
- delete_all_objects
- delete_object
- first
- foldl
- foldr
- from_ets
- info
- info-1
- init_table
- init_table-1
- insert
- insert_new
- is_compatible_bchunk_format
- is_dets_file
- lookup
- match
- match-1
- match-2
- match_delete
- match_object
- match_object-1
- match_object-2
- member
- next
- open_file
- open_file-1
- pid2name
- repair_continuation
- safe_fixtable
- select
- select-1
- select-2
- select_delete
- slot
- sync
- table
- table-1
- to_ets
- traverse
- update_counter