ttb
A base for building trace tools for distributed systems.
The Trace Tool Builder ttb
is a base for building trace
tools for distributed systems.
When using ttb
, dbg
shall not be used in parallel.
Functions
start_trace(Nodes, Patterns, FlagSpec, Opts) -> Result
Result = see p/2
Nodes = see tracer/2
Patterns = [tuple()]
FlagSpec = {Procs, Flags}
Proc = see p/2
Flags = see p/2
Opts = see tracer/2
This function is a shortcut allowing to start a trace with one command. Each
tuple in Patterns
is converted to list which is in turn passed to
ttb:tpl
.
The call:
ttb:start_trace([Node, OtherNode], [{mod, foo, []}, {mod, bar, 2}], {all, call}, [{file, File}, {handler,{fun myhandler/4, S}}])is equivalent to
ttb:start_trace([Node, OtherNode], [{file, File}, {handler,{fun myhandler/4, S}}]), ttb:tpl(mod, foo, []), ttb:tpl(mod, bar, 2, []), ttb:p(all, call)
tracer() -> Result
This is equivalent to tracer(node())
.
tracer(Shortcut) -> Result
Shortcut = shell | dbg
shell
is equivalent to tracer(node(),[{file, {local, "ttb"}}, shell])
.
dbg
is equivalent to tracer(node(),[{shell, only}])
.
tracer(Nodes) -> Result
This is equivalent to tracer(Nodes,[])
.
tracer(Nodes,Opts) -> Result
Result = {ok, ActivatedNodes} | {error,Reason}
Nodes = atom() | [atom()] | all | existing | new
Opts = Opt | [Opt]
Opt = {file,Client} | {handler, FormatHandler} | {process_info,PI} | shell | {shell, ShellSpec} | {timer, TimerSpec} | {overload, {MSec, Module, Function}} | {flush, MSec} | resume | {resume, FetchTimeout}
TimerSpec = MSec | {MSec, StopOpts}
MSec = FetchTimeout = integer()
Module = Function = atom()
StopOpts = see stop/2
Client = File | {local, File}
File = Filename | Wrap
Filename = string()
Wrap = {wrap,Filename} | {wrap,Filename,Size,Count}
FormatHandler = See format/2
PI = true | false
ShellSpec = true | false | only
This function starts a file trace port on all given nodes and also points the system tracer for sequential tracing to the same port.
The given Filename
will be prefixed with the node
name. Default Filename
is "ttb".
File={wrap,Filename,Size,Count}
can be used if
the size of the trace logs must be limited. Default values are
Size=128*1024
and Count=8
.
When tracing diskless nodes, ttb
must be started
from an external "trace control node" with disk access, and
Client
must be {local, File}
. All
trace information is then sent to the trace control node where
it is written to file.
The process_info
option indicates if process
information should be collected. If PI = true
(which is
default), each process identifier Pid
is replaced by a
tuple {Pid,ProcessInfo,Node}
, where ProcessInfo
is the process' registered name its globally registered name,
or its initial function. It is possible to turn off this
functionality by setting PI = false
.
The {shell, ShellSpec}
option indicates that the trace messages should
be printed on the console as they are received by the tracing
process. This implies {local, File}
trace client. If the ShellSpec
is only
(instead of true
), no trace logs are stored.
The shell
option is a shortcut for {shell, true}
.
The timer
option indicates that the trace should be
automatically stopped after MSec
milliseconds. StopOpts
are passed to ttb:stop/2
command if specified (default is []
).
Note that the timing is approximate, as delays related to
network communication are always present. The timer starts after
ttb:p/2
is issued, so you can set up your trace patterns before.
The overload
option allows to enable overload
checking on the nodes under trace. Module:Function(check)
is performed each MSec
milliseconds. If the check returns
true
, the tracing is disabled on a given node.
Module:Function
should be able to handle at least three
atoms: init
, check
and stop
. init
and
stop
give the user a possibility to initialize and clean
up the check environment.
When a node gets overloaded, it is not possible to issue ttb:p
nor any command from the ttb:tp
family, as it would lead to
inconsistent tracing state (different trace specifications on
different node).
The flush
option periodically flushes all file trace
port clients (see dbg:flush_trace_port/1
). When enabled,
the buffers are freed each MSec
milliseconds. This option is
not allowed with {file, {local, File}}
tracing.
{resume, FetchTimeout}
enables the autoresume feature.
Whenever enabled, remote nodes try to reconnect to the controlling node
in case they were restarted. The feature requires runtime_tools
application to be started (so it has to be present in the .boot
scripts if the traced nodes run with embedded erlang). If this is
not possible, resume may be performed manually by starting
runtime_tools
remotely using rpc:call/4
.
ttb
tries to fetch all logs from a reconnecting node before
reinitializing the trace. This has to finish within FetchTimeout milliseconds
or is aborted
By default, autostart information is stored in a file called
ttb_autostart.bin
on each node. If this is not desired
(i.e. on diskless nodes), a custom module to handle autostart
information storage and retrieval can be provided by specifying
ttb_autostart_module
environment variable for the runtime_tools
application. The module has to respond to the following API:
write_config(Data) -> ok
read_config() -> {ok, Data} | {error, Error}
write_config(Data)
.delete_config() -> ok
write_config(Data)
.
Note that after this call any subsequent calls to read_config
must return {error, Error}
.
The resume
option implies the default FetchTimeout
, which is
10 seconds
p(Procs,Flags) -> Return
Return = {ok,[{Procs,MatchDesc}]}
Procs = Process | [Process] | all | new | existing
Process = pid() | atom() | {global,atom()}
Flags = Flag | [Flag]
This function sets the given trace flags on the given
processes. The timestamp
flag is always turned on.
Please turn to the Reference manual for module dbg
for details about the possible trace flags. The parameter
MatchDesc
is the same as returned from dbg:p/2
Processes can be given as registered names, globally registered names or process identifiers. If a registered name is given, the flags are set on processes with this name on all active nodes.
Issuing this command starts the timer for this trace if
timer
option was specified with tracer/2
.
These functions should be used in combination with the
call
trace flag for setting and clearing trace
patterns. When the call
trace flag is set on a process,
function calls will be traced on that process if a trace
pattern has been set for the called function. Trace patterns
specifies how to trace a function by using match
specifications. Match specifications are described in the
User's Guide for the erlang runtime system erts
.
These functions are equivalent to the corresponding
functions in dbg
, but all calls are stored in the
history. The history buffer makes it easy to create config
files so that the same trace environment can be setup several
times, e.g. if you want to compare two test runs. It also
reduces the amount of typing when using ttb
from the
erlang shell.
tp
tpl
ctp
ctpl
ctpg
With tp
and tpl
one of match specification shortcuts
may be used (example: ttb:tp(foo_module, caller)
). The shortcuts are:
return
- for [{'_',[],[{return_trace}]}]
(report the return value)caller
- for [{'_',[],[{message,{caller}}]}]
(report the calling function){codestr, Str}
- for dbg:fun2ms/1
arguments
passed as strings (example: "fun(_) -> return_trace() end"
)
list_history() -> History
History = [{N,Func,Args}]
All calls to ttb
is stored in the history. This
function returns the current content of the history. Any entry
can be re-executed with run_history/1
or stored in a
config file with write_config/2/3
.
run_history(N) -> ok | {error, Reason}
N = integer() | [integer()]
Executes the given entry or entries from the history
list. History can be listed with list_history/0
.
write_config(ConfigFile,Config)
Equivalent to write_config(ConfigFile,Config,[])
.
write_config(ConfigFile,Config,Opts) -> ok | {error,Reason}
ConfigFile = string()
Config = all | [integer()] | [{Mod,Func,Args}]
Mod = atom()
Func = atom()
Args = [term()]
Opts = Opt | [Opt]
Opt = append
This function creates or extends a config file which can be used for restoring a specific configuration later.
The content of the config file can either be fetched from
the history or given directly as a list of
{Mod,Func,Args}
.
If the complete history is to be stored in the config file
Config
should be all
. If only a selected number
of entries from the history should be stored, Config
should be a list of integers pointing out the entries to be
stored.
If Opts
is not given or if it is []
,
ConfigFile
is deleted and a new file is created. If
Opts = [append]
, ConfigFile
will not be deleted.
The new information will be appended at the end of the file.
run_config(ConfigFile) -> ok | {error,Reason}
ConfigFile = string()
Executes all entries in the given config file. Note that the history
of the last trace is always available in the file named
ttb_last_config
.
run_config(ConfigFile,NumList) -> ok | {error,Reason}
ConfigFile = string()
NumList = [integer()]
Executes selected entries from the given config
file. NumList
is a list of integers pointing out the
entries to be executed.
The content of a config file can be listed with
list_config/1
.
Note that the history
of the last trace is always available in the file named
ttb_last_config
.
list_config(ConfigFile) -> Config | {error,Reason}
ConfigFile = string()
Config = [{N,Func,Args}]
Lists all entries in the given config file.
write_trace_info(Key,Info) -> ok
Key = term()
Info = Data | fun() -> Data
Data = term()
The .ti
file contains {Key,ValueList}
tuples. This function adds Data
to the ValueList
associated with Key
. All information written with this
function will be included in the call to the format handler.
seq_trigger_ms() -> MatchSpec
Equivalent to seq_trigger_ms(all)
seq_trigger_ms(Flags) -> MatchSpec
MatchSpec = match_spec()
Flags = all | SeqTraceFlag | [SeqTraceFlag]
SeqTraceFlag = atom()
A match specification can turn on or off sequential
tracing. This function returns a match specification which
turns on sequential tracing with the given Flags
.
This match specification can be given as the last argument
to tp
or tpl
. The activated Item
will
then become a trigger for sequential tracing. This
means that if the item is called on a process with the
call
trace flag set, the process will be "contaminated"
with the seq_trace token.
If Flags = all
, all possible flags are set.
Please turn to the reference manual for the
seq_trace
module in the kernel
application to see the possible values for
SeqTraceFlag
. For a description of the match_spec()
syntax, please turn to the User's guide for the
runtime system (erts). The chapter Match Specification in Erlang explains the general match
specification "language".
Note!
The system tracer for sequential tracing is
automatically initiated by ttb
when a trace port is
started with ttb:tracer/0/1/2
.
Example of how to use the seq_trigger_ms/0/1
function:
(tiger@durin)5> ttb:tracer(). {ok,[tiger@durin]} (tiger@durin)6> ttb:p(all,call). {ok,{[all],[call]}} (tiger@durin)7> ttb:tp(mod,func,ttb:seq_trigger_ms()). {ok,[{matched,1},{saved,1}]} (tiger@durin)8>
Whenever mod:func(...)
is called after this, the
seq_trace token will be set on the executing process.
stop()
Equivalent to stop([])
.
stop(Opts) -> stopped | {stopped, Dir}
Opts = Opt | [Opt]
Opt = nofetch | {fetch_dir, Dir} | format | {format, FormatOpts} | return_fetch_dir
Dir = string()
FormatOpts = see format/2
Stops tracing on all nodes. Logs and
trace information files are sent to the trace control
node and stored in a directory named
ttb_upload_FileName-Timestamp
, where Filename
is
the one provided with {file, File}
during trace setup
and Timestamp
is of the
form yyyymmdd-hhmmss
. Even logs from nodes on the same
machine as the trace control node are moved to this directory.
The history list is saved to a file named ttb_last_config
for further reference (as it will be not longer accessible
through history and configuration management functions (like
ttb:list_history/0
).
The nofetch
option indicates that trace logs shall not be
collected after tracing is stopped.
The {fetch, Dir}
option allows to specify the directory
to fetch the data to. If the directory already exists, an
error is thrown.
The format
option indicates that the trace logs
shall be formatted after tracing is stopped. All logs in the fetch directory will be merged.
You may use {format, FormatOpts}
to pass additional
arguments to format/2
.
The return_fetch_dir
option indicates that the return value
should be {stopped, Dir}
and not just stopped
.
This implies fetch
.
get_et_handler()
The et
handler returned by the function may be used with format/2
or tracer/2
. Example: ttb:format(Dir, [{handler, ttb:get_et_handler()}])
.
format(File)
Same as format(File,[])
.
format(File,Options) -> ok | {error, Reason}
File = string() | [string()]
Options = Opt | [Opt]
Opt = {out,Out} | {handler,FormatHandler} | disable_sort
Out = standard_io | string()
FormatHandler = {Function, InitialState}
Function = fun(Fd,Trace,TraceInfo,State) -> State
Fd = standard_io | FileDescriptor
Trace = tuple()
TraceInfo = [{Key,ValueList}]
Out
erlang
module for details.flags
, client
and node
, and if handler
is given as option to the tracer function, this is also included. In addition all information written with the write_trace_info/2
function is included. Reads the given binary trace log(s). The logs are processed
in the order of their timestamp as long as disable_sort
option is not given.
If FormatHandler = {Function,InitialState}
,
Function
will be called for each trace message. If
FormatHandler = get_et_handler()
, et_viewer
in
the Event Tracer application (et
) is used for presenting
the trace log graphically. ttb
provides a few different
filters which can be selected from the Filter menu in the
et_viewer
. If FormatHandler
is not given, a
default handler is used which presents each trace message as a
line of text.
The state returned from each call of Function
is passed to the next call,
even if next call is to format a message from another log file.
If Out
is given, FormatHandler
gets the
file descriptor to Out
as the first parameter.
Out
is ignored if et
format handler is used.
Wrap logs can be formatted one by one or all in one go. To
format one of the wrap logs in a set, give the exact name of
the file. To format the whole set of wrap logs, give the name
with '*' instead of the wrap count. See examples in the
ttb
User's Guide.