Name

msgwriter — Aitia SGA message storage writer daemon

Synopsis

msgwriter CONFIGFILE

msgwriter [OPTIONS]

EXAMPLES

Generating a default config for an instance named foo:

msgwriter -d >/etc/aitia/msgwriter.foo.rc

Note: the application won’t start without setting the mandatory parameters in the generated config.

Starting the foo instance:

msgwriter /etc/aitia/msgwriter.foo.rc

DESCRIPTION

msgwriter is a message storage daemon, which receives captured messages on its configured inputs and stores them to a date splitted directory hierarchy to time splitted SGA files in a time ordered manner.

The application detaches itself from the terminal on startup. It can be accessed via TCP command channel using rc(1) application or TCP socket. The application also responds to OS signals (see later).

The input buffer size is increased to 32MB in this application.

To be able to store unordered traffic, timestamp sorter should be configured for input connection. Sorter has a configurable time window. If older message received than the time window, the message will be dropped. Dropped messages can be stored by configuring an output channel with a channel parameter (see later) to the application. Note: The output sorter is disabled in this application.

The application writes an index file for every message data file, for the sake of faster time-based random access to the message data. The index file contains one record for each time period (confiurable) containing the actual data file position.

If configured, msgwriter filters out duplicated IPv4 messages.

The application can change the first letter of SGA link ID to uppercase in the messages, if configured.

There is a disk space watchdog function in the software, which monitors free space on the used disks. If the free space decreases below the configured limit, the processing stops automatically. The processing restarts once free disk space gets again above the limit. Recommended free disk space limit on EXT4 filesystem is 15-20% to prevent unwanted file fragmentation.

OPTIONS

The config file of the application instance is a mandatory parameter. Recommended directory for Aitia application config files is /etc/aitia, recommended naming convention for config files is executable.instance.rc, where executable is the name of the application binary and instance is a user selected custom name for the application instance.

Note: Without specifying a configuration file the application will fail on startup.

List of available options:

CONFIGFILE
path to the config file
-h, --help
print this help and exit
-j, --jsondoc
print documentation in json format
-v, --version
print version number
-d, --default
print default config file

CONFIGS

The config file contains the same commands that are available from command interface.

Note: The application will fail on start if the specification of any mandatory parameter is missing.

Note: Time duration type config parameters need the specification of the time unit. Available time units are: ns, us, ms, s, m, h, d. Examples: 100ms, 5m.

List of available config parameters:

app.address (mandatory)
Command channel socket address in host:port format
ipdedup.enabled (def: true)
Enable IP based duplication filtering
ipdedup.maxBufferSize (def: 10000)
Max number of messages stored in deduplicator
ipdedup.messageLifetime (def: 1ms)
Time period messages stored
writer.data.filePeriod (def: 15m)
Data file writing period
writer.data.path (mandatory)
The path of the message data
writer.freeDiskSpaceNeeded (def: 1GiB)
Free space limit for processing, recommended: 15-20% (in B, KiB, MiB, GiB).
writer.index.path (mandatory)
The path of the message indices
writer.index.recordPeriod (def: 1s)
Index record period
writer.mufi.maxFileSize (def: 1MiB)
Max file size of mufi files (in B, KiB, MiB or GiB)
writer.mufi.minLifetime (def: 1s)
Min lifetime of mufi files
writer.mufi.path (mandatory)
The path of the mufi files
writer.mufi.writePeriod (def: 1s)
Time period of writing records to mufi file
writer.uppercaseLinkId (def: false)
Make the first letter of link Id uppercase

COMMANDS

The application can receive user commands on the specified TCP/IP interface. Command interface is using a bash like format. Every command should be in a new separate line. Commands can have parameters, the command name and parameters are separated with whitespace characters.

If the parameter value contains whitespace, it should be escaped or quoted. The backslash (\) character escapes the following character (e.g: state.load /dir\ with\ spaces/statefile). The single quote (') character can escape multiple characters (e.g: state.load '/dir with spaces/statefile').

The hash (#) symbol can be used to comment out parts of the command. The commented part starts at the comment character and ends at the end of the line (e.g: state.load /path/statefile #opens the state file).

List of available commands:

app.quit <Force>

Quits the application. Inactive processing and saved state is required before non-forced quit.

<Force>

normal quit (by default) or forced.

(available values: force, normal)

configs.get <Parameter>

Returns a config parameter. Note: Glob pattern can be used to specify more parameters.

<Parameter>
config name or glob pattern
configs.set <Parameter> <Value>

Changes a config parameter.

<Parameter>
config name
<Value>
New value to be stored
info.get
Returns general application info in JSon format.
io.close <Direction> <Connection>

Close connection(s).

<Direction>

direction of the connection

(available values: in, out, both)

<Connection>
connection id or glob pattern
io.doc <Direction> <Transport> <Format>

Transport, format and parameter info of channels in JSon.

<Direction>

direction of the connection or glob pattern

(available values: in, out)

<Transport>

transport name or glob pattern

(available values: socket, sgacard, mufi, file, std, pcap, dir, pipe, ws)

<Format>

format name or glob pattern

(available values: adg, pcap, sga, hex, pstream, tapper, json, yaml, perldump, raw)

io.open <Direction> <Uri>

Opens a new input or output channel.

<Direction>

direction of the connection

(available values: in, out)

<Uri>
'transport-format://location?optional_parameter=value&optional_parameter=value
io.sorter <Direction> <Queue delay> <Queue length>

Sets message sorter for a connection.

<Direction>

direction of the connection

(available values: in, out, both)

<Queue delay>
sorting queue delay in milliseconds
<Queue length>
maximum size of the sorting queue
process.start
Starts the processing of the incoming data.
process.stop
Stops the processing of the incoming data.
process.stopsync
Stops the processing of the incoming data if there is no incoming traffic.
state.clear
Clears the state of the application. The temporally stored messages will be dropped.
state.load <State file>

Loads application states from file.

<State file>
state file name
state.save <State file>

Stores application states from file.

<State file>
state file name
vars.clear
Clears the nonprotected application counters.
vars.get <Counter>

Returns one or more counters in json format. Note: Glob pattern can be used to specify more parameters.

<Counter>
counter name or glob pattern

COUNTERS

List of counters:

io.all.input.bytes (unit: B)
Count of incoming bytes
io.all.input.msgs (unit: Msg)
Count of incoming messages
io.all.input.sorter
Incoming message sorter state
io.all.input.time
Time of the last incoming message
io.all.output.bytes (unit: B)
Count of outgoing bytes
io.all.output.msgs (unit: Msg)
Count of outgoing messages
io.all.output.sorter
Outgoing message sorter state
io.all.output.time
Time of the last outgoing message
io.state
Current processing state
ipdedup.duplicatedMsgs (unit: Msg)
Count of all duplicated message
ipdedup.incomingMsgs (unit: Msg)
Count of all incoming message
ipdedup.messageTime
Current message time
ipdedup.nonIP4Msgs (unit: Msg)
Count of all non IPv4 message
ipdedup.outgoingMsgs (unit: Msg)
Count of all outgoing message
ipdedup.processingErrors (unit: Msg)
Count of all message processing faild on
ipdedup.storedMsgs (unit: Msg)
Count of stored IPv4 messages
msg.input.freeDiskSpace (unit: B)
Free disk space
msg.input.messages (unit: Msg)
Count of all incoming message
msg.input.time
Current message time
msg.writer.error (unit: Err)
Count of Message Writer internal errors
msg.writer.storedMsgs (unit: Msg)
Count of messages written on disk

STATES

The application can load and save its states on demand (receiving corresponding commands). There is no automatic state handling.

Note: The state file is a TAR format file pack.

SIGNALS

The application handles the following Signals from the OS:

SIGHUP
Restarts the TCP command channel.
SIGTERM, SIGINT, SIGQUIT
The application exits gracefully stopping the processing and closing the connection channels.
SIGABRT, SIGFPE, SIGILL, SIGSEGV
The application exits ungracefully.

CONNECTIONS

The application can handle multiple input/output channels. The channels are composed of a transport method, a capture formatter used for packet encapsulation and optional filters. There are multiple transports and formats available, which can be used in different variations.

Channels are configured using URI format:

<transport>-<format>://<resource>?<parameters>

Examples:

Input ADG format over listen socket:

socket-adg://127.0.0.1:12345?listen=1&name=MyInput

Input Tapper format over listen socket:

socket-tapper://127.0.0.1:12345?listen=1&name=MyInput

Output ADG format over socket:

socket-adg://127.0.0.1:12345?name=MyOutput

Output sga format file for dropped messages:

file-sga://path/dropped.sga?name=MyDroppedOutput&passlabels=dropped
Input channel parameters:
name
User defined name of the channel, will be generated if not supplied.
resetwait
Waiting time before reset in msec.
wait
Waiting time in msec.
buffer
Nonzero value specifies the buffered messages count.
critical
If the criticality of a channel is set to false, processing will not stop if the channel stops. (def: true)
Output channel parameters:
name
User defined name of the channel, will be generated if not supplied.
resetwait
Waiting time before reset in msec.
wait
Waiting time in msec.
buffer
Nonzero value specifies the buffered messages count.
critical
If the criticality of a channel is set to false, processing will not stop if the channel stops. (def: true)
passlabels
Comma separated list of labels to be passed. (available values: normal, dropped)
blocklabels
Comma separated list of labels to be blocked. (available values: normal, dropped)
Input transport: socket

Reads data from TCP socket.

Parameters:
listen
Creates listen socket if set.
Input transport: sgacard
Reads data from SGA card.
Input transport: mufi

Reads multiple data files splitted by date/time using MuFi files as file pointers.

Parameters:
mufi
Mufi file path and name without extention.
verbose
Enables debug log on the transport.
cleanup
Deletes input data file after processing.
Input transport: file

Reads data from file.

Parameters:
loop
Number of repeated reading of the file.
from
The start position in the file.
Input transport: std
Reads data from standard input.
Input transport: pcap
Reads data from card using libPCap.
Input transport: dir

Reads data from multiple files in a directory.

Parameters:
mask
Glob matching pattern for files to use.
from
Filename of the first file to be used.
pos
Filename lst used by previous session.
loop
Number of repeated reading of the directory.
Input transport: pipe

Reads data from named pipe.

Parameters:
listen
Creates listen namedpipe if set.
Input transport: ws

Reads data from websocket.

Parameters:
sslcert
SSL Certificate File.
Output transport: null
Discards any data sent out.
Output transport: socket

Writes data to TCP socket.

Parameters:
listen
Creates listen socket if set.
Output transport: file
Writes data to file.
Output transport: std
Writes data to standard output.
Output transport: dir

Writes data to the specified directory.

Parameters:
mask
Glob matching pattern for files to use.
from
Filename of the first file to be used.
pos
Filename lst used by previous session.
file
The static part of the output file name.
ext
Extention of the output files.
interval
Time interval for file splitting in seconds.
Output transport: pipe

Reads data from named pipe.

Parameters:
listen
Creates listen namedpipe if set.
Output transport: ws

Writes data to websocket.

Parameters:
sslcert
SSL Certificate File.
Format: adg

Native format of the datagram.

Parameters:
compress
Activates the compression of the transmitted data.
Format: pcap

Payload and capture info (parts can be represented in) PCap file format.

Parameters:
fileheader
Data stream starts with a file header.
Format: sga

Payload and all additional info in SGA file format.

Parameters:
fileheader
Data stream starts with a file header.
storepos
Insert position of the message from input transport, ignore any position info already included.
Format: hex
Raw bytes of the datagram in text/hexdump.
Format: pstream

Payload and capture info (parts can be represented in) PacketStream/Tapper stream format.

Parameters:
version
Version of packet stream format (autodetected on input).
Format: tapper
Payload and capture info in Tapper stream format.
Format: json

Text serialization of the datagram.

Parameters:
indent
Use indentation if enabled.
Format: yaml

Text serialization of the datagram.

Parameters:
indent
Use indentation if enabled.
Format: perldump

Text serialization of the datagram.

Parameters:
indent
Use indentation if enabled.
Format: raw

Raw format wrapped to/extracted from payload content. Transport should be packet based.

Parameters:
linkName
If set, attaches capture info to input with supplied link name.

BUGS

Report bug at http://tsplab.hu/redmine.

COPYING

Copyright © 2020 Aitia International Inc.