Name

vtapericsson — Entry point for Ericsson 5g cloud monitoring

Synopsis

vtapericsson CONFIGFILE

vtapericsson [OPTIONS]

EXAMPLES

Generating a default config for an instance named foo:

vtapericsson -d >/etc/aitia/vtapericsson.foo.rc

Note: the application won’t start without setting the mandatory parameters in the generated config.

Starting the foo instance:

vtapericsson /etc/aitia/vtapericsson.foo.rc

DESCRIPTION

vtapericsson is a essential component of AITIA’s 5G monitoring. The application handles the incoming vTap datastream generated by Ericsson’s virtual network functions and filters and converts data format to the sga 5G dataformat described in the design documentation (SGA_5G_Monitorozas_alapjai).

The application has 3 functional layers:

  • Input layer, which handles the incoming connections and the IPv4/UDP/GRE/PCAPNG/IPv4/TCP transport/encapsulation layers. No data is captured from this headers, it is only used for statistic calculations. 5 minute metadata packets are discarded currently. Unlike other sga products a single GRE input channel can accept multiple connections
  • Formatter layer, which converts the JSON payload of the datastream to the unified sga 5G dataformat and generates linkID. Logic of the process is described in the design documentation (SGA_5G_Monitorozas_alapjai).
  • Outbound transport layer forwards the converted datastream for storage or distribution or writes the data to disk. All connection lib dataformats are supported. The application is stateless every GRE packet handled separately. The information used for LinkID creation is stored in SGA MySQL database which pooled periodically by the application for changes, changes to the database are applied automatically. If the database is not available, the application wont start if the database becomes unavailable during operation then the application will used the cached data until the database becomes available again.Performance considerations, packet loss under high loads is highly dependent on the OS’s UDP buffer size. Debian 11 has a very small buffer by default, it is 208KB. This should be increased to 64MB.

Query UDP buffer size:

sysctl net.core.rmem_max
net.core.rmem_max = 212992

Set UDP buffer size (volatile):

sudo sysctl -w net.core.rmem_max=67108864
net.core.rmem_max = 67108864

Set UDP buffer size (non-volatile):

sudo echo "net.core.rmem_default=67108864" > /etc/sysctl.conf
sudo echo "net.core.rmem_max=67108864" > /etc/sysctl.conf

Example rc file: 

# Command channel socket address in 'host:port' format
configs.set app.address 127.0.0.1:16022

# Automatic linkid database update period
configs.set linkid.refresh 1m

# URI for connecting to the SQL DB for linkid lookup. E.g.: 127.0.0.1?dbname=xxx&user=yyy&pass=zzz
configs.set linkid.uri '10.6.163.92:7008?dbname=sga&user=sga&pass=sga'
# The rulegroup of the application instance
configs.set linkid.group vTapEric

io.open in socket-gre://10.6.200.137:4754?type=udp&critical=false&name=c1

#io.open out file-sga://test_full.sga
io.open out socket-adg://127.0.0.1:9998?name=msgwriter_local
io.sorter in 1000 1000000

process.start

The input and output channel have to be configured and the process have to be started. If multiple input channels are used the input io.sorter have to be configured for 1s sorting, and every channel have to be configured as non-critical. The JSON value in the resp parameter have to be put between ''. The application only uses filtering rules defined for its group and group "All".

The application detaches itself from the terminal on startup. It can be accessed via TCP command channel using sga-rc(1) application or TCP socket. The application also responds to OS signals (see later).

OPTIONS

The config file of the application instance is a mandatory parameter. Recommended directory for Aitia application config files is /etc/aitia, recommended naming convention for config files is executable.instance.rc, where executable is the name of the application binary and instance is a user selected custom name for the application instance.

Note: Without specifying a configuration file the application will fail on startup.

List of available options:

CONFIGFILE
path to the config file
-h, --help
print this help and exit
-j, --jsondoc
print documentation in json format
-v, --version
print version number
-d, --default
print default config file

CONFIGS

The config file contains the same commands that are available from command interface.

Note: The application will fail on start if the specification of any mandatory parameter is missing.

Note: Time duration type config parameters need the specification of the time unit. Available time units are: ns, us, ms, s, m, h, d. Examples: 100ms, 5m.

List of available config parameters:

app.address (mandatory)
Command channel socket address in host:port format
linkid.group
Group name for Alias rule lookup
linkid.refresh (def: 1m)
Automatic linkid database update period
linkid.uri
URI for connecting to the SQL DB for linkid lookup. E.g.: 127.0.0.1?dbname=xxx&user=yyy&pass=zzz

COMMANDS

The application can receive user commands on the specified TCP/IP interface. Command interface is using a bash like format. Every command should be in a new separate line. Commands can have parameters, the command name and parameters are separated with whitespace characters.

If the parameter value contains whitespace, it should be escaped or quoted. The backslash (\) character escapes the following character (e.g: state.load /dir\ with\ spaces/statefile). The single quote (') character can escape multiple characters (e.g: state.load '/dir with spaces/statefile').

The hash (#) symbol can be used to comment out parts of the command. The commented part starts at the comment character and ends at the end of the line (e.g: state.load /path/statefile #opens the state file).

io.sorter: io.sorter can be configured to empty sorter buffer if no message received in the last n milliseconds. After the last incoming message a real-time timeout starts based on the CPU’s timestamp-counter register (rdtsc), if no message received in the configured timeout then the sorter will be flushed.

Format: io.sorter <delay> <length> <timeout> , if timeout set for 0, then there is no flushing legacy mode

  • Legacy mode is recommended for non-test, multi input softwares, which work from buffered data channels eg.: modules after storage1
  • New mode recommended for non-buffered inputs (vtaps, writers) and very low rate links
  • Can be imprecise on very old hardware: X7-X8. Works well and tested on X9-X10-X11-AMD Epyc gen2-4

List of available commands:

app.quit <Force>

Quits the application. Inactive processing and saved state is required before non-forced quit.

<Force>

normal quit (by default) or forced.

(available values: force, normal)

configs.get <Parameter>

Returns a config parameter. Note: Glob pattern can be used to specify more parameters.

<Parameter>
config name or glob pattern
configs.set <Parameter> <Value>

Changes a config parameter.

<Parameter>
config name
<Value>
New value to be stored
info.get
Returns general application info in JSon format.
io.close <Direction> <Connection>

Close connection(s).

<Direction>

direction of the connection

(available values: in, out, both)

<Connection>
connection id or glob pattern
io.doc <Direction> <Transport> <Format>

Transport, format and parameter info of channels in JSon.

<Direction>

direction of the connection or glob pattern

(available values: in, out)

<Transport>

transport name or glob pattern

(available values: socket, sgacard, mufi, file, std, pcap, dir, pipe, ws, http2, jbin, sgasocket)

<Format>

format name or glob pattern

(available values: adg, pcap, sga, hex, pstream, tapper, json, yaml, perldump, raw, gre, avro)

io.open <Direction> <Uri>

Opens a new input or output channel.

<Direction>

direction of the connection

(available values: in, out)

<Uri>
'transport-format://location?optional_parameter=value&optional_parameter=value
io.sorter <Direction> <Queue delay> <Queue length> <Queue timeout>

Sets message sorter for a connection.

<Direction>

direction of the connection

(available values: in, out, both)

<Queue delay>
sorting queue delay in milliseconds
<Queue length>
maximum size of the sorting queue
<Queue timeout>
sorting queue is flushed after inactivity (in clk cycles/1 ms)
process.start
Starts the processing of the incoming data.
process.stop
Stops the processing of the incoming data.
process.stopsync
Stops the processing of the incoming data if there is no incoming traffic.
state.clear
Clears the state of the application. The temporally stored messages will be dropped.
state.load <State file>

Loads application states from file.

<State file>
state file name
state.save <State file>

Stores application states from file.

<State file>
state file name
vars.clear
Clears the nonprotected application counters.
vars.get <Counter>

Returns one or more counters in json format. Note: Glob pattern can be used to specify more parameters.

<Counter>
counter name or glob pattern

COUNTERS

List of counters:

io.all.input.bytes (unit: B)
Count of incoming bytes
io.all.input.msgs (unit: Msg)
Count of incoming messages
io.all.input.sorter
Current state of the incoming message sorter
io.all.input.time
Time of the last incoming message
io.all.output.bytes (unit: B)
Count of outgoing bytes
io.all.output.msgs (unit: Msg)
Count of outgoing messages
io.all.output.sorter
Current state of the outgoing message sorter
io.all.output.time
Time of the last outgoing message
io.state
Current state of processing

STATES

The application can load and save its states on demand (receiving corresponding commands). There is no automatic state handling.

Note: The state file is a TAR format file pack.

SIGNALS

The application handles the following Signals from the OS:

SIGHUP
Restarts the TCP command channel.
SIGTERM, SIGINT, SIGQUIT
The application exits gracefully stopping the processing and closing the connection channels.
SIGABRT, SIGFPE, SIGILL, SIGSEGV
The application exits ungracefully.

CONNECTIONS

The application can handle multiple input/output channels. The channels are composed of a transport method, a capture formatter used for packet encapsulation and optional filters. There are multiple transports and formats available, which can be used in different variations.

Channels are configured using URI format:

<transport>-<format>://<resource>?<parameters>

Examples:

Input ADG format over listen socket:

socket-adg://127.0.0.1:12345?listen=1&name=MyInput

Output ADG format over socket:

socket-adg://127.0.0.1:12345?name=MyOutput
Input channel parameters:
name
User defined name of the channel, will be generated if not supplied.
resetwait
Waiting time before reset in msec.
wait
Waiting time in msec.
buffer
Nonzero value specifies the buffered messages count.
critical
If the criticality of a channel is set to false, processing will not stop if the channel stops. (def: true)
Output channel parameters:
name
User defined name of the channel, will be generated if not supplied.
resetwait
Waiting time before reset in msec.
wait
Waiting time in msec.
buffer
Nonzero value specifies the buffered messages count.
critical
If the criticality of a channel is set to false, processing will not stop if the channel stops. (def: true)
Input transport: socket

Reads data from TCP socket.

Parameters:
listen
Creates listen socket if set.
type
The type of the socket, values: tcp, udp. The default is: tcp
bindaddr
The local binding IP address of the socket.
Input transport: sgacard
Reads data from SGA card.
Input transport: mufi

Reads multiple data files splitted by date/time using MuFi files as file pointers.

Parameters:
mufi
Mufi file path and name without extention.
verbose
Enables debug log on the transport.
cleanup
Deletes input data file after processing.
Input transport: file

Reads data from file.

Parameters:
loop
Number of repeated reading of the file.
from
The start position in the file.
Input transport: std
Reads data from standard input.
Input transport: pcap
Reads data from card using libPCap.
Input transport: dir

Reads data from multiple files in a directory.

Parameters:
mask
Glob matching pattern for files to use.
from
Filename of the first file to be used.
pos
Filename lst used by previous session.
loop
Number of repeated reading of the directory.
Input transport: pipe

Reads data from named pipe.

Parameters:
listen
Creates listen namedpipe if set.
Input transport: ws

Reads data from websocket.

Parameters:
sslcert
SSL Certificate File.
Input transport: http2
Reads data from HTTP2 stream over TCP socket.
Input transport: jbin
Generate raw binary samples based from JSON description.
Input transport: sgasocket

Reads data from TCP socket.

Parameters:
listen
Creates listen socket if set.
Output transport: null
Discards any data sent out.
Output transport: socket

Writes data to TCP socket.

Parameters:
listen
Creates listen socket if set.
type
The type of the socket, values: tcp, udp. The default is: tcp
bindaddr
The local binding IP address of the socket.
Output transport: file
Writes data to file.
Output transport: std
Writes data to standard output.
Output transport: dir

Writes data to the specified directory.

Parameters:
mask
Glob matching pattern for files to use.
from
Filename of the first file to be used.
pos
Filename lst used by previous session.
file
The static part of the output file name.
ext
Extention of the output files.
interval
Time interval for file splitting in seconds.
Output transport: pipe

Reads data from named pipe.

Parameters:
listen
Creates listen namedpipe if set.
Output transport: ws

Writes data to websocket.

Parameters:
sslcert
SSL Certificate File.
Output transport: sgasocket

Writes data to TCP socket.

Parameters:
listen
Creates listen socket if set.
Format: adg

Native format of the datagram.

Parameters:
compress
Activates the compression of the transmitted data.
Format: pcap

Payload and capture info (parts can be represented in) PCap file format.

Parameters:
fileheader
Data stream starts with a file header.
Format: sga

Payload and all additional info in SGA file format.

Parameters:
fileheader
Data stream starts with a file header.
storepos
Insert position of the message from input transport, ignore any position info already included.
Format: hex
Raw bytes of the datagram in text/hexdump.
Format: pstream

Payload and capture info (parts can be represented in) PacketStream/Tapper stream format.

Parameters:
version
Version of packet stream format (autodetected on input).
Format: tapper
Payload and capture info in Tapper stream format.
Format: json

Text serialization of the datagram.

Parameters:
indent
Use indentation if enabled.
Format: yaml

Text serialization of the datagram.

Parameters:
indent
Use indentation if enabled.
Format: perldump

Text serialization of the datagram.

Parameters:
indent
Use indentation if enabled.
Format: raw

Raw format wrapped to/extracted from payload content. Transport should be packet based.

Parameters:
linkName
If set, attaches capture info to input with supplied link name.
Format: gre
Payload and capture info (parts can be represented in) Dg/GRE stream format.
Format: avro

Avro schema based data formatter

Parameters:
binding
Full path of the avro binding descriptor json file
schema
Full path of the avro schema descriptor json file

BUGS

Report bug at http://tsplab.hu/redmine.

COPYING

Copyright © 2020 Aitia International Inc.