TSP Lab	CDRWriter - CDR Writer module	CDRWriter

The CDR Writer module
`("CDRWriter")`

Basic Operation
User interface
Views
Configuration
Version history

Basic Operation

Receives assembled records from the protocol assemblers on its configured inputs and stores them to a date split directory hierarchy to time split CDR files (aka CDR database).
The application accepts ADG format records. The input records must contain a timestamp and a database record type ID in order to get stored. Records missing these informations will be rejected. Input records can be unordered by time. Every record will be appended to the end of the data file corresponding to its timestamp. To prevent frequent switching between output files, the application will leave the configured number of last used files open. The necessary number of parallel files can be calculated by dividing the time disordering interval of the input records by the configured file period.
The stored records are compressed by RLE encoding. Further Zip compression can be configured. The software uses compression level scale from 0 to 9.
Data protection of the stored records is supported via AES encryption if configured.
The application can be configured to flush the output buffers after every record. It is beneficial for slow traffic since the records will be accessible sooner in the output file, however this function has a performance penalty.
The application writes User ID (IMSI, MSISDN) index files if configured for the sake of faster record retrieval by User IDs. Index files are time split with the same time interval as data files. Every User ID with configured type found in any input record will result in a new index item.
There is a disk space watchdog function in the software, which monitors free space on the used disks. If the free space decreases below the configured limit, the processing stops automatically. The processing restarts once free disk space gets again above the limit. Recommended free disk space limit on EXT4 filesystem is 15-20% to prevent unwanted file fragmentation.
Operation can be followed on the screen by the counters
Logs the counters quarter-hourly and print out in CSV format

User Interface

The main menu

Menu	Menu item	Meaning
[File]
	Load state	Load a previously saved memory dump
	Save state	Save the internal memory content into a dump file
	Log counters	Save counters into log file
	Save counters	Save counters into CSV file
	Exit	Shuts down the program
[Tools]
	Start processing	Enable data processing
	Stop processing	Disable processing
	Clear state	Reset internal memory to initial value
	Flush state	Writes messages from internal buffers to CDR files
	Clear counters	Reset counters to zero
	Settings	Set INI file settings from menu
	Save settings	Save current settings into the INI file
[Help]
	Help	Currently not used
	About	Show information about the program

The bottom status-line

Current state of processing
Indication of the state of internal memories
Uptime: time elapsed since module has been started up

The "Log" tab

The "Log" view shows the latest log lines on the screen with their timestamps. The counters of the views are logged every quarter hour. The log level can be set through a drop-down list (Off, Normal, Detailed, Debug), the initial level is defined in the INI file. If the INI file does not exist (first-start without INI file), the default value is 'Normal'.

To search in the log entries, there is a 'Find' text-field. The 'next' button applies the text search and jumps to the next match.

Each log entry has a category, which is represented by the following icons:

Information about the operation
Warning: probably unplanned event occurred
Error: unplanned error during operation

The "CDR Writer" tab

Sections in views can be collapsed and expanded using the triangle button at the right up corner of the sections.

Counters in the "Incoming transactions" section

Records: Count of all incoming records
Free disk space: Free space on the target disk
Message time: Current message time

Counters in the "CDR Writer" section

Errors: Count of CDR Writer internal errors
IMSI indices: Count of stored IMSI indices
MSISDN indices: Count of stored MSISDN indices
CDRs written: Count of stored CDR records

Counters in the "Runtime" section

Preprocessing: Runtime percent of prepocessing
Output dropped: Runtime percent of output dropped
RLE compression: Runtime percent of RLE compression
Compression: Runtime percent of compression
Encription: Runtime percent of encription
Data writing: Runtime percent of data writing
Index writing: Runtime percent of index writing
Other: Runtime percent of input and watchdog

The "Connections" tab

This view shows the currently used input and output connections. Both input and output connections can be defined in INI file before starting the program, as well as during operation. Details of connections can be collapsed and expanded using the "expand" triangle button at the right up corner of the connections.

Incoming message time: The time of the last incoming message
Incoming bytes: The count of the incoming bytes
Incoming messages: The count of the incoming messages
Incoming sorter state: The incoming message sorted state
Outgoing message time: The time of the last outgoing message
Outgoing bytes: The count of the outgoing bytes
Outgoing messages: The count of the outgoing messages
Outgoing sorter state: The outgoing message sorted state

Parts of "Connections" tab

"New" button (): opens the 'Create connection' dialogue window
"Recent" button (): opens the list of the recently used connections
- "Clear history" button (): clears the recently used connections list
- "Copy history" button (): copies history to clipboard
Connection list: contains defined connections
- Connection state indicator
- "Remove connection" button (): closes and drops the connection (button is active when processing is disabled)
- Details can be collapsed and expanded using the triangle button at the right of the connections.

Connections of CDRWriter module

Input connection: The input of this module is an ADG type socket received from the CDR assembler module.

Input connection is defined in INI file. A sample configuration entry:
- cdrassembler=socket-adg://127.0.0.1:4243?compress=false&listen=true
Output connection: The CDRWriter module has no output connection.

Connection management

A new input connection can be added when processing state is inactive.
A connection:
- can be a new one created by pressing the "New connection" button or
- can be added a previously used one from the history list.
Connections can be removed using "Remove connection" button while processing is disabled.

The "Terminal" tab

The "Terminal" provides direct access to the core logic using commands. Available commands are listed on the right. Typically used for debug purposes.

Configuration (CDRWriter.ini) file settings

Section	Entry	Example	Meaning
[CDR]
	CompressionLevel	6	Compression level for CDRs. 0 means compression is switched off. Values 1..9 mean compression level.
	DataFilePeriod	15	Period of closing database files. Given in minutes
	DataPathPrimary	D:\CDR\	Primary path for CDR files
	DataPathAlternate	E:\CDR\	Alternative path for CDR files for the case of inaccessibility of the primary path
	DataPreallocationSize	100	Preallocated size of CDR files, in megabytes.
	Encrypt	1	Enables encryption of CDR files; 0 means off.
	IndexPath	D:\CDR\	Path for index files
	IndexPreallocationSize	10	Preallocation size in MB of index files
	MaxOpenFiles	2	Maximum number of files kept open simultaneously by type (CDR, Index)
	MinFreeDiskSpaceNeeded	1024	Min free space in MB needed on the disk. Below this limit processing will be stopped
	Autoflush	true	It set, will flush to disk after each record
	Caption	CDRWriter_11	Caption text for easy distinguishing amongst multiple instances of this module
[Counters]
	ClearOnWrite	false	Clear counters after history file written
	HistoryFile	D:\LogFiles\CDRWriter.csv	The path of the counter history file
	HistoryPeriod	15	The counter history file writing period in minutes
[Input]
	RestoreChannels	true	Restores the active channels after restart
	SortingQueueDelay	0	The sorting queue delay in milliseconds
	SortingQueueLength	0	The maximum size of the sorting queue
	StoragePointerFilePath	D:\DataFiles-7700	Path for optional storage pointer files (Sga-7700.xxx), if MUFI2 input is used
	SynchronizedIO	false	The input and the output channels will be synchronized
	CDRAssembler_socket	socket-adg://127.0.0.1:17003?listen=1	Definition of the listening TCP socket for incoming connection from the CDR Assembler module
[Log]
	LogDetailLevel	Normal	Stored level for Log message filter: 0: Off 1: Normal 2: Detailed 3: Debug
	LogFilesPath	D:\LogFiles	Directory used for storing log files
[Output]	Output connection is not used in this application
	RestoreChannels	false	Restores the active channels after restart
	SortingQueueDelay	0	The sorting queue delay in milliseconds
	SortingQueueLength	0	The maximum size of the sorting queue
	output_connection_name	output_connection_specification	Definition of the output connection
[RemoteUI]
	TCPHost	127.0.0.1	IP Address for command interface
	TCPPort	16007	Port for command interface
[State]
	AutoStart	true	The initial state of the processing
	StateFile	D:\CDRWriter.state	The path of the state file

Version history

v0.3.3

(c) Using AES encryption.

v0.3.2

(n) Add critical option to the channel parameters.
(c) Connection internal buffer size increased to 32MB.

v0.3.1

(f) Fixed error log line format;
(f) Log messages on process.start/stop reflect processing state;
(c) Processing cannot be started if not enough disk space;
(c) Processing won't restart after disk space freed if processing stopped by user.

v0.2.0

(n) Runtime counters;
(n) Autoflush INI parameter to configure flush after each record (true by default for backward compatibility);

v0.1.3

(n) Processing automatically paused till free disk space belowconfigured limit: [CDR]/MinFreeDiskSpaceNeeded

v0.1.2

(n) Keeps open CDR and index files simultaneously to prevent frequent reopening. Config: [CDR]/MaxOpenFiles
(n) Support for multiple IMSI/MSISDN index per record;
(f) Fixed wrong folder names;
(f) Fixed invalid magic code caused by faulty file closing;
(f) Fixed writing to separate drive;
(c) Variadic CDR size instead of fixed 4k;
(f) MSISDN index fixed - encoded missing length added;
(c) Error messages extended with timestamp;
(n) Errorneous input datagrams written out to configured output channel for debug purpose.
(n) Counters for IMSI/MSISDN indices;

v0.1.1

(n) Initial release

The CDR Writer module("CDRWriter")

Table of Contents