AITIA International, Inc. SgaMsgStore - Message Store module SgaMsgStore

Message Store module
("SgaMsgStore")

Table of Contents

Features

User interface


The main menu

Menu Menuitem Shortcut(s)/Button Meaning
[Log]      
  Find line containing... (Ctrl+F) Finds text in log window.
  Find next matching line (F3) Finds next occurrence of text in log window.
  Details   Detail level of logging (Off, Normal, Detailed, or Debug).
  Flush status line counters now   Writes status line counters into log file.
  Flush and zero status line counters now   Writes status line counters into log file and reset counters.
[Connections]      
  Accept incoming connection requests ( ) Enables or disables the acceptance of incoming connections
  Stop and disconnect all incoming connections   Makes existing incoming connection shut down in a graceful way
  Abort all incoming connections immediately (!!!BRUTAL!!!) Ctrl+X Aborts incoming connections immediately.
Note: When INI Advanced/bImmediateAbortConnection=true then the application will abort connections immediately without prompt (without "Are you sure to abort all connections?" window).
[Options]      
  Allow capturing onto disk ( ) Capturing can be enabled or disabled.
  Auto scroll ( ) Scrolls automatically the log window.
  Save window positions on exit   Saves actual window positions to ini file on exit.
  Saves settings   Saves actual settings into the configuration file.
[View]      
  Show log (Ctrl+1 / ) Shows log window.
  Show connection details (Ctrl+2 / ) Shows connection details window.
  Show next view (Ctrl+Tab) Shows next window.
[Help]      
  About... (Ctrl+F1) About the software (version number etc.).


Status line counters

"Connection" window

List of connections

"Link statistics" window

Link statistics:

Configuration (SgaMsgStore.ini) file settings

Section Entry Example Meaning
[Position]      
  posMain 8 115 954 864 1 -1 -1 Window positions at startup
[Options]      
  bSaveWinPosOnExit True Saves actual window position into the configuration file on exit
  bAutoScroll True Enable/Disable scrolling the window automatically
[Capture]      
  bAllowCapturingOntoDisk True Enable/Disable storage of captured data
[PacketSources]      
  wLocalTCPPort 7001 Local TCP Port for incoming connections
  127.0.0.1   Source IP Address of GyTapper/PaDi software or C-GEP device
[SgaFileSources]      
  wLocalTCPPort 7003 Local TCP Port for incoming connections
  127.0.0.1   Source IP Address for incoming TCP connections, parallel with "PacketSources" section list. SgaFile sources can be UDPProtFilter, TCPReassembler, etc.
[AllSources]      
  bAcceptConnectionRequest True Refers to the [Connections]/Accept incoming connection request menu item
[LinkID]     These settings refer to the first character of the LinkID
  sExpectCase U Defines the expected first character. Possible values: 'L' for lower-case, 'U' for upper-case, 'M' for mixed-case
  bToggleCase False If True then the capitalization of the fist character will be changed
[Advanced]      
  sCaption Message Store 2 Alternative caption text for easy distinguishing amongst multiple instances of this module
  wRecvBufferSizeMB 32 Size of receive buffer, given in megabytes
  wSgaFilePeriodMin 5 Close the current and open a new "SGA" file for operation, after the specified time (resolution is minute)
  dwSgaFilePreallocationSizeMB 100 Preallocation size of SGA files, in megabytes.
  wIndexPeriodMSec 1000 Index file resolution (in millisecond).
NOTE: if wIndexPeriodMSec is 0 then index files are not written
  wMaxSortDelayMSec 500 Sorting buffer size (in millisecond)
  dwMaxDupPacketDelayUSec 2000 Duplicated messages are omitted within this time frame, given is microsecond
NOTE: duplicate messages are omitted only at IPv4 packet sources;
dwMaxDupPacketDelayUSec=0 means keep all messages.
Deafult value is 2 ms
  bGbOverIP False If True then incoming messages are considered as Gb-over-IP (i.e. corresponding flag in message header is set)
  dwStatDumpIntervalSec 900 Time interval in second. Specifies the frequency of writing the statistical data into file for the SuperMon to send.
NOTE: dwStatDumpIntervalSec must be in the range 10...86400 and be a divisor of 86400
  dwMaxTrafficPauseSec 60 Alarms and traps generated if there is no traffic on a link after this period
NOTE: if dwMaxTrafficPauseSec == 0 then no alarm is sent when the traffic pauses
  wTimestampItemPeriodMSec 1000 Time interval in ms. Specifies the frequency of writing empty timestamp items
  wLogDetailLevel 2 Detail level of logging (0-Off, 1-Normal, 2-Detailed, or 3-Debug)
  dwMaxLinesInLogWindow 1000 Maximum number of lines in log window
  sLogFilesPath C:\LogFiles Path to the log files
  sTrapFilesPath d:\TrapFiles Output storage location, to store the trapfiles
  bAlternateTrapUID false Allows using fixed UIDs in SNMP traps (See SNMP trap list below for details)
  sDataPath C:\DataFiles Output storage location, to store the "SGA" and "SGAi" files
  sDataFile7700Path1 C:\DataFiles-7700 Output storage location, to store the SgaMUFI2 files
  bDataFile7700_UseMufiCom2 true MuFiCom writer version selectable. Default is 'true' (compatible with IPReassembler).
  sDataFile7001Path d:\DataFiles-7001 Output storage location, to store the statitics SgaMUFI2 files
  bImmediateAbortConnection True If this value is 'true' then the application will abort connections immediately without prompt (without "Are you sure to abort all connections?" window).

SNMP trap list

UID Type Text Remarks To Do
If bAlternateTrapUID is set to TRUE
5000 CEA 'SGA Message Store v1.00' module is started. Send a welcome message; indication of module start-up  
5000 CRI 'SGA Message Store v1.00' has been shut down by operator. Send a farewell message; indication of module shut down Start the 'SGA Message Store v1.00' module
5001 INF Accepted connection request from x.y.z.w ('Source1').    
5001 INF Connection to x.y.z.w ('Source1') closed. Connection closed by remote side. Check remote side of the connection.
5001 WAR Connection to x.y.z.w ('Source1') broke down. Connection broke down (unknown reason). Check remote side of the connection.
5100 WAR No traffic on connection x.y.z.w ('Source1'). No traffic indication on a connection, after the timer dwMaxTrafficPauseSec expired. Check remote side of the connection.
5100 CEA No traffic on connection x.y.z.w ('Source1').    
5200 ERR Could not write data to disk.   Check if there is enough space on disk.
5201 ERR N unsorted messages in the last period.    
If bAlternateTrapUID is set to FALSE
0 CEA 'SGA Message Store v1.00' module is started. Send a welcome message; indication of module start-up  
0 CRI 'SGA Message Store v1.00' has been shut down by operator. Send a farewell message; indication of module shut down Start the 'SGA Message Store v1.00' module
++N INF Accepted connection request from x.y.z.w ('Source1').    
++N INF Connection to x.y.z.w ('Source1') closed. Connection closed by remote side. Check remote side of the connection.
++N WAR Connection to x.y.z.w ('Source1') broke down. Connection broke down (unknown reason). Check remote side of the connection.
++N WAR No traffic on connection x.y.z.w ('Source1'). No traffic indication on a connection, after the timer dwMaxTrafficPauseSec expired. Check remote side of the connection.
N CEA No traffic on connection x.y.z.w ('Source1').    
++N ERR Could not write data to disk.   Check if there is enough space on disk.
++N ERR N unsorted messages in the last period.    

Frequently asked questions for sorting

 First of all let see some definitions for the used buffers in TCP handling.
MsgStore handles at most 20 TCP connections. Every TCP connections has a TCPStreamReader entity which reads the correspondig socket, buffers the incoming bytes, and splits these bytes into messages. Every TCPStreamReader (so every TCP connection) has its own buffer, where the received bytes are stored temporarily. The size of this buffer can be set in INI/Advanced/wRecvBufferSizeMB entry.
NOTE: it is not sure that the actual size of the buffer exactly equals to this value, because it is limited to 1024MB for each buffer and it is rounded up to the next power of 2 (considering bytes).
For example if the entry is 15 (MB) for every TCP connections it is allocated 16*1024*1024 bytes, and further 66KB for an additional maximum sized TCPItem arriving on the socket. This is the so called ReceivedBuffer.
An additional memory is allocated for sorting, this is rounded up to the next power of 2, as well. MsgStore assumes that an average TCPItem is 256 bytes, so calculates how many Item will be stored in the ReceivedBuffer. Looking at the previous example it will be 64*1024 TCPItem. Sorting occurs in a double linked list, where only extracts are stored, not the whole TCPItems. One extract is 24 bytes. This double linked list is the NodeBuffer.
If a bunch of bytes arrives on the stream (assuming a complete TCPItem), then these bytes are copied to the end of ReceiveBuffer, and the extract is inserted into the NodeBuffer (of course with sorting).

Questions:

  1. What does wMaxSortDelayMSec entry mean exactly?
    This is the maximum sorting delay, which is expected. MsgStore tries to sort TCPItems with this time window. If the INI settings fits to the traffic, then "... ms sorter delay" line on GUI must show approximately the wMaxSortDelayMSec value. See next question.
  2. What value can be seen in the "... ms sorter delay" line, and why can be differ from wMaxSortDelayMSec value, usually much less than the configured value?
    The real sorting time window is not affected by wMaxSortDelayMSec, because it is only an expectation. This window (where the actual sorting is working) is shown in "... ms sorter delay" line, this is the difference of the first and last timestamps of TCPItems in NodeBuffer. This can be much less than the expected because the handled message count (and indirectly the time window) depends on the buffer sizes and the traffic speed as we see above. Theoretically the time window equals the ratio of buffer size (wRecvBufferSizeMB) and the traffic speed. In practice the time window is much smaller, because MsgStore checks the following two conditions: If one of the conditions is met a protection mechanism is started. (This prevents buffer overflow.) In this time the median node is selected in NodeBuffer, and its timestamp remains the oldest one, all older Items/Nodes are removed from buffers (they are written into the sga files). The 25 percentages is a magical number, maybe it can be bigger, but it must be choosen very carefully to prevent buffer overrun. If the INI settings (configuration) does not fit into the traffic (receive buffer is too small), this protection mechanism switches on too early, the real sorting time window will be only some hundreds of msec despite of the expected sec value.
  3. What does the percentages in "queues sat." line mean exactly?
    The first value belongs to the ReceiveBuffer, the second one to the NodeBuffer. These percentages show the maximum ratio of the currently used and the allocated buffer size since the previous GUI refreshing (~1 sec). It comes from Answer 2. if the values equal or are above 25% the protection mechanism is working, and there is a chance that there will be unsorted messages/files.

Version history

Known bugs