Current view: XpoLog V5. Available: XpoLog V6 and XpoLog V7 (Latest)

Skip to end of metadata
Go to start of metadata

XpoLog enables users to use the Patterns language to normalize log records into tabular format.

Records in the log can usually be presented by a combination of types. It is recommended to present the logged information in as detailed way as possible. Doing so gives greater possibilities in manipulating and analyzing the data, such as filtering by a specific ip, priority, date, or specific text column. You should try to be as descriptive as possible.
Note that you can configure several patterns for one log using XpoLog multi pattern.
Each command will be treated as a column of data in the log view table. 
 
Note: If a pattern is formulated incorrectly, the log records may display wrong data or no data at all.

IMPORTANT!!!
After defining a pattern, it is highly recommended to click the verify pattern link to see the results of your definition in the table at the bottom of the page.
If you do not see data at all or you see wrong data, check your pattern definition.

For any definition problem, send XpoLog support the log example by email to support@xplg.com, and we will help you define a pattern.

The following table describes the Patterns language:

Pattern Type  Syntax Attributes Examples   
Text in Log PatternResult in Log View 
Stringany string of characters, including multi line strings{string}No special attributes

Hello world. 

Hello world.

Hello
world. 

{string}

H{string} wor{string}

{string}

Hello world.  

first column = ello; second column = ld.

Hello world.

Textany single-line string of characters{text}No special attributesHello world. {text}Hello world.
Datea date string{date, MM-dd-yyyy}

timeDiff: specify time offset in milliseconds
 locale: specify the locale in which the log was written
localeView: specify the locale in which the log should be displayed.

Also, see Optional Identifiers for Date Pattern section below.

18:05:23

18:05:23

{date,locale=it,dd MM yyyy HH:mm:ss}


{date,locale=it; localeView=en,dd MM yyyy}

{date,timeDiff=1000,HH:mm:ss}

{date,timeDiff=-5000,HH:mm:ss}

 18:05:24

18:05:18

Timestamp  a timestamp representing a date string {timestamp}product: XpoLog expects a timestamp in milliseconds, in case the timestamp is not in milliseconds use the product to add zeros at the end
  • 1388534400000
  • 1414973208
  • {timestamp,yyyy/MM/dd HH:mm:ss.SSS}  
  • {timestamp,product=1000,yyyy-MM-dd HH:mm:ss.SSS}
  •  2014/01/01 02:00:00.000
  • 2014/11/03 02:06:48.000
 Number a numeric string {number}No special attributes

Thread-1

Test 5 done 

calling to id 5667

 {string}-{number}

{string} {number} {string}

calling to id {number}

first column = Thread;second column = 1

first column = Test; second column = 5; third column = done

5667

Choice   a set of strings that can appear in a record

 {choice,value1;value2...,valueN}

Note: All optional choices should appear as a semicolon separated list inside the tag.

No special attributesfirst record = red,
second record = green,
third record = blue  
 {choice,red;green;blue}The matching choice of each record is displayed.
IP Address An IP address {ip}No special attributes 127.0.0.1  {ip}  127.0.0.1
Geo IP AddressA Geo IP address

 {geoip,type=country:region:city}

Note: All optional combinations of country, region and city are valid, for example: country:region, region:city, country etc.

No special attributes

66.249.71.162

193.35.249.128

{geoip:Remote Host,type=country:region:city}

{geoip:Remote Host,type=country}

66.249.71.162 United States CA Mountain View

193.35.249.128 United Kingdom


PriorityA set of priorities that can appear in a record

 {priority,priority1;priority2...,priorityN}

Note: All optional priorities should appear as a semi-colon separated list inside the tag.

No special attributesfirst record = DEBUG,
second record = INFO,
third record = ERROR
{priority,DEBUG;INFO;ERROR} The matching priority of each record is displayed.
ExpressionThe expression that will be used according to the source columns given

 {expression}

No special attributes

Assuming you have two columns in your log, and you want to create a new column in your log view, represents a combining of these two records (highlighted in the example):

Mon Jul 10 04:33:51 2006 100 300

{date:Date,EEE MMM dd HH:mm:ss yyyy} {num:Num1} {num:Num2}{expression,Num2-Num1}

 

Mon Jul 10 04:33:51 2006 100 300 200

 Regular ExpressionA regular expression, used to extract part of the data from another column
read more about regular expressions in the regular expressions help page
 {regexp,refIndex=index | refName=column_name;columnType=date/timestamp/number;dateUIFormat=DISPLAY_DATE_FORMATmultiLine=true/false,(regular_expression_to_extract)}

refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column

columnType (mandatory for date/timestamp only):
columnType=timestamp;dateFormat=<the desired format of the date>
columnType=date;dateFormat=<the format of the date in the log to be extracted> 
dateUIFormat=<the format of the date desired to be displayed in XpoLog> 

 

multiLine (optional): indicates whether the record spreads over more than one line

expression (mandatory): the regular expression that will be extracted out of the value in the source column

 

See More Examples

Assuming you have the next record in your log, and you need to extract the error code (highlighted in the example) that is embedded in the message:

Mon Jul 10 04:33:51 2006 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K
ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K...
 {date:Date,EEE MMM dd HH:mm:ss yyyy}{regexp:Error Code,refName=Message;multiLine=true,(ORA-\d+)}{string:Message}Mon Jul 10 04:33:51 2006 ORA-336 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K
ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K...  
PropertiesA set of key-value properties that can appear in a record{properties,keysSep=[Keys_Separator];propSep=[Properties_Separator];,key1;key2;....;keyN}

keysSep (mandatory): the character/s that separate the key from the value

propSep (mandatory): the character/s that separate properties

Assuming you have the next record in your log:

2014-06-02 09:06:45,100 Type=INFO, Account=admin, ID=54, Team=all

{date:Date,yyyy-MM-dd HH:mm:ss,SSS} {properties: pairs,keysSep==;propSep=,;,Type;Account;ID;Team}2014-06-02 09:06:45,100 INFO admin 54 all Type=INFO Account=admin ID=54 Team=all
  JsonA set of key-value pairs{json,key1;key2;key3;....;keyN}No special attributes

Assuming you have the next record in your log:

2014-11-24 12:02:32,083 {"red":"#f00", "green":"#0f0", "blue":"#00f", "yellow":"#ff0","black":"#000"}

{date:Date,yyyy-MM-dd HH:mm:ss,SSS} {json,red;green;blue;yellow}2014-11-24 12:02:32,083 #f00 #0f0 #00f #ff0 {"black":"#000"}
 Term a constant string that appears in a record and needs to be displayed in the log view

{term,TERM}

TERM is the constant string

No special attributes Assuming you have the next two records in your log:

03/07/2005 03:44:56 app1 IP=192.168.11.44 success (where IP is a source IP)
03/07/2005 03:44:57 app2 IP=192.168.12.33 failure (where IP is a destination IP)
 Multi-pattern:
{date,dd/MM/yyyy HH:mm:ss} {term,app1} IP={text:Source IP} {string}
{date,dd/MM/yyyy HH:mm:ss} {term,app2} IP={text:Destination IP} {string}
 03/07/2005 03:44:56 app1 192.168.11.44 success
03/07/2005 03:44:57 app2 192.168.12.33 failure
 Free Text  any text you wish to display in the log view, usually used in multi-pattern logs to distinguish records

 {freetext,FREETEXT}

FREETEXT is the text that you wish to display in the log view.

No special attributes Assuming you have the next records in your log:

03/07/2005 03:44:56 $ success
03/07/2005 03:44:57 % failure

 Multi-pattern:

{date,dd/MM/yyyy HH:mm:ss} {freetext,Dollar}$ {string}
{date,dd/MM/yyyy HH:mm:ss} {freetext,Percentage}% {string}

Note: The freetext tag does not replace any text in the pattern, therefore it could be placed anywhere in the pattern and should not be separated with a space.

 03/07/2005 03:44:56 Dollar success
03/07/2005 03:44:57 Percentage failure
Value Mapping
Option I (manual mapping)
maps an original value from log to a converted value

{map,val1=convertedVal1;val2=convertedVal2;val3=convertedVal3;...;valN=convertedValN}

No special attributesAssuming you have in the log under one of the columns the following values:
200,304,404,500 

Using the following will convert the values as follows:

map:status,200=OK;304=Forbidden;404=Page
Not Found;500=Internal Error Server}

200 will be presented as 'OK'
304 will be presented as 'Forbidden'
404 will be presented as 'Page Not Found'
500 will be presented as 'Internal Error Server'
Value Mapping
Option II
(mapping based on an external file)
maps an original value from log to a converted value

{map,refIndex=ORIG_COL_INDEX,file:FULL_PATH_TO_FILE}

refIndex: numeric value which represents the index of the column in which its values will be converted (first column index is 0)

file:
Full path to the properties file that will be used to get the converted values 

Assuming you have in the log under one of the columns the following values:
200,304,404,500 

Using the following will convert the values as follows:

{map,refIndex=8,file:C:\XpoLogConfig\HTTP_VALUES.txt}

  • Index 8 is the index of the original column in the log which its values will be converted
  • C:\XpoLogConfig\HTTP_VALUES.txt is the file containing the following:
    200=OK
    304=Forbidden
    404=Page Not Found
    500=Internal Error Server
  • A new column will be created with the converted values
200 will be presented as 'OK'
304 will be presented as 'Forbidden'
404 will be presented as 'Page Not Found'
500 will be presented as 'Internal Error Server'
Value Mapping
Option III (regular expression manual mapping)
maps an original value from log to a converted value{regexp,refIndex=index | refName=column_name;columnType=map;mapping=val1:convertedVal1^val2:convertedVal2^...^valN:convertedValN,(regular_expression_to_extract)}

refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column

columnType (mandatory)
  

mapping (mandatory)

expression (mandatory): the regular expression that will be extracted out of the value in the source column

Assuming you have the next record in your log, and you need to map the Logon Type (highlighted in the example) that is embedded in the message:

1400504701000*;*Microsoft-Windows-Security-Auditing*;*A logon was attempted using explicit credentials.

    Security ID:        S-1-0-0

    Logon ID:        0xdc01

    Logon Type: 3

{timestamp:Date,MM/dd/yyyy HH:mm:ss}*;*{text:Source}*;*{regexp:Logon Type,refName=description;columnType=map;mapping=2:Local^3:Network,Logon Type: ([\d]+).*}{string:Description}




 05/19/2014 16:05:01 Microsoft-Windows-Security-Auditing Network A logon was attempted using explicit credentials. Security ID:        S-1-0-0 Logon ID:        0xdc01 Logon Type: 3
 Block an optional string that does not appear in all records {block,start,emptiness=true}XXX{block,end,emptiness=true} start: indicates a block start

end: indicates a block end

emptiness: indicates whether the block's data could be missing

 first record = 64.236.16.52:8080,
second record = 64.236.16.52,
third record = 64.236.16.52:7001
 {string}{block,start,emptiness=true}:{string}{block,end,emptiness=true} 
 Horizontal Tab a tab delimiter {tab} No special attributes one 123 {string}{tab}{number}first column = one,
second column = 123
 End of Line end of line, used in records that spread over more than one line

 {eol} No special attributesname : logger
level : db0 name :
 {string}{eol}level : {priority,db0;db1}first column = logger,
second column = db0
 End of Entity end of entity, used to mark the end of a record, improves the efficiency of the parsing process

 {eoe}

 No special attributesname : logger
level : db0
name : logger
level : db1 name :
 string}{eol}level : {priority,db0;db1}{eoe}

first row = logger,db0
second row = logger,db1

Optional Identifiers for Date Pattern

The following table provides examples of optional identifiers that can be used in a Date pattern.

IdentifierText in LogPattern
MM - numeric month

01-25-1986  

{date,MM-dd-yyyy}

MMMMM - full textual month25/July/1986{date,dd/MMMMM/yyyy}
 

MMM - textual month

25/Jul/1986{date,dd/MMM/yyyy}
dd - numeric day 01:25:1986{date,MM:dd:yyyy} 
EEEEE - full textual day  Friday 01-25-00{date,EEEEE MM-dd-yy}
EEE - textual day Fri 01-25-00 {date,EEE MM-dd-yy}
yy - 2 digit year  25/Jul/86 {date,dd/MMM/yy}
 yyyy - 4 digit year 25/Jul/1986 {date,dd/MMM/yyyy}
 HH - 24 hour 18:05:23  {date,HH:mm:ss}
hh - 12 hour 6:05:23 PM{date,hh:mm:ss}
 a - AM/PM marker 6:05:23 PM{date,hh:mm:ss a}
 mm - minute 18-05-23  {date,HH-mm-ss}
 ss - second 18:05:23 {date,HH:mm:ss}
 SSS - millisecond 18:05:23 253 {date,HH:mm:ss SSS}
 z - general time zone  18:05:23 EST {date,HH:mm:ss z}
 Z - RFC 822 time zone 18:05:23 -0400 {date,HH:mm:ss Z}
 X - ISO 8601 time zone 18:05:23 -04:00 {date,HH:mm:ss XXX}
 'TEXT' - a constant text that appears in the date string 07-1986D25 {date,MM-yyyy'D'dd}

 Attributes Supported by All Types

All pattern types support the attributes described in the following table.

AttributeRemarkExamples
NameThis attribute should always follow the tag name with a leading colon.

{string:Title}

{date:Start Date,dd/mm/yyyy}

{priority:Severity,DEBUG;INFO;ERROR}

{number:Status Code}

uiMessageLength  This attribute allows you to limit the length of data displayed in a specific column. If the data is longer than specified, it will be divided to several lines. {string:Title,uiMessageLength=20}
charsLength This attribute allows you to force the existence of a fixed number of characters in a string, even if there are less characters in the record. {string:Title,charsLength=10}

 Special Cases

  • The left curly bracket character ( { ) can be used as literal only if surrounded by a single quote '{' .
  • The quote character ( ' ) can be used as literal only if preceded by another quote ''.

 Examples of Patterns Used on Logs

 
The following are examples of patterns that can be used to tune the parsing results of logs. 

LogParsed RecordsData Pattern
Log 12003-02-12 12:37:26 ContextConfig[/examples]: Missing application web.xml, using defaults only
2003-02-12 12:37:26 StandardManager[/examples]: Seeding random number generator class java.security.SecureRandom
2003-02-12 12:37:30 StandardManager[/examples]: Seeding of random number generator has been completed
2003-02-12 12:37:30 StandardWrapper[/examples:default]: Loading container servlet default
2003-02-12 12:37:30 StandardWrapper[/examples:invoker]: Loading container servlet invoker
{date,yyyy-MM-dd HH:mm:ss} {string}[/{string}]: {string}
Log2127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET / HTTP/1.1" 200 1494
127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET /apache_pb.gif HTTP/1.1" 200 2326
127.0.0.1 - - [26/Dec/2001:19:52:48 +0200] "GET /examples/ HTTP/1.1" 404 277
127.0.0.1 - - [26/Dec/2001:19:54:37 +0200] "GET /examples/jsp/snp/snoop.jsp HTTP/1.1" 404 294
127.0.0.1 - - [28/Dec/2001:09:54:37 +0200] "GET /puga/main.html HTTP/1.1" 404 282
{string} - - [{date,dd/MMM/yyyy:HH:mm:ss Z} +0200] "{string}" {number} {number}
Log3[Wed Dec 26 19:52:48 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/examples/
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/_vti_bin/owssvr.dll
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/msoffice/cltreq.asp
[{string} {date,EEE MMM dd HH:mm:ss yyyy] [{priority,debug;info;warn;error;fatal}] [{string} {string}] {string}
 Log4 28/02/03 20:23:16 ERR Critical error on section 34 on module 5 [Channel 9] 4.4.4.4
28/02/03 20:25:35 DBG information arrived to fusion zone, restoring states [Channel 39] 4.8.4.9
28/02/03 20:33:22 WRN port collision seeking another [Channel 19] 4.4.4.4
28/02/03 20:33:22 FLW DB connection open structure initiated [Channel 9] 4.23.12.5
{date,dd/MM/yy HH:mm:ss} {priority,DBG;FLW;WRN;ERR} {string} [{string}] {string} 
 Log5 5 d MBGN Talk to port
9 f MLPT1 Port open
0 x MCOM Com port open
 {number}{tab}{string}{tab}{string}{tab}{string}
 Log65 d MBGN procId=123 Talk to port
9 f MLPT1 Port open
0 x MCOM procId=456 Com port open
 {number}{tab}{string}{tab}{block,start,emptiness=true}procId={string}{block,end,emptiness=true}{string}{tab}{string} 

 

 

 

 



 

  • No labels