Current view: XpoLog V6. Available: XpoLog V5 and XpoLog V7 (Latest)

Skip to end of metadata
Go to start of metadata

Background

Apache HTTPD Log Analysis App is an Http server for Windows and Unix machines that automatically Collect - Read - Parse - Analyzes - Reports all machine generated log data of the server and presents a comprehensive automatic predefined set of Reports, Dashboards and Gadgets. Once you Setup and configure the Apache HTTPD App, you will be redirected to the dashboards where yuo will have graphs about: errors occured, geographic data of users and requests, Browsers related analytics, Pages and hits analysis, resources and many statistics about your servers' performance. You later use XpoLog built in Analytics features to zero in on errors and take actions to improve your system's uptime. Apache HTTP server logs data can be viewed, filtered and searched via the main XpoLog console.

Steps

 

  1. Add Log Data In XpoLog, When adding a log to XpoLog you can now select the Log Type (logtype) for Apache Httpd the are the following logtypes:
    1. httpd
    2. w3c
    3. webserver
      i. in addition select not only httpd but also the log type - access or error
  2. Once all required information is set click next and edit the log pattern, this step is crucial to the accuracy and deployment of the Apache Httpd App. Use the following conversion table to build the XpoLog pattern out of the access log format.

 

Example

In the Apache Httpd configuration file, usually httpd.conf by default, located under the conf/ directory (Linux "/etc/httpd/conf") search for the LogFormat directive:

LogFormat "%h %l %u %t \"%r\" %>s %b" common

In XpoLog such pattern will be translated into:

{text:RemoteHost,ftype=remotehost;,} {text:RemoteLog,ftype=remotelog;,} {text:User,ftype=remoteuser;,} [{date:Date,locale=en;,dd/MMM/yyyy:HH:mm:ss z}] "{choice:Method,ftype=reqmethod;,GET;POST} {string:URL,ftype=requrl;,}{block,start,emptiness=true}?{string:Query,ftype=querystring;,}{block,end,emptiness=true} {string:reqprotocol,ftype=reqprotocol;,}" {number:ResponseStatus,ftype=respstatus;,} {text:Bytes Sent,ftype=bytesent;,}{eoe}

 

for more information see below:

Apache Https Access Log Format Conversion Table

logtypes should be set to: httpd,w3c,webserver,access

Format StringDescriptionXpoLog PatternXpoLog ftype

%a

Remote IP-address of the request

{text:RemoteIP,ftype=remoteip}

remoteip
%{c}aUnderlying peer IP address of the connection {text:RemoteIP,ftype=remoteip}remoteip

%A

Local IP-address

{ip:LocalIP,ftype=localip}

localip

%B

Size of response in bytes, excluding HTTP headers.

{number:BytesSent,ftype=bytesent}

bytesent

%b

Size of response in bytes, excluding HTTP headers. In CLF format, i.e. a '-' rather than a 0 when no bytes are sent

{text:BytesSent,ftype=bytesent}

bytesent

%{FOOBAR}C

The contents of cookie FOOBAR in the request sent to the server. Only version 0 cookies are fully supported.

{string:Cookie_< FOOBAR >}

Replace < FOOBAR > with cookie name

 

%D

The time taken to serve the request, in microseconds.

{number:ResponseTimeMilliSecs}

 

%{FOOBAR}e

The contents of the environment variable FOOBAR

{string:EnvVariable_< FOOBAR >}

Replace < FOOBAR > with variable name

 

%f

Filename

{text:FileName}

 

 

%h

Remote host

{text:Remotehost,ftype=remotehost}

 

remotehost

%H

The request protocol

{text:RequestProtocol,ftype=reqprotocol}

 

reqprotocol

%{FOOBAR}i

The contents of FOOBAR: header line(s) in the request sent to the server. Changes made by other modules (e.g. mod_headers) affect this.

If you're interested in what the request header was prior to when most modules would have modified it, use mod_setenvif to copy the header into an internal environment variable and log that value with the %{FOOBAR}e described above.

{text:<FOOBAR>}

https://en.wikipedia.org/wiki/List_of_HTTP_header_fields and so on it goes for the different headers.

 

 

%{Referer}i

 

{text:Referer,ftype=referer}

 

referer

%{User-agent}i

 

{text:User-agent,ftype=useragent}

 

useragent
%{X-Forwarded-For}i 

{text:X-Forwarded-For,ftype=forwardforip} OR

{ip:X-Forwarded-For,ftype=forwardforip}

 

forwardforip

%k

Number of keepalive requests handled on this connection. Interesting if KeepAlive is being used, so that, for example, a '1'

means the first keepalive request after the initial one, '2' the second, etc...; otherwise this is always 0 (indicating the initial request).

{number:KeepAlive}

 

%l

Remote logname (from identd, if supplied). This will return a dash unless mod_ident is present and IdentityCheck is set ON.

{text:RemoteLog,ftype=remotelog}

 remotelog
%LThe request log ID from the error log (or '-' if nothing has been logged to the error log for this request). Look for the matching error log line to see what request caused what error{text:logId,ftype=logid}logid

%m

The request method

{text:RequestMethod,ftype=reqmethod}

reqmethod

%{FOOBAR}n

The contents of note FOOBAR from another module.

{string:<FOOBAR>}

 

 

%{FOOBAR}o

The contents of FOOBAR: header line(s) in the reply.

{string:<FOOBAR>}

 

 

%p

The canonical port of the server serving the request

{number:ServerPort,ftype=serverport}

serverport

%{format}p

The canonical port of the server serving the request or the server's actual port or the client's actual port. Valid formats are canonicallocal, or remote.

%{canonical}p

%{local}p

%{remote}p

{number:ServerPort,ftype=serverport}

{number:LocalServerPort,ftype=localserverport}

{number:RemotePort,ftype=remoteport}

serverport

localserverportt

remoteport

%P

The process ID of the child that serviced the request.

{text:ProcessID,ftype=processid}

processid

%{format}P

The process ID or thread id of the child that serviced the request. Valid formats are pidtid, and hextidhextid requires APR 1.2.0 or higher.

{text:ProcessID,ftype=processid}

Valid formats are pidtid, and hextid

processid

%q

The query string (prepended with a ? if a query string exists, otherwise an empty string)

{text:QueryString,ftype=querystring}

OR

Suggest a regexp that will build a list of parameters as cloumns.

The query string (prepended with a ? if a query string exists, otherwise an empty string)

querystring

%r

First line of request

  1. {choice:Method,ftype=reqmethod;,GET;POST}
  2. {string:URL,ftype=requrl}
  3. {string:Query,ftype=querystring} – Optional
  4. {string:reqprotocol,ftype=reqprotocol}

reqmethod

requrl

querystring

reqprotocol

%R

The handler generating the response (if any).

{text:ResponseHandler}

 

 

%s

Status. For requests that got internally redirected, this is the status of the *original* request --- %>s for the last.

{number:ResponseStatus,ftype=respstatus}

. For requests that got internally redirected, this is the status of the *original* request --- %>s for the last.

respstatus

%t

Time the request was received (standard english format)

{date:Date,locale=en,dd/MMM/yyyy:HH:mm:ss z}

 

 
%{format}t

The time, in the form given by format, which should be in an extended strftime(3) format (potentially localized). If the format starts with begin: (default)

the time is taken at the beginning of the request processing. If it starts with end: it is the time when the log entry gets written, close to the end of the request

processing. In addition to the formats supported by strftime(3), the following format tokens are supported:

sec

number of seconds since the Epoch

msec

number of milliseconds since the Epoch

usec

number of microseconds since the Epoch

msec_frac

millisecond fraction

usec_frac

microsecond fraction

These tokens can not be combined with each other or strftime(3) formatting in the same format string. You can use multiple %{format}t tokens instead.

The extended strftime(3) tokens are available in 2.4.13 and later.

{date:Date,locale=en,dd/MMM/yyyy:HH:mm:ss z}

sec number of seconds since the Epoch
msec number of milliseconds since the Epoch
usec number of microseconds since the Epoch
msec_frac millisecond fraction
usec_frac microsecond fraction

 

%T

The time taken to serve the request, in seconds.

{number:ResponseTimeSecs,,ftype=processrequestmilli}

 

processrequestmilli

%{UNIT}T

The time taken to serve the request, in a time unit given by UNIT. Valid units are ms for milliseconds, us for microseconds, and s for seconds. Using s gives the

same result as %T without any format; using us gives the same result as %D. Combining %T with a unit is available in 2.4.13 and later.

  1. {number:ResponseTimeMilliSecs,ftype=processrequestmilli}
  2. {number:ResponseTimeMicroSecs,ftype=processrequestmicrosecs}
  3. {number:ResponseTimeSecs,ftype=processrequestsecs}

processrequestmilli

processrequestmicrosecs

processrequestsecs

%u

Remote user (from auth; may be bogus if return status (%s) is 401)

{text:User,ftype=remoteuser}

Remote user (from auth; may be bogus if return status (%s) is 401)

remoteuser

%U

The URL path requested, not including any query string

{text:RequestURL,ftype=requrl}

The URL path requested, not including any query string.

requrl

%v

The canonical ServerName of the server serving the request

{text:ServerName,ftype=servername}

servername

%V

The server name according to the UseCanonicalName setting

{text:ServerName,ftype=servername}

The server name according to the UseCanonicalName setting.

servername

%X

Connection status when response is completed:

X =

connection aborted before the response completed.

+ =

connection may be kept alive after the response is sent.

- =

connection will be closed after the response is sent.

(This directive was %c in late versions of Apache 1.3, but this conflicted with the historical ssl %{var}c syntax.)

{text:ConnectionStatus}

Connection status when response is completed:

X =

connection aborted before the response completed.

+ =

connection may be kept alive after the response is sent.

- =

connection will be closed after the response is sent.

(This directive was %c in late versions of Apache 1.3, but this conflicted with the historical ssl %{var}c syntax.)

 
%IBytes received, including request and headers. Cannot be zero. You need to enable mod_logio to use this.

{number:TotalBytesWHeadersReceived,ftype=reqbyteswheaders}

(with headers)

reqbyteswheaders
%OBytes sent, including headers. May be zero in rare cases such as when a request is aborted before a response is sent. You need to enable mod_logio to use this.

{number:TotalBytesWHeadersSent,ftype=respbyteswheaders}

(with headers – can help compute header size)

respbyteswheaders
%SBytes transferred (received and sent), including request and headers, cannot be zero. This is the combination of %I and %O. You need to enable mod_logio to use this.

{number:TotalBytesWHeadersReceived,ftype=reqbyteswheaders}

{number:TotalBytesWHeadersSent,ftype=respbyteswheaders}

reqbyteswheaders

respbyteswheaders

%{FOOBAR}^ti

The contents of FOOBAR: trailer line(s) in the request sent to the server.

{text:Req_<FOOBAR>}

The content of FOOBAR: trailer line(s) in the request sent to the server.

 

%{FOOBAR}^to

The contents of FOOBAR: trailer line(s) in the response sent from the server.

{text:Resp_<FOOBAR>}

The contents of FOOBAR: trailer line(s) in the response sent from the server.

 



Error Log

In the Apache Httpd configuration file, usually httpd.conf by default, located under the conf/ directory (Linux "/etc/httpd/conf") search for the LogFormat directive:

ErrorLogFormat "[%{u}t] [%m:%l] [pid %P:tid %T] %F: %E: %M"

In XpoLog such pattern will be translated into:

[{date:Date,locale=en,EEE MMM dd HH:mm:ss.SSSSSS yyyy}] [{text:Module}:{priority:Level,ftype=status;,}] [pid {text:ProcessID,ftype=processid;,}:tid {text:ThreadId,ftype=threadid;,}]{text:ErrorCode,ftype=errorcode;,}:{block,start,emptiness=true} {text:SourceFileName}:{block,end,emptiness=true} {string:Message,ftype=Message;,}

for more information see below:

Apache Https Error Log Format Conversion Table

logtypes should be set to: httpd,w3c,webserver,error

Format StringDescriptionXpoLog PatternXpoLog ftype
%aClient IP address{ip:ClientIP,ftype=remoteip}remoteip
 The port making the request{text:ClientPort,ftype=port}port
%{c}aUnderlying peer IP address and port of the connection (see the mod_remoteip module)T.B.D 

%A

Local IP-address

{ip:LocalIP,ftype=localip}

localip
 The local port {text:LocalPort,ftype=port} localport
%{name}eRequest environment variable nameT.B.D 
%EAPR/OS error status code and string{text:ErrorCode,ftype=errorcode} errorcode
%FSource file name and line number of the log call{text:SourceFileName} 
%{name}iRequest header nameT.B.D 
%kNumber of keep-alive requests on this connection{number:KeepAlive} 
%lThe level of the message{priority:Level,emerg;alert;crit;error;warn;notice;info;debug;trace1;trace2;trace3;trace4;trace5;trace6;trace7;trace8;ftype=status}status
 Operation T.B.D{text:Operation} 
%LLog ID of the request{text:LogId,ftype=logid}logid
%{c}LLog ID of the connectionT.B.D 
%{C}LLog ID of the connection if used in connection scope, empty otherwiseT.B.D 
%mName of the module logging the message{text:Module} 
%MThe actual error message{string:Message,ftype=Message}Message
%{name}nRequest note nameT.B.D 
%PProcess ID of current process {text:pid,ftype=pid} pid
%TThread ID of current thread {text:ThreadID,ftype=threadid}threadid
%{g}TSystem unique thread ID of current thread (the same ID as displayed by e.g. top; currently Linux only) {text:ThreadId,ftype=threadid}threadid
%tDate{date:Date,locale=en,EEE MMM dd HH:mm:ss.SSSSSS yyyy} 
%{u}tThe current time including micro-seconds{date:Date,locale=en,EEE MMM dd HH:mm:ss.SSSSSS yyyy} 
%{cu}tThe current time in compact ISO 8601 format, including micro-seconds{date:Date,locale=en,EEE MMM dd HH:mm:ss.SSSSSS yyyy} 
%vThe canonical ServerName of the current server.

{text:ServerName,ftype=servername}


servername
%VThe server name of the server serving the request according to the UseCanonicalName setting.

{text:ServerName,ftype=servername}

The server name according to the UseCanonicalName setting.

servername
    



 

  • No labels