Current view: XpoLog V6. Available: XpoLog V5 and XpoLog V7 (Latest)

Skip to end of metadata
Go to start of metadata

Background

The NGINX server logs analysis App automatically Collect - Read - Parse - Analyzes - Reports all machine's generated log data of the server and presents a comprehensive set of graphs and reports to analyze machine generated data. Use a predefined set of dashboards and gadgets to visualize and address the system software, code written, and infrastructure during development, testing, and production. This NGINX server logs analysis App helps you measure, troubleshoot, and optimize your servers integrity, stability and quality with visualization and investigation dashboards.

Steps

  1. Add Log Data In XpoLog, When adding a log to XpoLog you can now select the Log Type (logtype) for NGINX with e the following logtypes:
    1. nginx
    2. w3c
    3. webserver
      1. In addition select not only httpd but also the log type - access or error
      2. See error log definition at the bottom of this page 
  2. Once all required information is set click next and edit the log pattern, this step is crucial to the accuracy and deployment of the Analytic App. Use the following conversion table to build the XpoLog pattern.

 

Example

In the NGINX configuration file, usually nginx.conf by default, located under the conf/ directory (Linux "NGINX ROOT DIR/conf/nginx.conf") search for the ______ directive:

Information from NGINX site:

"NGINX writes information about client requests in the access log right after the request is processed. By default, the access log is located at logs/access.log, and the information is written to the log in the predefined combined format. To override the default setting, use the log_format directive to change the format of logged messages, as well as the access_log directive to specify the location of the log and its format. The log format is defined using variables.

The following examples define the log format that extends the predefined combined format with the value indicating the ratio of gzip compression of the response. The format is then applied to a virtual server that enables compression.

access_log path [format [buffer=size] [gzip[=level]] [flush=time] [if=condition]];

access_log off;
Default:
access_log logs/access.log combined;

log_format main             '$remote_addr - $remote_user [$time_local] "$request" '
                                       '$status $body_bytes_sent "$http_referer" '
                                       '"$http_user_agent" "$http_x_forwarded_for"';

 

In XpoLog such pattern will be translated into:

{geoip:Remote IP,ftype=remoteip} - {text:User,ftype=remoteuser} [{date:Date,dd/MMM/yyyy:HH:mm:ss z}] "{text:RequestMethod,ftype=reqmethod} {text:RequestURL,ftype=requrl} {text:RequestProtocol,ftype=reqprotocol}" {text:Response Status,ftype=respstatus} {text:Bytes Sent,ftype=bytesent} "{text:Referer,ftype=referer}" "{text:User Agent,ftype=useragent}" "{string:HTTP_x_forwarded_for}"{eoe}

for more information see below:

 

Apache Https Access Log Format Conversion Table

logtyep should be set to: nginx, access

 

FieldAppears asDescriptionXpoLog PatternXpoLog ftype

$arg_name

 

argument name in the request line

 

 

$args

 

arguments in the request line

{text:QueryString,ftype=querystring}

querystring

$binary_remote_addr

 

 

client address in a binary form, value’s length is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses

 

 

 

$body_bytes_sent

 

 

number of bytes sent to a client, not counting the response header; this variable is compatible with the “%B” parameter of the mod_log_config Apache module

{number:BytesSent,ftype=bytesent}

bytesent

$bytes_sent

 

number of bytes sent to a client (1.3.8, 1.2.5)

 

{number:TotalBytesWHeadersSent,ftype=respbyteswheaders}

 

respbyteswheaders

$connection

 

 

connection serial number (1.3.8, 1.2.5)

 

 

 

$connection_requests

 

 

current number of requests made through a connection (1.3.8, 1.2.5)

 

 

 

$content_length

 

 

“Content-Length” request header field

 

 

 

$content_type

 

 

“Content-Type” request header field

 

 

 

$cookie_name

 

 

the name cookie

 

{string:CookieName}

 

$document_root

 

 

root or alias directive’s value for the current request

 

 

 

$document_uri

 

 

same as $uri

 

{text:RequestURL,ftype=requrl}

requrl

$host

 

 

in this order of precedence: host name from the request line, or host name from the “Host” request header field, or the server name matching a request

 


{text:ServerName,ftype=servername}

 

servername

$hostname

 

host name

 

{text:Remotehost,ftype=remotehost}

remotehost

$http_name

 

arbitrary request header field; the last part of a variable name is the field name converted to lower case with dashes replaced by underscores

 

{text:<name>_,ftype=<NAME>}

<NAME>

$https

 

 

“on” if connection operates in SSL mode, or an empty string otherwise

 

 

 

$is_args

 

 

“?” if a request line has arguments, or an empty string otherwise

 

 

 

$limit_rate

 

 

setting this variable enables response rate limiting; see limit_rate

 

 

 

$msec

 

 

current time in seconds with the milliseconds resolution (1.3.9, 1.2.6)

 

 

 

$nginx_version

 

 

nginx version

 

 

 

$pid

 

 

PID of the worker process

 

{text:ProcessID,ftype= processid}

processid

$pipe

 

 

“p” if request was pipelined, “.” otherwise (1.3.12, 1.2.7)

 

 

 

$proxy_protocol_addr

 

 

client address from the PROXY protocol header, or an empty string otherwise (1.5.12)

The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listendirective.

 

 

{ip: X-Forwarded-For,ftype=forwardforip}

forwardforip

$proxy_protocol_port

 

 

client port from the PROXY protocol header, or an empty string otherwise (1.11.0)

The PROXY protocol must be previously enabled by setting the proxy_protocol parameter in the listendirective.

 

 

 

$query_string

 

 

same as $args

 

{text:QueryString,ftype=querystring}

querystring

$realpath_root

 

 

an absolute pathname corresponding to the root or alias directive’s value for the current request, with all symbolic links resolved to real paths

 

 

 

$remote_addr

 

 

client address

 

{ip:RemoteIP,ftype=remoteip}

remoteip

$remote_port

 

 

client port

 

{number:RemotePort,ftype=remoteport}

remoteport

$remote_user

 

 

user name supplied with the Basic authentication

 

{text:User,ftype=remoteuser}

remoteuser

$request

 

 

full original request line

 

{text:RequestMethod,ftype=reqmethod} {text:RequestURL,ftype=requrl} {text:RequestProtocol,ftype=reqprotocol}

 

Reqmethod

Requrl

reqprotocol

$request_body

 

 

request body

The variable’s value is made available in locations processed by the proxy_passfastcgi_pass,uwsgi_pass, and scgi_pass directives when the request body was read to a memory buffer.

 

 

$request_body_file

 

 

name of a temporary file with the request body

At the end of processing, the file needs to be removed. To always write the request body to a file,client_body_in_file_only needs to be enabled. When the name of a temporary file is passed in a proxied request or in a request to a FastCGI/uwsgi/SCGI server, passing the request body should be disabled by the proxy_pass_request_body offfastcgi_pass_request_body offuwsgi_pass_request_body off, orscgi_pass_request_body off directives, respectively.

 

 

 

$request_completion

 

 

“OK” if a request has completed, or an empty string otherwise

 

 

 

$request_filename

 

 

file path for the current request, based on the root or alias directives, and the request URI

 

 

 

$request_id

 

 

unique request identifier generated from 16 random bytes, in hexadecimal (1.11.0)

 

 

 

$request_length

 

 

request length (including request line, header, and request body) (1.3.12, 1.2.7)

 

 

 

$request_method

 

 

request method, usually “GET” or “POST”

 

{text:RequestMethod,ftype=reqmethod}

reqmethod

$request_time

 

 

request processing time in seconds with a milliseconds resolution (1.3.9, 1.2.6); time elapsed since the first bytes were read from the client

 

 

$request_uri

 

 

full original request URI (with arguments)

 

{text:RequestURL,ftype=requrl}

 

requrl

$scheme

 

 

request scheme, “http” or “https”

 

{text:RequestProtocol,ftype=reqprotocol}

reqprotocol

$sent_http_name

 

 

arbitrary response header field; the last part of a variable name is the field name converted to lower case with dashes replaced by underscores

 

 

 

$server_addr

 

 

an address of the server which accepted a request

Computing a value of this variable usually requires one system call. To avoid a system call, the listendirectives must specify addresses and use the bind parameter.

 


{ip:LocalIP,ftype=localip}

 

localip

$server_name

 

 

name of the server which accepted a request

 

{text:ServerName,ftype=servername}

 

servername

$server_port

 

 

port of the server which accepted a request

 

{number:ServerPort,ftype=serverport}

serverport

$server_protocol

 

 

request protocol, usually “HTTP/1.0”, “HTTP/1.1”, or “HTTP/2.0

 

{text:RequestProtocol,ftype=reqprotocol}

reqprotocol

$status

 

 

response status (1.3.2, 1.2.2)

 

{number:ResponseStatus,ftype=respstatus}

respstatus

$tcpinfo_rtt, $tcpinfo_rttvar, $tcpinfo_snd_cwnd, $tcpinfo_rcv_space

 

 

information about the client TCP connection; available on systems that support the TCP_INFO socket option

 

 

 

$time_iso8601

 

 

local time in the ISO 8601 standard format (1.3.12, 1.2.7)

 

 

$time_local

 

 

local time in the Common Log Format (1.3.12, 1.2.7)

{date:Date,dd/MMM/yyyy:HH:mm:ss z}

 

$uri

 

 

current URI in request, normalized

The value of $uri may change during request processing, e.g. when doing internal redirects, or when using index files.

{text:RequestURL,ftype=requrl}

 

requrl

$http_user_agent

 

 

 

{text:User-agent,ftype=useragent}

useragent

$http_referer
 

 

 

{text:Referer,ftype=referer}

referer

$http_x_forwarded_for  {text:Forwarder} 

 

Error Log

 

Look for the error_log logs/error.log warn;  directive the nginx configuration file.

YYYY/MM/DD HH:MM:SS [LEVEL] PID#TID: *CID MESSAGE

With PID and TID being the logging process and thread id and CID a number identifying a (probably proxied) connection, probably a counter. The *CID part is optional.

debug, info, notice,warn, error, crit, alert, or emerg.

 

Default XpoLog Pattern:

{date:Date,yyyy/MM/dd HH:mm:ss} [{priority:Level,ftype=severity,debug;info;notice;warn;error;crit;alert;emerg}] {text:PID,ftype=processid}#{text:TID,ftype=threadid}:{block,start,emptiness=true}{text:CID,ftype=connectionid} {block,end,emptiness=true}{string:Message,ftype=message}

 

  • No labels