WASD Hypertext Services - Technical Overview

The WATCH facility is a powerful adjunct in server administration.  From the administration menu it provides an online, real-time, in-browser-window view of request processing in the running server.  The ability to observe live request processing on an ad hoc basis, without changing server configuration or shutting-down/restarting the server process, makes this facility a great configuration and problem resolution tool.  It allows (amongst other uses)

• assessment of mapping rules
• assessment of authorization rules
• investigation of request processing problems
• observation of script interaction
• general observation of server behaviour

A single client per server process can access the WATCH facility at any one time.  It can be used in one of two modes. 

• As a one-shot, one-off WATCH of a particular request.  This is available from the Request Report page of the Administration Menu.  In this case the single indicated request is tagged to be WATCHed in all categories (see below) for the duration of the request (or until the client stops WATCHing). 

• As described in the following chapter the server and all new requests being processed are candidates for being WATCHed.  Categories are selected before initiating the WATCH and the report can be generated for a user-specified number of seconds or aborted at any time using the browser's stop button. 

An event is considered any significant point for which the server code has a reporting call provided.  These have been selected to provide maximum information with minimum clutter and impact on server performance.  Obvious examples are connection acceptance and closure, request path resolution, error report generation, network reads and writes, etc.  Events are collected together into groupings to allow clearly defined areas of interest to be selected for reporting. 

  WATCH Selection Graphic

The report menu provides for the inclusion of any combination of the following categories. 

Request

• Processing - Each major step in a request's progress.  For example, path resolution and final response status. 

• Header - Provides the HTTP request header as a section of blank-line terminated text. 

• Body - In the case of HTTP POST or PUT methods any request body is displayed.  It will generally be URL-encoded but otherwise plain-text. 

• Processing - Each major step in generating a response to the request.  These generally reflect calls to a major server module such as file CACHE, FILE access, INDEX-OF, SSI processing, etc.  One or more of these events may occur for each request.  For instance a directory listing will show an INDEX-OF call and then usually a FILE call as any read-me file is accessed. 

• Header - The blank-line terminated HTTP header to the response.  Only server-generated headers are included.  Scripts that provide a full HTTP stream do not have the header explicitly reported.  The response body category must be enabled to observe these (indicated by a STREAM notation). 

• Body - The content of the response.  This is provided as a hexadecimal dump on the left and with printable characters rendered on the right, 32 bytes per line.  Some requests also generate very large responses which will clutter output.  Generally this category would be used when investigating specific request response body problems. 

• Connection - Each TCP/IP connection acceptance and closure.  The connect shows which service the request is using (scheme, host name and port). 

• Path Mapping - This, along with the authorization report, provides one of the most useful aspects of the WATCH facility.  It comprises an event line indicating the path to be mapped (it can also show a VMS file specification if a reverse-mapping has been requested).  Then as each rule is processed a summary showing current path, match "Y"/"N" for each path template and any conditional, then the result and conditional.  Finally an event entry shows the resulting path, VMS file specification, any script name and specification resolved.  The path mapping category allows the administrator to directly assess mapping rule processing with live or generated traffic. 

• Authorization - When authorization is deployed this category shows the rules examined to determine if a path is controlled, any authentication events in assessing username and password, and the consequent group, user and request capabilities (read and/or write) for that path.  No password information is displayed. 

• Error - The essential elements of a request error report are displayed.  This may include a VMS status value and associated system message. 

• CGI - This category displays the generated CGI variable names and values as used by subprocess and DECnet scripting, and by SSI documents. 

• DCL - Debugging scripts can sometimes present particular difficulties.  This category may help.  It reports on all input/output streams with the subprocess (SYS$INPUT, SYS$OUTPUT, SYS$COMMAND, CGIPLUSIN). 

• DECnet - For the same reason as above this category reports all DECnet scripting input/output of the DECnet link.  In particular, it allows the observation of the OSU scripting protocol. 

• Activity - For each raw network read and write the VMS status code and size of the I/O is recorded. 

• Data - For each raw network read or write the contents are provided as a hexadecimal dump on the left and with printable characters rendered on the right, 32 bytes per line. 

• Timer - Timers control the maximum periods each stage in request processing may procede before the server aborts processing.  Input, output and keep-alive timers are reported. 

• Logging - Access logging events include log open, close and flush, as well as request entries. 

• SSL - If the Secure Sockets Layer image is in use this category provides a indication of high-level activity. 

• Quotas - Display the server process quotas with every WATCH event reported.  This can add a lot of clutter to a report but may be useful when tracking down the cause of resource exhaustion (of course there is a "Catch 22" here - will WATCH be able to report the quotas if resources have been exhausted!)

• Processing - Each major step during the serving of a proxied request. 

• Request Header - The proxy server rebuilds the request originally received from the client.  This category shows that rebuilt request, the one that is sent to the remote server. 

• Request Body - In the case of HTTP POST or PUT methods any request body is displayed.  This is provided as a hexadecimal dump on the left and with printable characters rendered on the right, 32 bytes per line. 

• Response Header - The blank-line terminated HTTP header to the response from the remote, proxied server. 

• Response Body - The content of the response sent from the remote server.  This is provided as a hexadecimal dump on the left and with printable characters rendered on the right, 32 bytes per line. 

• Cache - When proxy caching is enabled this category provides information on cache reading (serving a request from cache) and cache loading (writing a cache file using the response from a remote server).  It will provide a reason for any request or response it does not cache, as well as report errors during file processing. 

• Cache Maintenance - This category is not related to request processing.  It allows routine and reactive cache purging activities to be watched. 

By default all requests to all services are WATCHed.  Fine control may be exercised over exactly which requests are reported, allowing only a selected portion of all requests being processed to be concentrated on, even on a live and busy server.  This is done by filtering requests according the following criteria. 

• Client - The originating host name or address.  Unless server DNS host name resolution is enabled this must be expressed in dotted-decimal notation. 

• Service - The service connected to.  This includes the scheme of the service (i.e. "http:", "https:"), the host name (real or virtual), and the port.  The host name is the official name of the service as reported during server startup.  As the port number is a essential part of the service specification it must always be explicitly supplied or wildcarded. 

• Path/Track - Either, the request path, or a specific track identifier string.  A path may be specified with a leading "/" for local paths or if WATCHing proxy requests with a full, or part of a full, URL.  To WATCH requests associated with a particular access track (see 6.5.5 - Access Tracking) enter the track's unique identifier string preceded by a dollar symbol (e.g. "$ORoKJAOef8sAAAkuACc"). 

These filters are controlled using fully-specified or wildcarded strings.  Requests that do not match the filter are not reported.  In the case of originating client and destination service, requests are eliminated before ever appearing in the report.  Path and track filtering is slightly different, requiring some request processing before the target can be determined.  Depending on the categories selected this may result in some events begin displayed.  It will be eliminated, with an accompanying explanatory message, as soon the path or track identifier has been determined. 

The following examples are grouped in the same order as the categories listed above; client, service and path. 

  alpha.wasd.dsto.defence.gov.au
  *.wasd.dsto.gov.au  
  131.185.250.202
  131.185.250.*
 
  beta.wasd.dsto.defence.gov.au:8000
  beta.wasd.dsto.defence.gov.au:*
  http://*
  https:*
  *:80
 
  /ht_root/src/*
  /cgi-bin/*
  /web/*/cyrillic/*
  $ORoKJAOef8sAAAkuACc
  http://proxied.host.name/*

The following example illustrates the format of the WATCH report.  It begins with multi-line heading.  The first two record the date, time and official server name, with underline.  The third provides the WASD server version.  The fourth provides some TCP/IP agent information.  Lines following can show OpenSSL version (if deployed), system infomation, server startup command-line, and then current server process quotas.  The last three lines of the header provide a list of the categories being recorded, the filters in use, and the last, column headings described as follows:

• time the event was recorded
• the module name of the originating source code
• the line in the code module
• a unique item number for each thread being WATCHed
• event category name
• free-form, but generally interpretable event data

  WATCH Report Graphic

Note that some items also include a block of data.  The request header category does this, providing the blank-line terminated text comprising the HTTP header.  Rule mapping also provides a block of information representing each rule as it is interpreted.  Generally WATCH-generated information can be distinguished from other data by it's uniform format and delimiting vertical bars.  Initiative and imagination is sometimes required to interpret the free-form data but a basic understanding of HTTP serving and a little consideration is generally all that is required to deduce the essentials of any report. 

  31-MAY-2000 18:20:25  WATCH REPORT  delta.wasd.dsto.defence.gov.au:80
  ---------------------------------------------------------------------
  HTTPd-WASD/7.0.0 OpenVMS/AXP SSL (30-MAY-2000 22:56:54.37 VMS 7.2-1 DECC 60290003)
  Compaq-TCP/IP TCPIP$IPC_SHR V5.0A-1 (20-MAY-1999 22:31:52.08)
  OpenSSL 0.9.5 28 Feb 2000 (5-MAR-2000 14:58:56.83)
  AlphaServer DS20 500 MHz VMS V7.2-1 (ODS-5 enabled, VMS NAML, VMS FIB)
  $ HTTPD /PRIORITY=4 /SYSUAF=(ID)
  AST:1995/2000 BIO:509/512 BYT:479200/499296 DIO:512/512 ENQ:504/512 FIL:295/300 PGFL:192992/200000 PRC:0/100 TQ:98/100
  BYTLM-available:495008 BYTLM-per-subproc:14592 (approx 26 subprocesses) BYTLM-net-accept:1024 BYTLM-net-listen:1024
  Watching: connect, request, response (19)
  Client: "*" Service: "*" Path: "*"
  |Time_______|Module__|Line|Item|Category|Event...|
  |18:20:46.91 NET      1626 0001 CONNECT  ACCEPT 131.185.250.108,47111 on https://131.185.250.203:443|
  |18:20:47.00 REQUEST  1211 0001 REQUEST  HEADER 285 bytes|
  GET /ht_root/ HTTP/1.0
  If-Modified-Since: Thursday, 07-Oct-99 03:51:23 GMT; length=4644
  Connection: Keep-Alive
  User-Agent: Mozilla/3.0Gold (X11; I; OSF1 V4.0 alpha)
  Pragma: no-cache
  Host: delta.wasd.dsto.defence.gov.au
  Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
 
  |18:20:47.00 NET      1148 0001 CONNECT  VIRTUAL delta.wasd.dsto.defence.gov.au:443|
  |18:20:47.00 REQUEST  2270 0001 REQUEST  GET /ht_root/|
  |18:20:47.00 MAPURL   0312 0001 MAPPING  PATH /ht_root/|
  /ht_root/  ..  REDIRECT  /*.*.htmlx  /*.htmlx?httpd=ssi&__part=*
  /ht_root/  ..  PASS      /httpd/-/change/*  /httpd/-/change/*
  /ht_root/  ..  MAP       /htroot/*  /ht_root/*
  /ht_root/  ..  SET       /web/*   stmLF
  /ht_root/  ..  SET       /ht_root/*   stmLF
  /ht_root/  ..  MAP       /httpd-internal-icons/*  /httpd/-/*
  /ht_root/  ..  PASS      /ht_root/runtime/*  /ht_root/runtime/*
  /ht_root/  ..  PASS      /*/-/*  /ht_root/runtime/*/*
  /ht_root/  Y-  PASS      /ht_root/*  
  |18:20:47.00 MAPURL   0335 0001 MAPPING  RESULT /ht_root/ /ht_root/ HT_ROOT:[000000] - -|
  |18:20:47.00 CACHE    0604 0001 RESPONSE CACHE DSA811:[HT_ROOT.][000000]HOME.HTML|
  |18:20:47.00 NET      2427 0001 RESPONSE HEADER 231 bytes|
  HTTP/1.0 200 Success
  Server: HTTPd-WASD/7.0.0 OpenVMS/AXP Digital-TCPIP SSL
  Date: Wed, 31 May 2000 18:20:47 GMT
  Last-Modified: Tue, 30 May 2000 03:51:23 GMT
  Content-Type: text/html; charset=ISO-8859-1
  Content-Length: 4644
 
  |18:20:47.00 REQUEST  0407 0001 REQUEST  STATUS 200 rx:457 tx:5084 bytes 0.0000 seconds|
  |18:20:47.00 REQUEST  0252 0001 CONNECT  KEEP-ALIVE 131.185.250.108,47111|
  |18:20:52.06 NET      1502 0001 CONNECT  CLOSE 131.185.250.108,47111|

The following provides a brief explanation on the way WATCH operates and any usage implications. 

A single client may be connected to the WATCH facility at any given time.  When connecting the client is sent an HTTP response header and the WATCH report heading lines.  The request then remains connected until the WATCH duration expires or the client overtly aborts the connection.  During this period the browser behaves as if receiving a sometimes very slow, sometimes stalled, plain-text document.  As the server processes WATCHable events the text generated is sent to the WATCH-connected client. 

If the connection is aborted by the user some browsers will consider document retrieval to be incomplete and attempt to reconnect to the service if an attempt is made to print or save the resulting document.  As the printing of WATCH information is often quite valuable during problem resolution this behaviour can result in loss of information and generally be quite annoying.  Appropriate use of the duration selector when requesting a report can work around this, as at expiry the server disconnects, browsers generally interpreting this as legitimate end-of-document (when no content-length has been specified). 

During report processing some browsers may not immediately update the on-screen information to reflect received data without some application activity.  If scroll-bars are present on the document window manipulating either the horizonal or vertical slider will often accomplish this.  Failing that minimizing then restoring the application will usually result in the most recent information being visible. 

Browser reload/refresh may be used to restart the report.  A browser will quite commonly attempt to remain at the current position in the document, which with a WATCH report's sustained but largely indeterminate data stream may take some time to reach.  It is suggested the user ensure that any vertical scroll-bar is at the beginning of the current report, then refresh the report. 

Selecting a large number of categories, those that generate copious output for a single event (e.g. response body) or collecting for extended periods can all result in the receipt of massive reports.  Some browsers do not cope well with documents megabytes in size. 

NOTE

---

WATCH reports are written using blocking I/O. This means when large bursts of data are being generated (e.g. when WATCHing network data, response bodies, etc.) significant granularity may be introduced to server processing.  Also if the WATCH client fails or blocks completely server processing could halt completely!  (This has been seen when WATCHing through a firewall.)

---

When supplying WATCH output as part of a problem report please ZIP the file and include it an an e-mail attachment.  Mailers often mangle the report format making it difficult to interpret. 

16.5 - Command-Line Use

Although intended primarily as a tool for online use WATCH can be deployed at server startup with a command-line qualifier and provide report output to the server process log.  This is slightly more cumbersome than the Web interface but may still be useful in some circumstances.  Full control over event categories and filters is possible. 

• /NOWATCH Disables the use of the online WATCH facility. 

• /WATCH= Enables the server WATCH facility, dumping to standard output (and the server process log if detached).  When in effect the online facility is unavailable.  The string supplied to the qualifier may comprise four comma-separated components.  Only the first is manadatory.  Stated order is essential.  It will probably be necessary to enclose the complete string in quotation marks. 
  1. integer representing the categories to be reported
  2. client filter
  3. service filter
  4. path filter

  The category integer must be the bitwise-OR of the constants found in the ADMIN.H source code header file.  The end of the category list in the on-line WATCH report header contains a parenthesized integer.  This is the number representing the enabled categories for that report and can be used for those same categories from the command line. 

  The following examples illustrate the command-line WATCH specification. 

    /NOWATCH
    /WATCH=88
    /WATCH="88,*,*,/cgi-bin/*"