WASD Hypertext Services - Technical Overview

The most fundamental component of the WASD VMS Hypertext Services environment is the HTTPd, or HyperText Protocol Transport Daemon, or HTTP server.  It has a single-process, multi-threaded, asynchronous I/O design. 

2.1 - Server Features

The package provides a complete implementation of a basic HTTP/1.0 server, including:

• concurrent, multi-threaded client support

• "GET", "HEAD", "POST" and "PUT" HTTP method support

• virtual services (servers)

• optional enhanced privacy using Secure Sockets Layer (SSL) technology (USA domestic, full-strength encryption)

• proxy serving, with local caching, for all HTTP/1.0 methods, plus the CONNECT method (allowing SSL tunnelling)

• "If-Modified-Since:" / "304 Not Modified" functionality (document is only sent if modified since time specified by client)

• HTTP/1.0 de-facto persistent connections (i.e. "Keep-Alive:", reducing the number of TCP/IP connects required)

• versatile directory listing (generic and VMS-style)

• CGI-compliant subprocess scripting (with configurable, automatic, MIME content-type initiated activation)

• non-server and user account scripting (requires a minimum of VMS V6.2)

• "CGIplus" scripting (offering reduced latency, increased throughput and reduced system impact)

• "Persistent" scripting, Run-Time Environments (RTEs) that provide for simple persistent scripting (with example Perl engine)

• "ISAPI" extensions/scripting (also offering reduced latency, increased throughput and reduced system impact)

• DECnet-based CGI scripting (with connection reuse)

• OSU (DECthreads server) scripting emulation, with connection reuse (as per OSU 3.3a), allowing many OSU scripts to be employed unmodified

• script processor (e.g. PERL) configurable on file type (extension)

• Server-Side Includes (HTML pre-processing)

• configurable cache, with time-based and forced re-validation (reload)

• clickable-image support (both NCSA and CERN formats)

• smart rule mapping (conditional rules)

• "Basic" and "Digest" user authentication and path/group-based authorization

• SYSUAF-authentication and VMS user security profile based file access control

• Web-standard, "common" and "combined" access log formats (allowing processing by most log-analysis tools), along with a user-definition capability allowing custom log formats

• logging periods, where log files automatically change on a daily, weekly or monthly basis (keeps log files ordered and at a managable size)

• host-level access control, on per-host or per-domain acceptance or rejection

• single server can support multi-homed (host name) and multiple port services

• customizable message database (capable of supporting non-English and concurrent, multiple languages)

• on-line server configuration, including reports on requests, loaded configuration, mapping rules, authorization information and graphical activity displays

• "single button" control of multiple servers on both single systems and clusters

• on-line, live server processing event report (WATCH)

The HTTPd executes permanently on the server host, listening for client connection requests on TCP/IP port 80 (by default).  It provides concurrent services for a (technically) unlimitted number of clients (constrained only by the server resources).  When a client connects the server performs the following tasks:

1. creates a thread for this request
2. reads and analyzes the HTTP request sent,
   depending on the nature of the request ...
   • transfer of a file (possibly from cache)
   • processing of an SSI HTML file
   • processing of a clickable-image mapping file
   • directory listing
   • file/directory create/update
   • server administration
   • web file-system update
   • interpretation of a menu file
   • spawn a subprocess to to provide a CGI scripting environment:
     • standard CGI
     • WASD CGIplus
   • connects to a DECnet object to provide a scripting environment:
     • standard CGI
     • OSU (DECthreads) emulated
3. closes the connection to the client and disposes of the thread data structures

For I/O intensive activities like file transfer and directory listing, the AST-driven code provides an efficient, multi-threaded environment for the concurrent serving of multiple clients. 

For scripts, the technique of using multi-threaded, concurrent, spawned subprocesses or network processes, attached to standard input/output streams, provides a versatile, extensible, powerful scripting environment.  Any DCL procedure or image executing within the process can behave essentially as an HTTP server.  This capability is employed to easily extend the basic services provided by the core daemon code.  An HTTP script/server for this environment does not need to concern itself with network activities, it merely reads and writes from the standard I/O streams. 

2.3 - VMS Versions

The WASD server is officially supported on any VMS version from V6.0 upwards, on both Alpha and VAX architectures.  Obviously no guarantees can be made for yet-to-be-released versions but at a worst-case these should only require relinking.  WASD has also been known to compile and run successfully under V5.5-2 (see 4.7 - VMS 5.5-2).

Note that the non-server account scripting requires a minimum VMS V6.2, to support the $PERSONA services required for this functionality. 

The WASD distribution and package organisation fully supports mixed-architecture clusters (both Alpha and VAX in the one cluster) as one integrated installation. 

2.4 - TCP/IP Packages

The WASD server uses the Compaq TCP/IP Services (UCX) BG $QIO interface.  The following packages support this interface and may be used. 

• Compaq TCP/IP Services for OpenVMS (Compaq Computer Corporation), any version
• Digital TCP/IP Services for OpenVMS (aka UCX), any version
• MultiNet for OpenVMS (Process Software Corporation), any version
• TCPware (Process Software Corporation), any version

As of version 4.4 the HTTP server has undergone a limited degree of internationalization.  The impetus for this development was the use of the package in some European countries where English is not the first language. 

As this was not within the original critera while building the server it has meant an extensive rewrite of some functionality within the code (I hope it's appreciated ;^) The objective is to provide users of the server with a more familiar and therefore more comfortable environment.  Unfortunately the server administrators are still confronted by an English-language interface and documentation, but then the package doesn't pretend to be international software! 

Aspects

The international features only apply to the server, not necessarily to any scripts! 

• Server Messages

  The server uses an administrator-customizable database of messages that can contain multiple language instances of some or all messages, using the Latin-1 character set (ISO8859-1). Although the base English messages can be completely changed and/or translated to provide any message text required or desired, a more convenient approach is to supplement this base set with a language-specific one. 

  One language is designated the prefered language.  This would most commonly be the language appropriate to the geographical location and/or clientele of the server.  Another language is designated the base language.  This must have a complete set of messages and is a fall-back for any messages not configured for the additional language.  Of course this base language would most commonly be the original English version. 

  More than just two languages can be supported.  If the browser has prefered languages set the server will attempt to match a message with a language in this preference list.  If not, then the server-prefered and then the base language message would be issued, in that order.  In this way it would be possible to simultaneously provide for English, French, German and Swedish audiences, just for example. 

  For message configuration information see 9 - Message Configuration. 

• Server Dates

  Dates appearing in server-generated, non-administrative content (e.g. directory listings, not META-tags, which use Web-standard time formats) will use the natural language specified by any SYS$LANGUAGE environment in use on the system or specifically created for the server. 

• Virtual Services

  Virtual-server-associated mapping, authorization and character-sets allow for easy multiple language and environment sites.  Further per-request tailoring may be deployed using conditional rule mapping described below. 

  For virtual services information see 6 - Server Configuration. 

• Conditional Rule Mapping

  Mapping rules map requested URL paths to physical or other paths (see 10 - Mapping Rules).  Conditional rules are only applied if the request matches criteria such as prefered language, host address (hence geographical location to a certain extent), etc.  This allows requests for generic documents (e.g. home pages) to be mapped to language versions appropriate to the above criteria. 

  For conditional mapping information see 10.7 - Conditional Mapping. 

• Character Sets

  Generally the default character set for documents on the Web is ISO-8859-1 (Latin-1). The server allows the specification of any character set as a default for text document responses (plain and HTML).  In addition, text document file types may be modified or additional ones specified that have a different character set associated with that type.  Furthermore, specific character sets may be associated with mapping paths.  A site can therefore relatively easily support multiple character set document resources. 

  Major WASD scripts also use the server default character set and/or any request determined character set (based on request path with conditional mapping to language or character set preferences) via CGI variable contents. 

  For further information see [CharsetDefault] and [AddType] in 10.3.5 - SET Rule. 

This section describes WASD-specific characteristics of the available HTTP/1.0 request methods. 

2.6.1 - GET

Of course, the GET method is used to access documents supplied by the server.  There is nothing WASD server-specific about this method. 

2.6.2 - POST & PUT

The WASD HTTPd does not differentiate between POST and PUT methods, both are handled identically. 

Script Handling

The "normal" usage of the POST method is to return data from a <FORM>..</FORM> construct to a script running on the server.  In this regard WASD is no different to other any web server; the form data is delivered to the script's standard input as a stream of URL-encoded text.  For example:

  name=Fred+Nurk&address=Fred%27s+House%0D%0A0+Nowhere+Lane&submit=Submit

Note that WWW_CONTENT_LENGTH will be the length of the form data.  See "Scripting Environment" document for further information. 

File Creation/Upload

If the client sends data back to the server using either the POST or the PUT methods, without a script being mapped to be executed in response to that data, the WASD HTTPd will create a file corresponding to the specified path.  The data stream may be text or binary. 

Of course, for the server to accept POST and PUT data in this manner, authentication and authorization must be enabled and allow such access to the request path. 

The data stream is processed according to MIME content-type:

• application/x-www-form-urlencoded

  The server specially processes "application/x-www-form-urlencoded" POSTS (i.e. those generated by <FORM>...</FORM>, allowing files to be created directly from HTML forms.  The processing eliminates any field names from the URL-encoded data stream, placing only field values into the file.  This capability can be quite useful and is demonstrated in the Update HTTPd module (UPD.C).

• multipart/form-data

  This server can process a request body according to RFC-1867, "Form-based File Upload in HTML".  As yet it is not a full implementation.  It will not process "multipart/mixed" subsections.  The implementation is basic, providing a facility to allow the upload of a file into the server administered file system.  The ACTION= parameter of the <FORM> tag must specify the directory (as a URL path) in which the uploaded file will be created. 

  The following example HTML illustrates how a form may be used to upload a file from the browser host file system:

    
    <FORM METHOD=POST ACTION="/web/directory/" ENCTYPE="multipart/form-data">
    <INPUT TYPE=submit VALUE=" Upload document ... ">
    <INPUT TYPE=file SIZE=50 NAME=uploadfile>
    </FORM>
  
  

  Again, the Update module (UPD.C) illustrates the use of this facility.

  NOTE

  ---

  This capability has only been tested against Netscape Navigator versions 2 and 3. VMS Netscape Navigator 3.0b5 hangs if an upload of a variable-record format file is attempted.  Stick to STREAM-LF or fixed, or convert the file to STREAM-LF. 

  ---

• text/html
  text/plain

  Text file created according to the path, VMS record type is STREAM-LF. 

• image/gif
  application/octet-stream
  etc., etc., etc. 

  Any other MIME type is considered binary and the created file is made an UNDEFINED record type. 

A directory will be created by the HTTPd if a directory path is provided with the POST or PUT methods.  For example:

  /dir1/dir2/dir-to-be-created/

A file will be deleted by the HTTPd if the file path ending with a wildard version specification is provided with the POST or PUT methods.  For example:

  /dir1/dir2/file-to-be.deleted;*

A directory will be deleted by the HTTPd if a directory path ending with a wildard version specification is provided with the POST or PUT methods.  For example:

  /dir1/dir2/dir-to-be-deleted/;*

The DELETE method should delete the file or directory corresponding to the supplied path.