Project  : Netrek Metaserver Overhaul 1995
Document : Design Specification
Revision : BX2
Dated    :  1-Aug-1996 
Approved : (nobody)

Design Specification (Metaserver)

  1. single process listening on UDP port and specific TCP ports, handles UDP updates inline, forks to handle TCP connections so as to avoid stall by remote end of TCP connection.

  2. optional logging of incoming packets to a file with timestamps,

  3. optional logging of TCP requests to a file with timestamps,

  4. use httpd logging format so that httpd log analysis/graphing programs may be used,

  5. use textual UDP packet format with newline delimiters,

  6. provide one TCP port that acts like a httpd using HTTP/1.0 and provides two minute updates to a netscape client.

The diagram below is a level zero data flow diagram.

The following processes are defined on the diagram;

0 Load

During initialisation, loads the saved disk copy of the server data into the in-memory servers data store. This is so that the metaserver is able to report following a shutdown or restart.

1 UDP Listen

During initialisation, enables the receipt of UDP packets from servers to the metaserver.

2 TCP Listen

During initialisation, enables the connection by clients to the TCP ports on which the metaserver provides information.

3 UDP Read

On receipt of a UDP packet, this process reads it from the system buffers.

4 Parse Packet

This process parses the packet, and stores the data into the server data structure in memory. (Optimisation may allow for this process to trigger process 7, Format Output, so that TCP connections can be handled more rapidly).

5 TCP Accept

Accepts a connection from a client, and passes the socket to the Fork process.

6 Fork

Duplicates the running process so that output via TCP will not stall other requests to the metaserver.

7 Format Output

Prepares textual output as determined by the request.

8 TCP Write

Writes output to the client.

9 Save

Periodically, or on termination signal, writes the server data store to a disk file for later use.


Design Specification (Server)

  1. allow server owner to specify list of metaservers to be used, including host name, port number, and five update frequency maximums;

  2. when player count changes, or on expiry of zero players maximum frequency timer, send an update _provided_ the time interval since the last update will not exceed the new update frequency maximum.


Design Specification (Protocol)

  1. Server will send UDP packets to Metaserver(s) under control of the server owner,

  2. Metaserver will not reply. (If packet loss occurs enough to be a problem, then we don't want players to go and make it worse.)

  3. Update packet from server will be limited to maximum practical UDP packet size, no follow-on packet. If truncation will occur, server may omit player details.

  4. Update packet will be in ASCII text with newline (\n) delimiters. Any delimiters found in text to be sent will be removed.

  5. Update packet will include a version for the format, to allow for significant changes to the format in future.

  6. Update packet format will be as follows;

    So, an example packet would be;

    The keywords and their meanings are;


Program Structure (Metaserver)

The program will consist of multiple C source modules linked by function calls. Each module will have a set of public definitions defined by a header file. Modules will be responsible for certain actions or data structures.

There is no direct relationship between the data flow diagram and the program structure.


Modules within Metaserver

Network Module
Mainline of code. Responsible for interacting with the host network services. Waits for packets from servers. Waits for connections via TCP.

                main()
                {
                    /* read configuration */
                    man_initialise();
                    
                    /* open log file */
                    log_open();
                    
                    /* open network */
                    ...
                    
                    /* process packets and requests */
                    while ( TRUE )
                    {
                        /* process UDP packets */
                        inc_handle();
                        
                        /* process TCP connections */
                        rep_handle();
                    }
                } /* main() */
Logging Module
Manages the logging of events within the metaserver. A logging mask defines the classes of events that are to be logged. The output is sent to standard output, and includes timestamps. The format to be used will match the httpd server log format.

                
                log_open()
                {
                }
                
                log_send(char *something)
                {
                }
                
                log_close()
                {
                }
Data Module
Manages an in-memory database of information about servers. The database may also reside in a disk file. The database consists of packets received from servers, along with an indexing structure. The indexing is to provide the ability to replace old packets from the same server, and generate output in the appropriate order to the user(s).

Only the main thread will update the database. The forked threads for handling TCP connections will read the database. Some form of synchronisation must be used to ensure that a reader does not see a view of the database that is inconsistent due to concurrent updates. (Discussion)

The packet format is human readable. For this reason, the data format in the disk file should also be human readable. Some information about the packet does not arrive in the packet, and must be stored with the packet. This includes actual source IP address, and date time of receipt. The internal representation of this data is to be described by a packet structure.

                
struct packet
{
    char *buffer;       /* pointer to packet buffer  */
    size_t length;      /* byte length of buffer     */
    time_t received;    /* date time packet received */
    addr_t source;      /* ip address of packet source */
    struct server       /* pointers within packet buffer to server items */
    {
        char *address, *type, *port, *players, *free, *tmode, *comment;
    } server;
    struct player       /* pointers within packet buffer to player items */
    {
        char *slot, *team, *class, *rank, *name, *login;
    } player[MAXPLAYER];
};
The external data represenation in the disk file is to be identical to the packet format as it arrived from the server, with the following exceptions;

The start of each record will be identified by a special character followed by the textual date time received and the textual IP address.

The index is to consist of a set of single dimension array of pointers into the packet structures. A primary index array will provide ordering by IP address and port number. Secondary index arrays, if required, will provide ordering in a request specific manner.

(bsearch() and qsort() will be used)

                
                /* Sets up empty packet list pointer array */
                dat_initialise()
                {
                }
                
                /* Find a packet with the same key as a received packet */
                struct packet *dat_find(struct packet *packet)
                {
                }
                
                /* Insert a new packet into the list */
                struct packet *dat_insert(struct packet *packet)
                {
                }
                
                /* Replace an old packet with a new packet */
                void dat_replace(struct packet *old, *new)
                {
                }
                
                /* Expire and delete old packets */
                void dat_expire()
                {
                }
                
                /* Traverse list for client request */
                void dat_traverse
                (
                    int ((*compar)(const struct packet *,
                        const struct packet *)),
                    int ((*action)(const struct packet *)
                )
                {
                    /* allocate a private copy of the packet list */
                    auto struct packet *ours[MAXSERVERS];
                    
                    /* copy the real one into our copy */
                    memcpy ( ours, packets,
                        sizeof ( struct packet * ) * MAXSERVERS);
                    
                    /* sort the copy using the caller's comparison routine */
                    qsort ( ours, MAXSERVERS, sizeof ( struct packet * ),
                        compar );
                    
                    /* call the caller's action routine for every packet */
                    for ( i=0; i<MAXSERVERS; i++ )
                        if ( ours[i] != NULL ) action ( ours[i] );
                }
                
                /* Save packet list structure to file */
                int dat_save()
                {
                }
                
                /* Load packet list structure from file */
                int dat_load()
                {
                }
Incoming Module (Probably a part of main)
        
                /*
                **  Handle an incoming UDP packet, validate it,
                **  and update the server packet database.
                */
                inc_handle( char *buffer, size_t length, addr_t source )
                {
                    struct *packet new, old;
                    
                    new = inc_decode ( buffer, length, source );
                    if ( new == NULL ) return;
                    
                    old = dat_find ( new );
                    if ( old == NULL )
                        dat_insert ( new );
                    else
                        dat_update ( old, new );
                }

/*
**  Build struct of pointers into data fields within packet.  Return either
**  NULL for an invalid packet, or a pointer to the allocated struct.
*/
struct *packet inc_decode ( char *buffer, size_t length, addr_t source )
{
    struct packet packet, *returned;
    
    packet.buffer = buffer;
    packet.length = length;
    packet.source = source;
    packet.received = time ( NULL );
    
    packet.server.address = strtok ( buffer, delimiter );
    if ( packet.server.address == NULL ) return NULL;
    
    packet.server.type    = strtok ( NULL,   delimiter );
    if ( packet.server.type    == NULL ) return NULL;
    
    packet.server.port    = strtok ( NULL,   delimiter );
    if ( packet.server.port    == NULL ) return NULL;
    
    packet.server.players = strtok ( NULL,   delimiter );
    if ( packet.server.players == NULL ) return NULL;
    
    packet.server.free    = strtok ( NULL,   delimiter );
    if ( packet.server.free    == NULL ) return NULL;
    
    packet.server.tmode   = strtok ( NULL,   delimiter );
    if ( packet.server.tmode   == NULL ) return NULL;
    
    packet.server.comment = strtok ( NULL,   delimiter );
    if ( packet.server.comment == NULL ) return NULL;
    
    /* ... player stuff ... */
    
    returned = malloc ( sizeof ( struct packet ) );
    if ( returned == NULL ) return NULL;
    
    memcpy ( returned, packet, sizeof ( struct packet ) );
    return returned;
}
Reports Module
Responsible for handling of TCP request from mainline. Causes fork of process. Explores the packet list to generate output to client. On completion, closes the connection and exits.

                        
                rep_handle(int socket, int port)
                {
                }
Management Module
                /* Read configuration file */
                man_initialise()
                {
                }

--
James Cameron
Digital Equipment Corporation (Australia) Pty. Ltd. A.C.N. 000 446 800
(cameron@stl.dec.com)