Isect Middleware How-To - isectd

Go to the first, previous, next, last section, table of contents.

## isectd

`isectd` doesn't really have arguments per se, but it does have a rather elaborate command language for configuration and administration. Commands are not read from a configuration file but are instead sent to `isectd` by using `isdclient`.

For instance, typing "`isdclient -c help`" would result in:

    abort                    isectd calls abort() - creates a core file
    [un]block clientid       [un]blocks incoming transactions from clientid
    buckets num secs         set the number of buckets to 'num' and the bucket
                             size to 'secs' seconds
    client id                examine the client structure for client id id
    clients                  displays the client list
    clone workerid           clones a worker id
    config events n          remember n events (listevents)
    delete {worker|client} id
                             deletes a client or worker
    halt                     isectd calls exit(1)
    idle id                  forcibly idles the busy worker id
    listevents [n]           list interesting events
    pause service | worker | all
                             pauses a worker (makes lazy) without killing it
    services                 lists isectd's services
    preempt [on|off]         sets or reports preempting option
    pri id n                 changes a client's priority to n
    ready service | worker | all
                             unpauses a worker
    reject                   force isectd to reject new client logins
    reset                    resets statistics
    send worker_id|service_name|all command:(reinit, status(TP workers only))
                             sends command to a worker, service or all services
    service id newservice    changes a client's service to newservice
    shutdown fd | all        shutdown the fd for sending
    start service | all      start processing messages for service
    stop service | all       stop processing messages for service
    unblock clientid         reverses "block clientid"
    worker id                examine the worker structure for worker id id
    workers [lazy | idle | busy | all]
                             displays the worker queues
    <nil>                    examine isdData

There are five types of commands: configuration, service control, worker control, client control, isectd control.

### Configuration

#### Config Service

Services are fundamental to how isectd manages the relationships between clients and workers. The contract clients and workers have with each other begins with how they find each other. When clients connect to isectd they tell it what service they're interested in. Respectively, when workers are configured they are assigned to services.

    isdclient -c "config service echo"

In the example above, "config service" is the command and "echo" is the service name being created. Services don't do much of anything on their own except exist. They provide an umbrella for workers to organize under.

#### Config Worker

Workers are the programs that do the work of the application. Isectd needs to know what service name to assign a worker to, what host computer it needs to run on, the path of the executable (program), and any additional parameters the program may require.

    isdclient -c "config worker echo localhost /usr/local/bin/isdecho"

The format of the command (above) is:

    config worker servicename hostname pathname [parameters]

`servicename` is the name of the service the worker will be assigned to. It must already have been defined using `config service` (above). For programmers, it's important to understand the parameters passed to the worker program will not start at argc == 1. The first few arguments to workers are used to communicate important information for them to rendezvous with isectd. Additional parameters for application programmer use start at argv[ISDARGMAZ].

#### Config Events

Of the more useful debugging features implemented in isectd during its creation is its ability to maintain an in-memory log of things its done. Though the types of events logged can not be controlled (at this time) the number of them can. Events are kept in a circular linked-list.

The command below set the number of events to be remembered at 100.

    isdclient -c "config events 100"

isectd's response would be:

    isdData.NumEvents=100

Events are viewed using the "listevents" command.

### Service Control

#### Services

Purpose: lists the configured services

Format: services

Example:

    isdclient -c "services"
                             service          id
                                 eat           3
                                echo           2
                                  op           1

#### Start

Purpose: Requests the workers assigned to a particular service (or all services) be started.

Format: start servicename|all

Example:

    isdclient -c "start echo"
    Ok.

    isdclient -c "start all"
    Ok.

Whenever isectd starts or stops services it will log messages to stderr reporting when the workers login. `isdexecd` also logs messages to its stderr indicating the command lines and the subprocess IDs (pids).

#### Stop

Purpose: Stops the workers associated with a particular (or all) service(s).

Format: stop servicename|all

Example:

    isdclient -c "stop echo"
    Ok.

    isdclient -c "stop all"
    Ok.

#### Pause

Purpose: Halts service without terminating workers

Format: pause servicename|all

Example:

    isdclient -c "pause echo"
    paused worker->id = 1, worker->name = echo

    isdclient -c "pause all"
    paused worker->id = 2, worker->name = eat
    paused worker->id = 1, worker->name = echo

Sometimes it's useful to halt transaction processing for a single service or all services. After executing the command, all the workers should show-up in the LAZY list, though their processes won't have been terminated.

#### Ready

Purpose: Undo a previous "pause" command

Format: ready echo

Example:

    isdclient -c "ready echo"
    Ok.

    isdclient -c "ready all"
    Ok.

### Worker Control

This is easily where most programmers and system administrators will spend their time. Tuning the system has more to do with the numbers of workers than most anything else. Also, worker administration is probably the most likely place problems will occur since isectd does not have direct control over the execution of worker processes. Remember, they are often running on remote machines and are started by another program, isdexecd.

#### Workers

Purpose: Displays the worker queues

Format: workers [lazy|idle|busy]

An example doesn't fit to well in the documentation, and it's probably more important to understand what it's showing more so than recognizing what one looks like.

Without any commands, "workers" displays the configured workers on the lazy, idle, and busy queues. The LAZY queue lists workers that for some reasone are configured but are unable to accept messages. This may be because they're associated service is paused or perhaps they haven't connected to isectd.

Idle workers are have logged on to isectd (they've connected) but aren't currently busy doing anything.

Busy workers have connected and accepted a message. Workers remain busy until isectd receives a reply from them that indicates that no more messages (replies) are expected back from the worker for this unit of worker (isdHeader.more).

Specifying any of lazy, idle, or busy, administrators can list only those queues they're interested in.

Regardless which queue is being examined, the following fields are included: service, worker id, avail? hostname, host pid, xactions, avg. resp., last resp., restarts, client id, and fd.

The `service` is the service name the worker is processing transactions for. As noted earlier, this is the same service name clients will request.

The `worker id` is a unique number assigned to each worker. The ID is used in commands like clone and delete for singling out workers.

`avail?` indicates with a 'yes' or 'no' whether or not the worker is available to accept transactions. Avail is typically no whenever the worker hasn't logged in, has been pause, or has been blocked.

The `hostname` and `host pid` display the hostname that was used to configure the worker. The pid is the process ID used on the remote host to identify the worker.

NOTE: the hostname will likely change in the future from displaying the configured hostname to displaying the IP address or DNS name for the remote end of the socket.

The number of transactions, average response times, and last response time are displayed next. Fory BUSY workers, the last response time is a running clock on the current transaction.

`restarts` is a cound of how many times isectd has had to restart the worker (or request it to restart). isectd will automatically attempt to restart any worker that closes its end of the socket without isectd's knowledge. This typically happens when the worker's pid is terminated or the process abends. If the service or worker is paused or blocked, isectd will not attempt to restart it.

`client id` is relevant only to busy workers. It reflects the client for which the worker is busily processing.

The column `fd` is used for the `shutdown` command explained in the "isectd control" sub section.

#### Worker

Purpose: examine detailed information about a specific worker

Format: worker worker_id

Example: isdclient -c "worker 1" worker->node.next = 0x08055eb0 worker->node.previous = 0xffffffff worker->node.queue = 0x0805004c worker->id = 1 worker->service_id = 2 worker->hostpid = 895 worker->name = "echo" worker->host = "localhost" worker->path = "/usr/local/bin/isdecho" worker->args = "" worker->Transactions = 1 worker->fromCount = 1 worker->toCount = 1 worker->restarts = 1 worker->stopped = 0 worker->up_not_log_in = 0 Most of the information is self-explanatory. But the node information at the top shows the forward and backward links to adjacent workers as well as a pointer to the head of the queue. The values for `fromCount` and `toCount` are recorded separately from `Transactions` becuase some clients will send requests in pieces, and workers may reply in pieces. Such is the case when a worker is created that processes large files (like print jobs). The client may send 2048-byte chunks one at a time. Each chunk, supposing it was sent to this worker, increases the `toCount` by one. NOTE: one of the things I'd love to add to this command is the ability to examine either the last transaction sent or the current one. In the case that workers behave poorly (as in, they hang) it would be nifty to be able to get a hex-dump of the last thing they were sent since it's sometimes difficult to determine what input the worker was unable to handle.

#### Clone Worker

Purpose: creates an identical worker from an existing one

Format: clone wid

    isdclient -c 'clone 2'
    cloned 4 from 2

One of the benefits of having isectd in the middle is your system can be scaled while it's running. What is meant by that is the number of workers assigned to any service can be dynamically increased simply by cloning or configuring new ones.

When a worker is cloned, everything is the same as in the original worker; same host, pathname, arguments, service, everything! The only difference is the new worker is assigned a new unique worker ID and the transactions counters begin at zero.

#### Delete Worker

Purpose: removes a worker from the configuration

Format: delete worker wid

    isdclient -c "delete worker 3"
    worker 3 deleted

Before workers can be deleted they must be on the LAZY queue, which means they must first be stopped.

As easily as configurations can be dynamically scaled up using the "config worker" and "clone" commands, they can be scaled down. Oftentimes during the development of a worker (or the early beta testing of it) they are more easily debugged when only one is running. When only one is running attaching a debugger predictably catches the next transaction since there are no other workers for the scheduler to assign the transaction to.

A preferable way of doing this is to use the "service" command documented under the subsection "Client Control."

#### Pause

Purpose: move a worker from the IDLE to the LAZY queue

Format: pause wid|all

Example:

    isdclient -c 'pause 2'
    paused worker->id = 2, worker->name = eat

Pausing a worker is a good way to stop isectd from sending transactions to a it without terminating the worker. To re-enable the worker use the 'ready' command.

#### Ready

Purpose: readies a previously 'paused' worker

Format: ready wid

Example:

    isdclient -c 'ready 2'
    readied worker->id = 2, worker->name = eat

#### Idle

Purpose: forces a worker on the busy queue onto the idle queue

Format: idle 2

    isdclient -c 'idle 2'
    idled worker 2

Forcing a worker onto the IDLE queue is not an operation to be taken lightly. There are times however, during the development of a new worker where the worker itself may have become confused over the state its in and has forgotten to reply back to isectd. A programmer may have determined that it is safe to "idle" the worker. In isectd determines the worker was busy on a client's transaction (which it probably was) it will send a message back to the client indicating the worker died. It does this by setting header.error to '-1' and the message to "PROTOCOL INTERVENTION".

The benefit of doing this vs. just killing the worker and letting isectd/isdexecd restart it are probably minimal.

### Client Control

#### Clients

#### Client

#### Block

#### Unblock

#### Delete Client

#### Pri

#### Service

Though this is one way of doing it, an easier way is actually to configure a separate service and assign a single worker to it. After that worker has started (and presumably, a debugger attached to it) the operator can assign a client to the debugged service using the "service" command. The service command (documented in the Client Control subsection) is a sneaky way of moving clients to other services.

Note: it is not adviseable to assign clients to workers that won't understand the messages sent to them. Doing so can confuse both or either of the client and the worker--which may make sense in situations where programmers or QA analysts are interested in checking how either process handles exceptions of that nature.

### Isectd Control

The isectd control commands are pretty straightforward. They are intended to aid in the runtime-control of the middleware daemon as well as to assist in tuning it.

Since the original creation of these commands various other pieces of data have been added to the structures that would probably be worth displaying but haven't been coded yet. Eventually those, too, will be gotten around to and more information will be available.

#### Status

Purpose: displays the root isectd structure and its interesting members

Format: <nil> (don't send anything)

Example:

    isdclient -c ""
    isdData.configfile       = ""
    isdData.host             = "isect"
    isdData.ServiceList      = 3
    isdData.PriList          = 1
    isdData.LazyWorkers      = 0
    isdData.IdleWorkers      = 2
    isdData.BusyWorkers      = 0
    isdData.socket           = 4
    isdData.port             = 12417
    isdData.req_socket       = 5
    isdData.req_port         = 5501
    isdData.NextClientID     = 4
    isdData.MaxUsers         = 0
    isdData.command_worker   = "op"
    isdData.Events           = 0
    isdData.NumEvents        = 0
    isdData.NumBuckets       = 6
    isdData.BucketSecs       = 10
    isdData.PeakBucket       = 0
    isdData.Buckets          = 0  0  0  0  0  0  €

configfile - obsolete.

host - The hostname of the computer running isectd - usually the alias "isectd".

ServiceList - The number of services defined

PriList - The number of clients on the PriList queue. This is the same number as the number of clients attached.

LazyWorkers, IdleWorkers, BusyWorkers - the number of workers on each of the Lazy, Idle, and Busy queues.

socket - the file descriptor assigned to the socket isectd is listening to for worker connection requests

port - the TCP port isectd was assigned randomly by the host that it listens to for worker connections. This is one of the arguments passed to each worker through isdexecd.

req_socket - the file descriptor assigned to the socket isectd is listening to for client connection requests.

req_port - the TCP port isectd was configured to use for client connection requests. This should be the same port number listed in /etc/services.

NextClientID - the next client ID to assign. Increments sequentially starting at 0.

MaxUsers - The number of simultaneous client connections isectd will allow. If the value is 0, isectd will calculate the number of permitted connection by calculating the number of available file descriptors less the number of worker connections less a mysterious number called TOMS_CONSTANT I invented myself.

command_worker - the name of the service isectd intercepts for its own commands.

Events - The number of events currently in the Events queue

NumEvents - The maximum number of events isectd will remember. It was always remember the last 'NumEvents' events.

NumBuckets - For histogramming isectd performance, this is the number of buckets isectd is remembering

BucketSecs - The size, in seconds, of each bucket. 12 five-second buckets is a minutes' worth of data.

PeakBucket - The highest transactions completed value since histograming was started.

Buckets - A list of NumBuckets integers showing the number of completed transactions during each BucketSecs window.

#### Help

Purpose: Displays a list of isectd commands

Format: help

Example:

    isdclient -c "help"
    abort                    isectd calls abort() - creates a core file
    [un]block clientid       [un]blocks incoming transactions from clientid
    buckets num secs         set the number of buckets to 'num' and the bucket
                             size to 'secs' seconds
    client id                examine the client structure for client id id
    clients                  displays the client list
    clone workerid           clones a worker id
    config events n          remember n events (listevents)
    delete {worker|client} id
                             deletes a client or worker
    halt                     isectd calls exit(1)
    idle id                  forcibly idles the busy worker id
    listevents [n]           list interesting events
    pause service | worker | all
                             pauses a worker (makes lazy) without killing it
    services                 lists isectd's services
    preempt [on|off]         sets or reports preempting option
    pri id n                 changes a client's priority to n
    ready service | worker | all
                             unpauses a worker
    reset                    resets statistics
    send worker_id|service_name|all command:(reinit, status(TP workers only))
                             sends command to a worker, service or all services
    service id newservice    changes a client's service to newservice
    shutdown fd | all        shutdown the fd for sending
    start service | all      start processing messages for service
    stop service | all       stop processing messages for service
    unblock clientid         reverses "block clientid"
    worker id                examine the worker structure for worker id id
    workers [lazy | idle | busy | all]
                             displays the worker queues
    <nil>                    examine isdData

Actually, any command isectd doesn't recognize, like "razmafritz," will result in a display of understood commands.

#### Abort

Purpose: isectd calls abort()

#### Halt

Purpose: isectd calls exit(1)

#### Buckets

Purpose: set the number and size of buckets for histogramming

Format: buckets num size

Example:

    isdclient -c "buckets 12 5"

The above example specifies 12 buckets of five-seconds each.

#### Listevents

Purpose: lists interesting events starting with the most recent

Format: listevents [num]

Example:

    isdclient -c 'config events 20'
    isdData.NumEvents = 20
    [do some stuff]
    isdclient -c 'listevents 10'
      len=14, seq=1, more=0
    request from client 404
    select() = 1
    client: priority=2, hostpid=7243, id=404, loghost=localhost
    select() = 1
    request from client 403
    select() = 1
    worker x0x8055478 write-ready
    select() = 1
      len=7, seq=1, more=0

Before you can listevents, you must first configure isectd to start remembering them with the 'config events' command. The output of listevents is usually of little interest but to developers and programmers. In more interestingly configured isectds (than the one I'm testing with right now) listevents does a good job of showing why isectd did or did not schedule a transaction from a client, or what's happening with various workers.

#### Shutdown

Purpose: closes an socket for writing

Format: shutdown all|fd

Connected, two-sided TCP sockets allow the closing down of one direction without closing the other. Isectd can 'shutdown' its write-end without shutting down its read-end. Shutting down a socket (usually used on worker sockets) makes the worker think isectd is closing down, which makes the worker shutdown, which makes isectd detect an abnormal worker termination and attempt to restart it. If this doesn't happen when a socket is shutdown the worker is not listening for messages from isectd an may have become catatonic.

Go to the first, previous, next, last section, table of contents.