Queue Manager Infrastructure

John Bowe, bowe@osf.org
15-September-1996


This document is organized into sections on Data Structures, Operations At a Glance, and Some Implementation Details.

For convenience matching IDL files are available


Data Structures

There will be two IDL files that define the "publicly visible" data structures and constants. One is for API users, and one for direct queue manager users (ie, for the API to use). The latter also uses the former.

Also included are data structures that may be written to a backing store. An IDL file is used to generate the encoding and decoding functions.

Structures used internally by the q-mgr need not be defined here and may differ, depending on implementation. Internally-used structures will probably be defined for efficiency of the q-mgr, not elegance of external interface.

The names of attributes and operations here are not necessarily those of the actual IDL file.

There will be some "support" structured defined for the RPCs, such as lists of some of the structures defined below. (A "list" is simply a count (unsigned32) and an array of the structure of interest, [ptr,size_is(count)] my_struct_t *list;.) These need no be defined in this document, though they will be defined in the IDL file.

Attributes

Queue Attributes

The following are attributes of each queue.

Attribute Type Description
id uuid_t UUID of this queue
name string name of the queue - either simple string (for queue name under "junction") or full CDS name for where to try to export.
annotation string uninterpreted annotation string
aliases array of strings alternate names of this queue
len unsigned32 number of items currently available in queue
max_len unsigned32 max items allowed in queue
flags unsigned32 various bit flags (persistence, etc.)
empty_timeout utc_t time of "empty inactivity" before deleting queue
last_activity utc_t time of last activity
create_time utc_t time of queue creation

Message Attributes

The following are attributes of each message (or queue item).

Attribute Type Description
id string UUID of this item in the queue (this is also used as a key for storing the items to a backing store)
datatype uuid_t semantically identifies the message, and allows recipient to determine how do decode the message
msgtype enum (data, ack, etc.) data, ack, etc.
notice unsigned32 bitmask of options for notices to send (upon dequeue, expiration, exception, none)
sender sec_id_pa_t PAC of the principal who sent the message
persistence enum persistence behavior
flags unsigned32 various bit flags (protection, etc.)
priority unsigned32 priority of the item
time_enqueued utc_t when message was enqueued by q-mgr
expire_time utc_t when this message will expire
valid_time utc_t when this message will be available for dequeue
reply_queue string name of queue for replies

Filtering Masks for Selection Criteria

A structure is defined for the mask used to specify selection criteria. A mask is a typed-structure with a triplet of: datatype (int, uuid_t, utc_t, string), relation (equal, less than, etc.), and a value. Selection is based on an array of these masks.

A Message

A message is very simple, as far as the system is concerned. It is simply a pointer to an array of bytes (idl type byte) and the length of the array.

Operations At a Glance

All calls take a binding handle as an in parameter, and error_status_t as out.

Management (Queue Manager) Operations

These may also be thought of as operations on a queue manager as opposed to a queue itself.

Operation Description Parameters
mos_mgmt_create_queue create a new queue In: name, queue attributes, flags
Out: UUID of new queue
mos_mgmt_get_id_list get list of queues IDs In: -
Out: array of queue UUIDs
mos_mgmt_get_name_list get list of queues names In: -
Out: array of queue names
mos_mgmt_rename rename a queue In: old queue name, new queue name
Out: -
mos_mgmt_move migrate a queue from one q-mgr to another (this operation is performed on the destination q-mgr) (This needs more thought. Probably better for API to do the work.) In: queue name (destination), queue name (source)
Out: -

Queue (Data) Operations

These are operations on a particular queue. We think of these as the "data" operations.

Operation Description Parameters
mos_q_enqueue add an item to an existing queue In: queue name, attributes, message
Out: UUID of item
mos_q_dequeue dequeue first item from a queue that matches given selection criteria; optionally blocks, depending on flags In: queue name, selection filter, flags
Out: attributes, message
mos_q_dequeue_next dequeue first item on a queue In: queue name
Out: attributes, message
mos_q_peek return copy of first item from a queue that matches given selection criteria; optionally blocks, depending on flags In: queue name, selection filter, flags
Out: attributes, message
mos_q_purge purge/delete item in queue with given message ID optionally blocks, depending on flags In: queue name, message ID
Out: attributes, message
mos_q_retrieve_attr retrieve attributes of next item from queue that matches given selection criteria In: queue name, selection filter
Out: attributes
mos_q_ping determines if a given queue exists In: queue name
Out: uuid
mos_q_retrieve_attr_by_id retrieve attributes of item from queue with given message ID In: queue name, message ID
Out: attributes
mos_q_ping determines if a given queue exists In: queue name
Out: queue ID
mos_q_get_id_list get list of message IDs In: queue name
Out: array of message IDs
mos_q_delete_queue delete a queue In: queue name, flags
Out: -
mos_q_get_attr_by_name get the attributes of a given queue In: queue name
Out: attributes
mos_q_get_attr_by_id get the attributes of a given queue, based on ID In: queue ID (UUID)
Out: attributes
mos_q_set_attr_by_name set the attributes of a given queue In: queue name, attributes
Out: -
mos_q_set_attr_by_id set the attributes of a given queue, based on ID In: queue ID (UUID), attributes
Out: -
mos_q_move_item move a single message from one queue to another, given message ID In: queue names, message ID
Out: -

Some Implementation Details

All operations assume the caller has permission, based on the ACLs.

There will be several backing stores: one for the ACL database, one for queue attributes, and one for queue items (messages) themselves. The latter is use only for messages specified as "persistent".

Queue attributes will be kept in memory. (They are not that large, and this information should be readily available for various reasons, such as testing timeouts and checking if name already exists.)

Management (Queue Manager) Operations

mos_mgmt_create_queue

Create a queue. Steps:
  1. Ensure that a queue by this name does not already exist. If a queue by this name exists, return its ID.
  2. Add to the in-memory list of queues.
  3. Write to the queue-info backing store.

mos_mgmt_get_id_list

Return a list of the IDs of all queues. Steps:
  1. Return UUIDs from the in-memory list of queues.

mos_mgmt_rename

Rename a queue within queue manager. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Rename it.
  3. Write to the queue-info backing store.

mos_mgmt_move

Move a queue from one queue manager to another, retaining all atributes.

NOTE: This still needs some thought. The API can probably do all the work better, or at least more conveniently.

Queue (Data) Operations

mos_q_enqueue

Enqueue a message in a given queue. The message is placed in the queue according to priority. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Add the new message into its queue, based in priority.
  3. If the persistence flag is set, write it to the backing store.
  4. If the notification is desired, send an appropriate ack to the sender.

mos_q_dequeue

Dequeue a message (body and attributes) from a queue, optionally dequeuing it and optionally blocking until a message is available. Also optionally accepts a selection filter. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Find the matching message in the queue (selection filter, if specified) (if no filter, get first item).
  3. Extract item (attributes and body).
  4. If the persistence flag is set, remove it from the backing store.
  5. If the notification is desired, send an appropriate ack to the sender.

mos_q_dequeue_next

Simpler version mos_q_dequeue. Dequeues next available message and attributes from a queue.

mos_q_peek

Similar to mos_q_dequeue, but only a copy of the message attributes and body are returned; it is not dequeued.

mos_q_purge

Purge a message with a given ID. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Find the matching message in the queue, given ID.
  3. Remove it from the list.
  4. If the notification is desired, send an appropriate ack to the sender.

mos_q_retrieve_attr

Retrieve message attributes from a queue, optionally blocking until a message is available. Also optionally accepts a selection filter. Similar to mos_q_dequeue, but a message body is not returned and the message is not dequeued. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Find the matching message in the queue (selection filter, if specified) (if no filter, get first item).
  3. Extract attributes of item.

mos_q_retrieve_attr_by_id

Retrieve message attributes from a queue, optionally blocking until a message is available. Also optionally accepts a selection filter. Similar to mos_q_dequeue, but a message body is not returned and the message is not dequeued. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Find the matching message in the queue (selection filter, if specified) (if no filter, get first item).
  3. Extract attributes of item.

mos_q_ping

Determine if a queue with a given name exists. Its ID is returned, since queues may have aliases, and UUIDs are unique. This is a simplified version of mos_q_retrieve_attr, but optimized for just determining if a queue is there. It is indended for the API's "attach" call. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Return its ID.

mos_q_get_id_list

Return a list of IDs of all messages on a queue. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Return array of UUIDs of items.

mos_q_delete_queue

Delete a queue. Steps:
  1. If there are items in the queue and the flag saying what to do about queue items says to delete them, delete the items.
  2. Remove the queue information from the queue-info backing store and in-memory list of queues.

mos_q_get_attr_by_name and mos_q_get_attr_by_id

Return attributes of the queue described by the given name or UUID. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Extract and return attribute data.

mos_q_set_attr_by_name and mos_q_set_attr_by_id

Set attributes of the queue described by the given name or UUID. Steps:
  1. Find the matching queue in the in-memory list of queues.
  2. Replace the writable attributes in the in-memory list.
  3. Write to the queue-info backing store, if persistent.

mos_q_move_item

Move a single message from one queue to another. Both queues must be on the queue manager. Steps:
  1. Ensure both queues and the message item exist
  2. Remove the item from one queue, adding to the other
  3. Change the item's attribute that says to what queue it belongs
  4. Write message to backing store, if persistent.

Startup

Yes, more needed here.

Steps: read queue-info backing store, populating in-memory list of queue information; read queue-data backing store, populating queue items (these are the persistent messages).

In production dced will start the queue manager. dced will maintain startup parameters, such as directory, principal, etc.


[ Detailed Design TOC | Messaging ]