Spread Transport

This page describes the protocol used by the Spread-based transport.

Note

Starting with version 0.9 of RSB, the C++ implementation does no longer include the Spread-based transport in the RSB core. Instead, the transport is available as a plugin.

Schema

See also

URIs

Use of URIs in RSB.

The Spread transport uses the spread URI schema.

Options

See also

Configuration

Specification of configuration mechanism.

The following configuration options are accepted by the Spread-based transport:

Name                   Type     Comment
+ transport
+-- spread
+---- host             string   Name of the host running the Spread daemon
+---- port             uint     Port on which Spread daemon runs
+---- maxfragmentsize  uint     Maximum Spread message size
+---- tcpnodelay       boolean  Implementation detail

Notification Format

Data exchanged on the wire by the Spread-based transport is encoded using Google protocol buffers. A Spread message always contains a fragment of a notification elementary communication unit:

message definitions are hidden

syntax = "proto2";

package rsb.protocol;

import "rsb/protocol/Notification.proto";

option java_outer_classname = "FragmentedNotificationType";

/**
 * A message which wraps the normal Notification and extends it with
 * fragmentation information.
 *
 * This message is, for example, used in the Spread transport. See
 * https://code.cor-lab.org/projects/rsb/wiki/SpreadProtocol for
 * information regarding the event <-> notification transformation in
 * this case.
 *
 * @author Michael Goetting <mgoettin@techfak.uni-bielefeld.de>
 * @author Johannes Wienke <jwienke@techfak.uni-bielefeld.de>
 * @author Jan Moringen <jmoringe@techfak.uni-bielefeld.de>
 */
message FragmentedNotification {

    /**
     * The original notification in some cases with only required
     * fields filled.
     *
     * The first fragment contains the full meta-data set whereas
     * subsequent FragmentedNotification s only carry payload chunks
     * in this field and the event id for association.
     */
    required Notification notification            = 1;

    /**
     * The number of notifications an event was split into.
     *
     * Splitting only happens when message size would have been to big
     * for transmission in a single notification. 1 indicates a
     * self-contained, i.e. not split, notification.
     */
    required  uint32      num_data_parts          = 2 [default = 1];

    /**
     * The index of the current fragment of a fragmented notification.
     *
     * The initial fragment has index 0.
     */
    required uint32       data_part               = 3 [default = 0];

}

syntax = "proto2";

package rsb.protocol;

import "rsb/protocol/EventId.proto";
import "rsb/protocol/EventMetaData.proto";

option java_outer_classname = "NotificationType";

/**
 * The Notification message is the wire format and serialization
 * mechanism used by the RSB Spread transport and other transports.
 *
 * Notification messages contain
 *
 * * event routing information,
 * * event payloads
 * * and optional event meta-data such as timestamps
 *
 * See
 * https://docs.cor-lab.org/rsb-manual/trunk/html/specification-event.html
 * for more information regarding events.
 *
 * @author Sebastian Wrede <swrede@techfak.uni-bielefeld.de>
 * @author Johannes Wienke <jwienke@techfak.uni-bielefeld.de>
 * @author Jan Moringen <jmoringe@techfak.uni-bielefeld.de>
 */
message Notification {

    // Identity, transportation and routing information fields

    /**
     * The unique ID of the event, consisting of the unique id of
     * sending participant and the sequence number of the event.
     *
     * The odd field number is a historical artifact.
     */
    required EventId     event_id               = 108;

    /**
     * The scope to which the encoded event was sent.
     */
    optional bytes      scope                   = 6;

    /**
     * An ASCII-string designating the role of this event within a
     * communication pattern.
     *
     * Two such roles could be "REQUEST" and "REPLY" in RPC
     * communication.
     *
     * Note: The bytes type has been chosen to avoid UTF-8 de- and
     * encoding.
     */
    optional bytes      method                  = 14 [default = ""];

    // Data fields

    /**
     * An ASCII-string designating the wire schema with which the
     * event payload (in the "data" field) has been serialized.
     *
     * Note: The bytes type has been chosen to avoid UTF-8 de- and
     * encoding.
     */
    optional bytes      wire_schema             = 7;

    /**
     * The serialized payload of the event.
     */
    optional bytes      data                    = 9;

    // A collection of id of other events which have in some way
    // caused this event to be sent.
    repeated EventId    causes                  = 13;

    /// Meta-data fields

    // meta-data storage
    optional EventMetaData meta_data            = 15;
}

Fragmentation

Because Spread has a message size limit, a single notification may not be sufficient to transport a whole event when the payload is large. Hence, event s may be encoded in several FragmentedNotification s which are sent sequentially. Multiple FragmentedNotification objects are constructed according to the following rules:

• The size of individual fragments (i.e. serialized size of the FragmentedNotification objects) must not exceed 100,000 octets

• Fragment numbers are in the range [0, NUMBER-OF-FRAGMENTS - 1]

• The fields which differ among FragmentedNotification s for one event are:

  • The number of the fragment (FragmentedNotification.data_part field)

  • The payload (FragmentedNotification.notification.data field)

  • Which fields of the Notification embedded in FragmentedNotification.notification are present (see below)

• Each of the FragmentedNotification objects contains a Notification object

  • The event id (Notification.event_id field) is always present (to specify the event to which a fragment belongs)

  • For the initial fragment (fragment number 0), all fields of the embedded Notification object are present

  • For subsequent fragments (fragment number >= 1), only the following fields of the embedded Notification object are present:

    • Notification.event_id

    • Notification.data

Hierarchical Bus

The hierarchical bus is created by sending each message to multiple Spread-groups: a Spread-group corresponding to its scope as well as Spread-groups corresponding to all superscopes including the root-scope (/) (In Spread terminology this is called “multigroup multicast”). Spread’s protocol takes care of only transmitting the message to interested recipients and transmitting it only once to each of them.

Example for scope /foo/bar/:

super-scopes(/foo/bar/, include-self? = yes) = /, /foo/, /foo/bar/

Group Names

Group names are created by applying the following steps to the fully formal scope string representation (including trailing slash):

1. Hash the scope string using MD5.

2. Convert the 16 bytes of output to a 32 character string by concatenating the zero-padded hex (base-16) representations of the individual bytes. Letters of the hex representation have to be lower case.

3. Remove the final character of the hex representation of the hash. (Since Spread group names can only be 32 bytes long including the 0-terminator)

Example:

/         -> "6666cd76f96956469e7be39d750cc7d\0"
/foo/     -> "4f87be8f6e593d167f5fd1ab238cfc2\0"
/foo/bar/ -> "1c184f3891344400380281315d9e738\0"

Quality of Service

The following table explains how the 2D RSB QoS settings are mapped to Spread message types:

Quality	unreliable	reliable
unordered	UNRELIABLE_MESS	RELIABLE_MESS
ordered	FIFO_MESS	FIFO_MESS

Implementations

Language	File(s)
C++	“1.0” branch of https://github.com/open-rsx/rsb-spread-cpp at src/rsb/transport/spread
Java	“1.0” branch of https://github.com/open-rsx/rsb-java at src/rsb/transport/spread
Python	upstream/rsb-python/rsb/transport/rsbspread/__init__.py
Common Lisp	“1.0” branch of https://github.com/open-rsx/rsb-cl at src/transport/spread