About integrations

rtd.pub relies on a plug-in system to connect different data sources to Excel. Each plug-in is a small program that implements a gRPC service definition used to stream data directly into Excel, via the add-in. It can be written in any language that supports gRPC, but we make SDKs available for go and Python to hopefully make usage more idiomatic in those languages.

Terminology A plugin is an executable that provides data. A connector is a named, reified instance of a plugin, configured and ready to use. This will become clearer in the examples below.

Subscription cells take the connector name the first parameter and then an arbitrary number of string parameters. If you wanted to observe data through a configured NATS connector, you would use

=rtdpub.Sub("nats", "hello.world")

Plugins may expose configuration variables. Check the documentation for each connector. For example, the NATS plugin yields a connector configured to connect to the NATS server nats.mycorp.com with the following configuration.

rtdpub:
  plugins:
    path: ./plugins
    connectors:
      prices:
        uses: nats
        properties:
          natsServer: nats.mycorp.com:4222

Multiple connectors are common:

rtdpub:
  plugins:
    path: ./plugins
    connectors:
      prices.dev:
        uses: nats
        properties:
          natsServer: nats-dev.mycorp.com:4222
      prices.prod:
        uses: nats
        properties:
          natsServer: nats.mycorp.com:4222
          auth:
            type: user-credentials
            file: arthur.creds
      clock:
        uses: clock.py
        properties:
          tz: bst
          tick: 1s

This configuration exposes some connectors: NATS for prod and dev and a "time of day" plugin. All are accessed by the same Excel add-in.

Note that prices.prod and clock have additional configuration within properties which gets passed to the plugin when the connector is created.

Where do connectors run?

Connectors run alongside the rtd.pub add-in as separate, local processes, each connecting out to the ultimate source such as a NATS server, Kafka cluster or external data vendor.

The Excel add-in connects to a single plugin host process, which supervises the plugin processes and aggregates their output. Local IPC is done through Unix sockets or named pipes where available. gRPC (so therefore protobuf) is used as the basis of the protocol.

In environments with a lot of users connecting to the same external sources, the external bandwidth is multiplied by the number of users. It is better to bring this data in once and fan out within your local network. This is only an issue with high velocity data, but shared connectors are planned for the future.