About plugins

rtd.pub uses a plugin system to connect different data sources to Excel.

Each plug-in is a small program that implements a gRPC service definition used to stream data into Excel, via the plugin host. The plugin host is responsible for spawning a plugin subprocess. Plugins can be written in go and Python.

Terminology A plugin is an executable that provides data. A connector is a named, reified instance of a plugin, configured and ready to use. This will become clearer in the examples below.

Subscription cells take the connector name the first parameter, a subject/topic string as the second, and then up to 8 more string parameters after that. If you wanted to observe data through a configured NATS connector, you would use

=rtdpub.Sub("nats", "hello.world")

Plugins can expose configuration variables. Check the documentation for each connector. For example, the NATS plugin yields a connector configured to connect to the NATS server nats.mycorp.com with the following configuration.

rtdpub:
  plugins:
    path: ./plugins
    connectors:
      prices:
        uses: nats
        properties:
          natsServer: nats.mycorp.com:4222

Multiple connectors are common:

rtdpub:
  plugins:
    path: ./plugins
    connectors:
      prices.dev:
        uses: nats
        properties:
          natsServer: nats-dev.mycorp.com:4222
      prices.prod:
        uses: nats
        properties:
          natsServer: nats.mycorp.com:4222
          auth:
            type: user-credentials
            file: arthur.creds
      clock:
        uses: clock.py
        properties:
          tz: bst
          tick: 1s

This configuration exposes some connectors: NATS for prod and dev and a "time of day" plugin. All are accessed by the same Excel add-in.

Note that prices.prod and clock have additional configuration within properties which gets passed to the plugin when the connector is created.

Where do connectors run?

Connectors run alongside the rtd.pub add-in as separate, local processes, each connecting out to the ultimate source such as a NATS server, Kafka cluster or external data vendor.

The Excel add-in connects to a single plugin host process, which supervises the plugin processes and aggregates their output. Local IPC is done through Unix sockets or named pipes where available. gRPC (so therefore protobuf) is used as the basis of the protocol.

In environments with a lot of users connecting to the same external sources, the external bandwidth is multiplied by the number of users. It is better to bring this data in once and fan out within your local network.

This is only an issue with high velocity data and moderate numbers of users, but shared connectors are planned for the future.

Real world example

The plugin shown in the video below uses the polygon.io stocks websocket API (we have no affiliation). This plugin consists of a small amount of go code as we were able to leverage their existing go client library.