Hosting a Webhook

How to use webhooks to incorporate live and on-demand data

It is an expectation that the BitBroker catalog contains information which is useful to enable search and discovery of entity instances. Hence, it contains key metadata - but it does not normally contain actual entity data. This is pulled on-demand via a webhook hosted by the data connector who contributed the entity record.

The distinction between data and metadata is covered in more detail in the key concepts documentation. Depending on how data and metadata is balanced in a BitBroker instance, there may or may not be a requirement to host a webhook.

In this section, we will outline how to implement a webhook within a data container.

A quick way to get going integrating a webhook into your own data connector is to adapt the example connectors which have been built for a range of data sources.

It is permitted and valid for one webhook to be servicing the needs of multiple data connectors. Sufficient inbound information will be provided to allow the webhook to be clear about which entity instance data is being request.

Registering your Webhook

The first step is to register your webhook with BitBroker. This is done when the connector is created or can be done later by updating the connector. These actions are part of the Coordinator API and hence can only be performed by a coordinator user on your behalf.

Your webhook should be an HTTP server which is capable of receiving calls from the BitBroker instance. You can host this server in any manner you like, however the coordinator of your BitBroker may have their own hosting and security requirements of it.

You need to maintain your webhook so that it is always available to its connected BitBroker instance. If your webhook is down or inaccessible when BitBroker needs it, this will result in a poor experience for consumers using the Consumer API. In this scenario, they will only see partial records. Information about misbehaving data connectors will be available to coordinator users.

Required End-points

You are required to implement two end-points as part of your webhook deployment.

Whilst BitBroker advertises its own key space to its consumers, there is no need for data connectors to take heed of these. They can continue to concern themselves with only their own domain key space. When BitBroker makes requests of your webhook, it will only ever use its own key space.

Whenever your webhook is called, it will be in the context of an on-demand request - meaning that the call is in the direct line of response to a waiting user of the Consumer API. Hence, you should endeavor to respond to webhook calls in a timely manner. Information about poorly performing data connectors will be available to coordinator users.

Entity End-point

The entity end-point is used by BitBroker to get a full data record for an entity instance which you previously submitted into the catalog.

The entity end-point has the following signature:

   HTTP/GET /entity/:type/:id

Where:

Attribute	Presence	Description
`type`	always	The entity type ID, for this entity instance
`id`	always	Your own domain key, which you previously submitted into the catalog

The entity type is presented here to allow for scenarios where one webhook is servicing the needs of multiple data connectors.

In response to this call, you should return a JSON object consisting of an entity and instance attribute only - all other attributes will be ignored. The object you return will be merged with the catalog record, which you provided earlier. Hence, there is no need to resupply the catalog information you have already submitted in previous steps.

For example, consider this (previously submitted) catalog record:

{
    "id": "GB",
    "name": "United Kingdom",
    "type": "country",  
    "entity": {
        "area": 242900,
        "calling_code": 44,
        "capital": "London",
        "code": "GB",
        "continent": "Europe",
        "currency": {
            "code": "GBP",
            "name": "Sterling"
        },
        "population": 66040229
    },
    "instance": {
        "independence": 1066
    }
}

If there is a call for the detail of this record made on the Consumer API, the system will callback on the entity end-point as follows:

HTTP/GET /entity/country/GB

Then the webhook should respond with any extra / live / on-demand entity and instance data:

{
    "entity": {
        "inflation": 4.3
    },
    "instance": {
        "temperature": 18.8
    }
}

The system will then merge this live information with the catalog record to send a combined record to the consumer.

{
    "id": "GB",
    "name": "United Kingdom",
    "type": "country",
    "entity": {
        "area": 242900,
        "calling_code": 44,
        "capital": "London",
        "code": "GB",
        "continent": "Europe",
        "currency": {
            "code": "GBP",
            "name": "Sterling"
        },
        "population": 66040229,
        "inflation": 4.3  // this has been merged in
    },
    "instance": {
        "independence": 1066,
        "temperature": 18.8  // this has been merged in
    }
}

Timeseries End-point

The timeseries end-point is used by BitBroker to get a timeseries information associated with an entity instance previously submitted into the catalog.

Not all entity type will have timeseries associated with them. When they do, then this callback is vital, since no timeseries data points are held within the catalog itself. Only the existence of timeseries and key metadata about them is stored.

The timeseries end-point has the following signature:

HTTP/GET /timeseries/:type/:id/:tsid?start=:start&end=:end&limit=:limit

Where:

Attribute	Presence	Description
`type`	always	The entity type ID, for this entity instance
`id`	always	Your own domain key, which you previously submitted into the catalog
`tsid`	always	The ID of the timeseries associated with this entity instance
`start`	sometimes	The earliest timeseries data point being requested When present, an ISO 8601 formatted date
`end`	sometimes	The latest timeseries data point being requested When present, an ISO 8601 formatted date
`limit`	always	The maximum number of timeseries points to return An integer greater than zero

Further information about the possible URL parameters supplied with this callback are:

Attribute	Information
`start`	Should be treated as inclusive of the range being requested When not supplied, assume a `start` from the latest timeseries point
`end`	Should be treated as exclusive of the range being requested When present, this will always after the `start` Never present without `start` also being present When not supplied, defer to the `limit` count
`limit`	Takes precedence over the `start` and `end` range The `end` may not be reached, if `limit` is breached first

Then the webhook should respond timeseries data points as follows:

[
    {
        "from": 1910,
        "to": 1911,
        "value": 5231
    },
    {
        "from": 1911,
        "to": 1912,
        "value": 6253
    },
    // other timeseries points here
]

Where:

Attribute	Necessity	Description
`from`	required	An ISO 8601 formatted date
`to`	optional	When present, an ISO 8601 formatted date
`value`	required	A valid JSON data type or object

You should return your timeseries points with the latest first. Taking the first item of a returned array, should always represent the latest data point.

Specifying both from and to is rare - in most cases, only a from will be present. You can place any data type which makes sense for your timeseries in the value attribute. But this should be consistent across all the timeseries points you return.