Data Connectors

APIs for creating and manipulating data connectors

Connectors are a main component of the BitBroker system. You will find more details about connectors, within the key concepts section of this documentation.

Connectors are always created within the context of housing entity types, which can be created and manipulated using other part of this API, described earlier in this documentation.

In order to use the sample calls in this section, first create the housing entity as outlined in the previous section.

In our sample calls, we use the standard server name and port format outlined for local installations and demos. If you have an alternative API host base URL, then enter it in the box below to update all the sample calls on this page :

Your API Host Base URL

All API calls in BitBroker require authorization. The sample calls below contain a placeholder string where you should insert your coordinator API authorization token. If you already have a token, enter it in the box below to update all the sample calls on this page:

Your Coordinator API Authorization Token

Creating a New Connector

New connectors can be created by issuing an HTTP/POST to the /entity/:eid/connector/:cid end-point.

Connectors are always created within the context of a housing entity type and, hence, you must know its ID (eid). In order to create a connector, you must select a unique connector ID (cid) for it.

curl http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia \
     --request POST \
     --include \
     --header "Content-Type: application/json" \
     --header "x-bbk-auth-token: your-token-goes-here" \
     --data-binary @- << EOF
     { \
         "name": "Wikipedia", \
         "description": "Country information from Wikipedia", \
         "webhook": "http://my-domain.com/connectors/1", \
         "cache": 0 \
     }
EOF

This will result in a response as follows:

HTTP/1.1 201 Created
Location: http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia

The body of this response will contain the connector ID and connector authorization token, which the new data connector should utilize to make its data contributions. For example:

{
    "id":"9afcf3235500836c6fcd9e82110dbc05ffbb734b",
    "token":"ef6ba361-ef55-4a7a-ae48-b4ecef9dabb5.5ab6b7a2-6a50-4d0a-a6b4-f43dd6fe12d9.7777df3f-e26b-4f4e-8c80-628f915871b4"
}

It is expected that the coordinator user will securely distribute the connector ID and authorization token to the operator of the new data connector.

Connector authorization tokens are not stored within the system. If you lose this connector authorization token, you will be forced to delete and recreate the connector to obtain a new one. This can have major implications for an operating instance.

The following validation rules will be applied to the body of a new connector request.

Attribute	Necessity	Validation Rules
`cid`	required	String between 3 and 32 characters long Consisting of lowercase letters, numbers and dashes only Starting with a lowercase letter Conforming to the regex expression `^[a-z][a-z0-9-]+$`
`name`	required	String between 1 and 64 characters long
`description`	required	String between 1 and 2048 characters long
`webhook`	optional	String between 1 and 1024 characters long Must conform to URI format
`cache`	optional	Integer between 0 and 31536000

Details about the meaning of these attributes can be found within the key concepts section of this documentation.

Connector IDs are required to be unique across an operating BitBroker instance.

Connector who present no webhook are required to submit complete entity instance records to the catalog. Before making the decision as to whether to host a webhook or not, careful note should be taken on the distinction between data vs metadata within the catalog.

You should choose your connector ID (cid) with care, as these cannot be changed once created.

Updating a connector

Existing connectors can have their profile updated by issuing an HTTP/PUT to the /entity/:eid/connector/:cid end-point.

In order to update a connector, you must know the ID of its housing entity type (eid) and it’s connector ID (cid).

curl http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia \
     --request PUT \
     --include \
     --header "Content-Type: application/json" \
     --header "x-bbk-auth-token: your-token-goes-here" \
     --data-binary @- << EOF
     { \
         "name": "Wikipedia", \
         "description": "A new description for the Wikipedia connector", \
         "webhook": "http://my-domain.com/connectors/1", \
         "cache": 0 \
     }
EOF

This will result in a response as follows:

HTTP/1.1 204 No Content

The validation rules for updated connector information, are the same as that for creating new connectors.

List of Existing Connectors

You can obtain a list of all the existing connectors housed within a parent entity type by issuing an HTTP/GET to the /entity/:eid/connector end-point.

curl http://bbk-coordinator:8001/v1/entity/country/connector \
     --header "x-bbk-auth-token: your-token-goes-here"

This will return a JSON array as follows:

[
    {
        "id": "wikipedia",
        "url": "http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia",
        "name": "Wikipedia",
        "description": "A new description for the Wikipedia connector"
    }
    // ... other connectors here
]

Each connector, housed within a parent entity type, will be returned within this array. Note: There is currently no paging on this API.

Details of an Existing Connector

You can obtain the details of an existing connector by issuing an HTTP/GET to the /entity/:eid/connector/:cid end-point.

In order to obtain details of a connector, you must know the ID of its housing entity type (eid) and it’s connector ID (cid).

curl http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia \
     --header "x-bbk-auth-token: your-token-goes-here"

This will return a JSON object as follows:

{
    "id": "wikipedia",
    "url": "http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia",
    "name": "Wikipedia",
    "description": "A new description for the Wikipedia connector",
    "entity": {
        "id": "country",
        "url": "http://bbk-coordinator:8001/v1/entity/country"
    },
    "contribution_id": "9afcf3235500836c6fcd9e82110dbc05ffbb734b",
    "webhook": "http://my-domain.com/connectors/1",
    "cache": 0,
    "is_live": false,
    "in_session": false
}

Connector authorization tokens are not stored in an accessible way within the system and hence not present in this returned document.

Other sections of this document will explain what the is_live and in_session attributes refer to.

Promoting a Connector to Live

Connectors can be promoted to live status by issuing an HTTP/POST to the /entity/:eid/connector/:cid/live end-point.

In order to promote a connector, you must know the ID of its housing entity type (eid) and it’s connector ID (cid).

curl http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia/live \
     --request POST \
     --include \
     --header "x-bbk-auth-token: your-token-goes-here"

This will result in a response as follows:

HTTP/1.1 204 No Content

When getting details for promoted connectors, their is_live status will be reflected in the corresponding attribute:

{
    "id": "wikipedia",
    "url": "http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia",
    "name": "Wikipedia",
    "description": "A new description for the Wikipedia connector",
    "entity": {
        "id": "country",
        "url": "http://bbk-coordinator:8001/v1/entity/country"
    },
    "contribution_id": "9afcf3235500836c6fcd9e82110dbc05ffbb734b",
    "webhook": "http://my-domain.com/connectors/1",
    "cache": 0,
    "is_live": true,
    "in_session": false
}

If you attempt to promote a connector which is already live, it will still result in an HTTP/1.1 204 No Content response. Such requests are benign and will not impact the connector’s status.

Promoting connectors will cause all the staged records which they have contributed to the Catalog to become visible for calls to the Consumer API.

Demoting a Connector from Live

Existing connectors can be demoted from live status by issuing an HTTP/DELETE to the /entity/:eid/connector/:cid/live end-point.

In order to demote a connector, you must know the ID of its housing entity type (eid) and it’s connector ID (cid).

curl http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia/live \
     --request DELETE \
     --include \
     --header "x-bbk-auth-token: your-token-goes-here"

This will result in a response as follows:

HTTP/1.1 204 No Content

When getting details for such users, their is_live status will be reflected in the corresponding attribute:

{
    "id": "wikipedia",
    "url": "http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia",
    "name": "Wikipedia",
    "description": "A new description for the Wikipedia connector",
    "entity": {
        "id": "country",
        "url": "http://bbk-coordinator:8001/v1/entity/country"
    },
    "contribution_id": "9afcf3235500836c6fcd9e82110dbc05ffbb734b",
    "webhook": "http://my-domain.com/connectors/1",
    "cache": 0,
    "is_live": false,
    "in_session": false
}

If you attempt to demote a connector which is not live, this will still result in an HTTP/1.1 204 No Content response. Such requests are benign and will not impact the connector’s status.

Demoting connectors will remove all the records which they have contributed to the Catalog from calls to the Consumer API. Users holding on to full qualified URLs for such records, will instead receive HTTP/1.1 404 Not Found responses.

Deleting a connector

Existing connectors can be deleted from the system by issuing an HTTP/DELETE to the /entity/:eid/connector/:cid end-point.

In order to delete a connector, you must know the ID of its housing entity type (eid) and it’s connector ID (cid).

curl http://bbk-coordinator:8001/v1/entity/country/connector/wikipedia \
     --request DELETE \
     --include \
     --header "x-bbk-auth-token: your-token-goes-here"

This will result in a response as follows:

HTTP/1.1 204 No Content

Deleting a connector will also remove all the records which it has contributed to the Catalog.

Deleting a connector will also remove its ability to contribute data. No further contribution will be accepted and all subsequent calls to the Contributor API will fail.

Any policy authorization tokens which were issued where this connector’s records formed part of the data segment, will no longer return entity instances contributed by it. In some circumstances, this could render policy authorization tokens unfit for purpose.