Master Overview: Custom API Integrations

HOW TO

How to connect Custom API Integrations to Databox

IN THIS SECTION

How to connect Custom API Integrations to Databox

Connecting a Custom API integration to Databox is no different than connecting one of our native integrations. For details on the connect flow of a specific Custom API integration, please refer to the dedicated Custom API integration’s help article:

New integrations are added regularly, and we'd love to hear what you’d like to see next. You can request a new Data Source integration here. 

What's the maximum amount of historical data initially available when you use a new Custom API Metric in Databox?

The setup of your Custom Metrics dictates the amount of data we are able to fetch and display in Databox. 

The amount of historical data you are able to see in Databox depends on the instructions you give us on how we should prepare API requests to retrieve the data for your Custom Metric. Those instructions may be impacted by limitations of the API itself (which we do not have control over) and potential limitations with our integration related to timestamps, dynamic date/time parameters, and pagination. 

If the API endpoints accept date/time parameters, you can choose to add dynamic date/time parameters to your requests. This allows you to simulate a rolling date range. Learn more here.

If the API endpoints allow pagination, you can influence the amount of data returned in each sync by setting this when building the custom metric. There is a limit in Databox that restricts pagination up to 100 pages. Learn more here.
Custom Metrics are synced hourly for Professional and Performer customers. More historical data accumulates over time as we continue to make regular syncs for the metric. 

What time zone does Custom API Integration data sync in? 

API settings determine the time zone that your Custom API data syncs in. 

In some cases, APIs return your data in a time zone you have set in the app itself. In other cases, APIs process user's data exclusively in UTC and do not allow users to adjust this setting. 
If the API supports it, you can adjust the time zone your Custom API data syncs with Databox in by following these steps.

How to Create Custom Metrics

IN THIS SECTION

How to use the Wizard vs Manual setup

The Metric Builder for Custom API Integrations allows you to choose between the Wizard or Manual Setup. It is recommended that you use the Wizard to provide more guidance and make it even easier to build Custom Metrics from your Data Source. 

The Wizard presents you with a Prepare API Request form that enables you to prepare and execute API requests. Once you execute an API request, you are presented with an API response or, if the request failed, an error returned from the API. The Wizard then guides you through the selection of a metric value, dimension, and date for the metric. Behind the scenes, we’re using the information you provide to build JSONata expressions. You can then manually review and customize the JSONata expression and save your custom metric

If you’d rather create the JSON expression yourself without our guidance, you can do so by selecting Switch to Manual Setup on the Prepare API Request page. 

How to access the Metric Builder

1
   Navigate to  Metrics > Custom Metrics to access the Metric Builder
2
  Click the green + Create Custom Metric button and select your connected Custom API Integration from the Data Source drop-down list.

How to prepare API Requests

IN THIS SECTION

How to prepare API requests

In order to prepare API requests, you should reference API-specific documentation. This can be found in the Databox through the API Docs link in the top right of the Prepare API Request page.

1
  Select GET or POST as the HTTP Method. This is dependent on the information you found in the API documentation.
2
 Enter the relevant endpoint in the text field next to the base URL.
Pro Tip: Whenever possible base URLs do not include API versions. This allows you to use different API versions. However, it’s possible that credentials you provide when connecting your Data Source limit the use of certain API versions. Learn more about how we handle versioning for Custom API Integrations here.
3
  Add Parameters based on your HTTP Method. Parameters limit the records returned for the API request.
  • If you’re creating a GET request, add the relevant Key and Value pair(s). The API documentation provides information on endpoints for a field/ argument, which will be the Key. The condition (i.e., an ID, true/ false, etc.) will be the Value. 
    • If the API documentation for the Custom API Integration states that the Integration supports dynamic date ranges, you will be able to click the calendar icon to add datetime parameters. 
  • If you’re creating a POST request, define the relevant JSON body.
    • If the API documentation for the Custom API Integration states that the Integration supports dynamic date ranges, you will be able to click the calendar icon to add datetime parameters. 
4
 You can add Headers that include Key and Value pairs if the API needs this to successfully return data.
All the API request information you provide is saved to the Metric Settings for the Custom Metric. This allows us to regularly sync the appropriate data from the API, transform it as needed, save it to our database, and make it available in your Databox Account. 

Pagination information

Pagination means that records are returned on multiple pages and in order to retrieve each page additional API requests have to be made. 

If the API supports pagination, you should select a valid pagination type within the Pagination dropdown. When this setting is selected, we make two API requests in order to ensure we’re able to access multiple pages of data. 

If the first API request is successful but the second fails, pagination is not set correctly, and you are not able to proceed with the metric setup. Update the Pagination selection and try again.

You can increase or decrease the number of results returned by including the appropriate parameter in the request (limit, per_page, etc.). 

Pagination capabilities are dependent on the API's default limits and settings in Databox. Currently, the pagination limit for Custom API Integrations is set to 100 max. REST API Data Source Connections do not currently support pagination. Learn more about REST API Data Source Connections here.

Pro Tip: When you enable pagination, we automatically prepare the logic for the first and all subsequent API requests. This includes adding “page”, “next link”, “cursor”, “offset” and similar API request parameters needed to get data for up to 100 pages. If you include any of these parameters in the API request itself, you should not enable pagination. Currently, we only support pagination starting from the first page. 
If Page is selected as the pagination type, we add a “page” parameter to the API request and increment its value by 1 for each subsequent page (up to 100 pages). 

If Next is selected as the pagination type, we extract the “next” page link from the previous API response and use it to prepare the API request for the next page. 

If Cursor is selected as the pagination type, we extract “cursor” from the previous API response and add it to the API request for the next page.

If Offest or Seek are selected as the pagination type, we take over the logic preparation of API requests. You should not add pagination parameters to the API requests. 

Authentication of API requests

We automatically add relevant authentication parameters as parameters or headers in API requests. You will see authentication parameters as disabled key/ value pairs in the API request. 

Dynamic date/time parameter information

Most APIs allow date/ time parameters in API requests, so we include an option to set up dynamic date/ time parameters when building your Custom Metrics. 

However, dynamic date parameters are not supported within the JSON body of the POST API requests.

To transform a parameter to a dynamic date/ time parameter, click on the Calendar icon within the Key input field. You’ll then be presented with two fields: Datetime and Datetime format. Be sure to review relevant API documentation to get information on which date/time format you can use for specific API requests. 

Pro Tip: If the wrong format is selected for a specific API request, the API will respond with an error.

Based on your selection we are dynamically setting date parameters when making API requests for your custom metric, which means that each next day the date increases for one day.

For example, if you set “startTime” to 1 month ago and “endTime” to today, you will simulate a rolling time frame. We’ll continuously accumulate data for the metric.

How to select a metric, value, date and dimension

After you prepare and execute the API request, you’re presented with the API response in JSON format. The Wizard will guide you through selecting a metric (numerical) value, (optional) date, and (optional) dimension. Suggested selections are provided as you work through this flow.

Based on your selections, we’ll automatically generate a JSONata expression to extract the appropriate data. 
Pro Tip: If an API request fails, you will see a raw API error. You should recheck the API request and, once you resolve the issue, execute it again.

All of this information is saved to the Metric Settings for the Custom Metric. This allows us to regularly sync the appropriate data from the API, transform it as needed, save it to our database, and make it available in your Databox Account

How to preview & customize your Custom Metric

On the Preview & Customize page, you are once more presented with an API response and automatically generated JSONata expression. You can now review and manually customize the JSONata expression and Save your custom metric. 

Click Preview data button to run the JSONata expression on your API response and extract your data.

JSONata expression

JSONata is a powerful JSON query and transformation language that allows you to extract data from data in JSON format. Check JSONata’s official documentation for all the tools it offers.

JSONata expressions give Databox instructions on how to extract your data from API responses. The result should be either an object or an array of objects. The object will always contain a “value” (numerical). “Date” and “dimension” data is optional. If included, the “date” should be either a string or a numeric value (for unix seconds or milliseconds), and the “dimension” should be a numeric value, a boolean or a string.

We have covered all possible date formats so you can select any date from the API response. We do not recommend defining date values in the JSONata expression on your own unless there is no option for you to select a date from the API response, keep in mind that the date value should be formatted in a way that’s specific to the response of the given API. Review API documentation for more information on which date formats are supported by the API

If you prepare a JSONata expression that does not result in a demanded data structure, you will be presented with validation errors and will not be able to proceed with the metric setup until you fix the JSONata expression.

How to complete the setup of your Custom Metric

IN THIS SECTION

Describe Metric

The Describe Metric screen allows you to set the name of the Custom Metric, define the basic data aggregation, and set the trend and format. 

It’s imperative that you set up valid data aggregation and metric types. By default, the data aggregation and data type are set to Latest and All Values. The 2 major data types that can be sent to Databox are the Total/ Current Data Type and the Event Data Type. Learn more about Data Aggregation here.

Select Advanced to define additional elements for the Custom Metric. 

Data Aggregation and Data Types

Data Aggregation and Data Type instruct the Databox system on how to handle the data for your custom metric. 

Learn more about Data Types here.
Total/Current Data Type

Aggregation function: Latest

Data type: All values

You should use this setup when your metric reflects the current total value for the metric. In Databox, the latest entry within the interval will be displayed as a total value. 

For example, on a chart for “Month-to-Date,” the latest value within the month will be displayed as the total number above the chart and the latest value for each day will be displayed as the daily values on the chart. 

Metrics like “Total Followers” and “Current Account Balance” should be set in this format.

Example 1:

Example of API response:

{

“data”: {

“total_followers”: 1345

}

}

Example of JSONata expression:

{

“value”: data.total_followers

}

Event Data Type

Aggregation function: SUM [Default], MIN, MAX, AVG, Count

Data type: All values

You should select your desired aggregation function along with “All values” as the Data Type when you are storing individual values for the metric at the specified date/timestamp. For longer date ranges with multiple data points, individual metric values will be aggregated for the entire date range and the aggregated value will be displayed for the metric. Examples of the metrics that should be set like this are “Sales”, “Views”, “New subscriptions”, “Cancellations”.
Metrics like “Sales Closed Won,” “New Subscriptions,” and “Pageviews” should be set in this format. 

You can choose from the following aggregation functions:

  • SUM [default]: display the aggregated total (sum) of sent data for the Date Range.
  • MIN: display the lowest (minimum) sent value for the Date Range.
  • MAX: display the highest (maximum) sent value for the Date Range.
  • AVG: display the average of sent data for the Date Range.
  • COUNT: display the number (count) of sent for the Date Range.
When an API response contains values with identical timestamps, you should always explicitly define if you would like us to Override or Append the values for identical timestamps using the Handle multiple values for identical date/ times setting.  
Choosing Override will result in the values from the last fetch always overriding the previously stored values for a given timestamp. The Append option will result in the values from each fetch being appended to the previous values stored for identical timestamps.
Example 1:

Aggregation function: SUM

Data type: All values

You should use this setup when you would like to prepare a metric from values with unique timestamps. In the example, below each post has a unique “created_at” timestamp and the number of “likes”. This setup allows you to visualize the sum of Likes for the intervals containing both “created_at” timestamps.
See Example 2. In the “Override” case the value stored to the same “created_at” timestamp on our side will be [45, 13], since values from the last page (in this case second page) [45, 13] will override the values from the first page [98, 67]. At the end the metric will show 68 on databoard.
Once you finish the setup you are ready to save the metric and allow us to fetch the data for it periodically. You can always find your custom metrics under Metrics/Metric builder.

Additional Information

IN THIS SECTION

API Versioning

Our Custom API integrations have predefined base URLs, which by default do not include API versions. This was designed to enable you to use various API versions when preparing your Custom Metrics. 
However, there is a possibility that the credentials you use when connecting your Data Source do not allow you to access data from all API versions. This is dependent on the API providers access management for different API versions and will result in you not being able to access data for specific API versions. 

In this case, you can either connect a custom REST API data source or you can request we build a new version of the Custom API Integration as part of a paid service. This will only be possible if the API supports authentication with access tokens retrieved through OAuth authorization.

IN THIS SECTION

Migrating to new API versions

We strive to ensure our Custom API Integrations support the latest API versions, however due to the dynamics of API development it is not always possible to migrate existing Custom Metrics to new APIs. 

If an API provider decides to introduce breaking changes to the current API or to deprecate the current API and replace it with a new one, the migration of Custom Metrics will only be possible if specific prerequisites are met. This includes: 

1
   Unchanged API responses containing values, dates and dimensions on the same paths as in the previous API version and
2
  Logical equivalence of the endpoints. For example, the new endpoints could not contain dynamic fields if the old ones did not contain them and vice versa. 
If the necessary prerequisites are not met and the migration of Custom Metrics to the new API is not possible or reasonable, there is a possibility we will not be able to keep your current custom metrics intact.

If you would like to use an unsupported API version, you can either connect a custom REST API data source or you can request we build a new version of the Custom API Integration as part of a paid service. This will only be possible if the API supports authentication with access tokens retrieved through OAuth authorization.

Updating an APIs base URL

Although updating the base URL would be bad practice by the API provider, it is a possibility and is outside of our control. 

If this happens with any of our Custom API integrations and all the endpoints for the API stay the same, we would be able to simply update the base URL for the Custom API Integration and all Custom Metrics that you already created.

Error handling

IN THIS SECTION

Data Source Errors

When you connect a Custom API Integration, we make a test API call with the credentials you provided to ensure we can make API requests. Once we verify we can make active API requests, the Data Source is active in Databox.

If we receive an error from the API, it indicates that we are no longer able to make API requests with the credentials you provided during the connection flow. As a result, we are no longer able to sync data for your Custom Metrics. This could happen due to permissions being changed, a username/ password being updated, the API key being changed, etc.

This status indicates that we are no longer able to retrieve data from the API. You will need to reconnect the Data Source, so we can continue to fetch your data.

We do not test connection during connecting flow in case you create a custom REST API data source. If you submitted incorrect values when creating a custom REST API Data Source, you will not be able to execute a successful request when preparing API request in the first step of creating your custom metric. To resolve this issue, you should review your Data Source and edit it within the Data Manager. 

Custom Metric Errors

Due to various reasons that are in the API provider’s control, APIs can respond to API requests with errors.

If you receive an error from an API while creating a Custom Metric, you will not be able to move to the next step. This could be due to a malformed request or an issue on the APIs side. The error message should help guide you to fix the issue.

There is a possibility that errors can be returned from an API after a Custom Metric is already set up and was previously syncing data without issue. This could be due to an issue with the API at the time we made a request, hitting rate limits, changes on the API’s side that demands the editing of Custom Metrics, etc. 

If we receive an error for your Custom Metric that is not related to a Data Source error (like unauthorized access or invalid access tokens), we propagate it to the status of the metric and present it to you. We will show you the raw API response in the Custom Metric’s status, which typically informs you of the reason that API request returned an error.

Please note that although we propagate the error to your Custom Metric, we do not stop fetching the data for it unless we receive an error at the Data Source level. Based on the content of the error received, you should decide if it is related to a temporary API issue or the issue with the request you prepared for a given Custom Metric. 

While temporary errors will be cleared with the next successful fetch for the given metric, you should edit the metric if the error message indicates that the issue is with the API request. 

Risks of editing Custom Metrics

It’s important to note that editing Custom Metrics could result in completely different context for your metric. This may result in the historical data that was previously synced for the metric not fitting in the new context. Because of this, we advise you to always purge historical data when editing critical parts of your Custom Metrics

Rate Limits

Currently we do not handle rate limiting when making requests to APIs on your behalf.

This means that if you attempt to sync more data than is permitted by the API, you may receive a rate limit error. These errors will be presented to you as Custom Metric Errors. 
You should reference Databox’s Guide for the Custom API Integration to understand the rate limit of the API. 

Combining data from multiple endpoints

Currently, we do not support Custom Metrics where you combine data from multiple endpoints. This means each Custom Metric can only sync data from a single API endpoint. 

If you would like to combine data from multiple endpoints, you should prepare separate metrics and use Databox’s Data Calculations tool.  

Dependent API requests

Many APIs require that you insert specific parameters into requests that must first be retrieved via another API request. We do not currently support this out-of-the-box, but you can use the Prepare API request form to execute relevant API requests and get the desired parameters.

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Related Articles