Data sync, storage, and security in Databox

Understanding what happens behind the scenes in a data platform is crucial for any company relying on accurate, secure data. This knowledge helps businesses avoid discrepancies, ensure timely updates, and mitigate data security risks. Ultimately, it empowers teams to make confident, data-driven decisions while safeguarding critical business information.

Data pipelines

The diagram below illustrates how Databox processes data to provide you with valuable insights. Think of it as a journey where your raw data is refined into meaningful metrics.

1. Data collection

The process begins by connecting to an integration and selecting an available data source. Once the connection is made, data is collected in one of two ways, depending on the source and available mechanisms:

• API (Application Programming Interface): Used to retrieve pre-aggregated values or, when local replication isn’t supported or available, pull raw data directly from the provider and calculate it on demand.
• Replication: A local copy of the data source is created and maintained to be used for metric creation, drill-down analysis, and further insights.

2. Data preparation

Next, the data is prepared for analysis. This step involves organizing, cleaning, filtering, and merging data into a structured format suitable for calculations and metric creation. This is done via datasets, i.e. custom tables where each row represents an individual record and each column defines a specific attribute of that record.

3. Metric creation

At this stage, the criteria for calculating the metric is defined. The metric builder tool is used for this, working with APIs or datasets as available. In the metric builder, you select the measure or metric, date, and optionally a dimension. Filtering options are available, and in some cases, you can customize the aggregation function (e.g., sum, avg, min, max).

4. Visualization

Once metrics are created, they can be added to Databoards, reports, and notifications. These metrics can also be used for forecasting, identifying correlations, and detecting anomalies. The numbers you see are calculated based on the saved criteria and selected date range, with further modifications possible (e.g., calculating cumulative totals instead of discrete period values).

Sync frequency

The Databox system uses scheduling to manage data synchronization efficiently. Each dataset or metric is assigned a schedule when it is first used within the application, and the schedule remains active until the dataset or metric is no longer in use.

The frequency of data collection depends on several factors:

• Subscription Plan: Determines the maximum refresh rate under ideal conditions.
• External Limits: Some data providers impose restrictions on how often or how much data can be retrieved.
• Data Source Settings: The selected update frequency when adding or editing a data source.
• Account Activity: If an account remains inactive for an extended period, the data refresh rate may be progressively reduced.

In addition to regular synchronization, schedules are created to ensure fresh data is available before notifications are sent. These notification-related schedules run independently of in-app activity but remain subject to the same conditions outlined above.

Data storage and security

All Databox products operate on best-in-class Amazon AWS servers located in the United States, ensuring that customer data remains secure and never leaves the data centers. We adhere to enterprise-grade security standards and the highest encryption protocols to protect customer data at all times.

Data encryption and protection

• Data in transit is encrypted using TLS 1.2 or higher.
• Data at rest is secured with AES-256 encryption, with key management handled via AWS Key Management Service.
• Backups are managed using AWS Cloud Object Storage and Cold Storage.

Monitoring and threat detection

• Databox continuously monitors server and workstation infrastructure to maintain a comprehensive security overview.
• Administrative access, privileged commands, and system calls on production servers are logged and actively monitored.
• Automated log analysis detects potential issues and alerts the appropriate personnel.

Ongoing security practices

Annual internal and external application security testing is conducted, including code reviews, penetration testing, and automated vulnerability scanning integrated into the CI/CD pipeline. This ensures continuous monitoring of vulnerabilities throughout the development lifecycle.

Security practices are regularly evaluated to ensure alignment with industry best practices and to address emerging threats. Recognized security frameworks, such as OWASP, are incorporated into the development process to ensure that information security and privacy policies consistently meet high industry standards.

Business continuity and disaster recovery

In the event of a disruption, Databox has a robust business continuity and disaster recovery plan to ensure uninterrupted technical and customer support. Dedicated teams, equipped with defined roles, tools, and processes, manage emergencies while maintaining communication and service levels.

Privacy and compliance