Skip to main content

How to export new data daily?

Updated

This article describes how to configure daily exports that only contain new data (for example, "all events from yesterday") and how to handle this more robustly with Engagement BigQuery (EBQ).

The platform doesn't have a built‑in "only new since last export" toggle. Instead, you configure time‑window filters and scheduling at the export definition level to approximate incremental exports.

At a high level, you have two options:

  • Daily time‑window export (recommended for SFTP/file‑based flows)
  • Engagement BigQuery (EBQ) for proper incremental storage and querying

Option 1: Daily time‑window export

Use this when you want a single file per day (for example, on SFTP, S3, Azure, or GCS) containing only events from that day.

Prerequisites

  • Access to Data & Assets → Exports with Exports module and Personal Data Viewer permissions.
  • A configured file storage integration (for example, SFTP, GCS, S3, or Azure Storage) as the export target.

Step 1: Open or create the event export

  1. In app, go to Data & Assets → Exports.
  2. Either:
    • Open your existing event export (for example, campaign events), or
    • Click New export and choose the appropriate event type as the data source (for example, campaign, purchase, purchase_item).

Step 2: Filter to a daily time window

  1. In Add filters, define a relative time filter on the event timestamp (for example, the campaign event timestamp).
  2. Use a time window that represents exactly one day, for example:
    • "Events from the last 1 day", or
    • "Yesterday", if you prefer a named relative period.

Events are exported based on their timestamp attribute, not on when they were tracked, so late‑arriving events with older timestamps will not appear if they fall outside this time window.

Result of this step: Each export run will include only events whose timestamp falls within the configured 1‑day window.

Step 3: Configure file format, target, and dynamic file naming

  1. In Format and target:
    • Choose the desired Format (CSV, JSON, or JSON Lines).
    • Select your target storage integration (for example, SFTP, GCS, S3, Azure Storage).
  2. Set:
    • Path to the parent directory on the target system, and
    • A filename pattern that includes a dynamic timestamp so each day's file is unique (for example, campaign_events_%Y-%m-%d.csv).

The platform supports dynamic file naming with time‑based tokens so that each run can produce a file with a different timestamped name (for example, filename_1589460477.9546165). The SFTP (or other) integration receives the file name and content produced by the export definition.

Result of this step: Your storage (SFTP, S3, and more) will contain one file per export run, each with a distinct timestamp in the filename, making it easy for your DWH to treat them as daily partitions.

Step 4: Schedule the export to run daily

  1. In the Execute export/scheduling section, configure the export to run as a scheduled export once per day (for example, shortly after midnight in your reporting timezone).
  2. Save the export definition.

From now on, the app will automatically run the export every day using:

  • The relative time filter you configured, and
  • The dynamic filename pattern you set.

As long as your time filter always covers exactly one completed day, each generated file will contain only that day's events, with no historical re‑send.

Option 2: Engagement BigQuery (EBQ)

If your project has EBQ, this is usually the better long‑term solution for analytics and warehousing:

  • EBQ exports data into BigQuery tables, where new rows are appended incrementally, and BigQuery is designed for DWH‑style querying and partitioning.
  • Your data team can then:
    • Use BigQuery SQL to define their own daily partitions or views, and
    • Pull daily deltas directly from BigQuery, instead of ingesting SFTP/CSV files.

Typical pattern:

  1. Enable and configure EBQ for the project (via your CSM / TC).
  2. Let EBQ handle continuous, append‑only exports of customer and event data into BigQuery.
  3. In your DWH layer, query "events where timestamp between yesterday_start and yesterday_end" and persist the result into your own schema (for example, a daily partitioned table).

These steps remove the need to maintain daily SFTP exports and provide stronger scalability and query flexibility, especially for large datasets.

Behavior and limitations to keep in mind

  • No implicit "stateful" incremental logic: the platform doesn't remember "the last exported timestamp"; your time filter and schedule entirely govern the behavior.
  • Event timestamp vs. tracking time: Exports include events based on their timestamp attribute, which can differ from the time the event was tracked (for example, backfilled or delayed events).
  • Export limitations: Very large exports can hit size and concurrency limits (for example, Azure Storage limits exports to ~48.8 GB; only one run per export definition at a time, and at most four exports running concurrently). In those scenarios, EBQ is recommended instead of attempting "export all data".

By combining relative time filters, daily scheduling, and dynamic filenames, you can reliably deliver your data warehouse with a single file per day containing only that day's new events. For larger or more complex use cases, consider EBQ as the more scalable, fully incremental alternative.

Related articles