A pipeline is the core component that automates the flow of data in IMSURGE. It connects one source integration to one target integration and runs on a schedule you define to keep your data synchronized.

How a Pipeline Works

Every pipeline follows the same core workflow. It uses the authentication details stored in your Credentials to access both the source and target systems.

  1. Fetch & Parse: On its schedule, the pipeline activates the source integration, which fetches raw data from the external system and parses it into a standardized format.
  2. Deliver: The pipeline then delivers this standardized data to the target integration, which pushes it to the final destination.
flowchart TD
    A["Source Integration (Credentials)"]
    B{Pipeline}
    C["Target Integration (Credentials)"]
    A -- Fetches & Parses Data --> B
    B -- Delivers Standardized Data --> C

Key Characteristics

  • One-to-One Rule: A pipeline must have exactly one source and one target. To send the same data to multiple destinations, you must create a separate pipeline for each target.
  • Scheduling: You define how often the pipeline runs, from every minute to every 24 hours. For guidance, see the Polling Rates guide.
  • Backtracking: On its first run, a pipeline can backtrack to fetch historical data (up to 7 days, depending on the source).
  • Data Retention: The standardized data from each pipeline execution is stored for 90 days, allowing you to review and troubleshoot past runs.

Reference

ComponentPurpose
Source IntegrationDetermines where data comes from (e.g., Aeroqual, Svantek).
Target IntegrationThe destination for your parsed data (e.g., Sensemetrics, iTwin IoT).
SchedulerDefines how frequently the pipeline runs and how far back to fetch data on the first run.
Execution LogA detailed record of every time the pipeline has run, including its status and any errors.
ActionDescription
Create PipelineBuild a new pipeline from scratch. See the Creating a Pipeline guide.
Edit SettingsModify an existing pipeline’s schedule, source, or target.
DuplicateCreate a copy of a pipeline. This is useful for creating similar data flows or for testing.
View ExecutionsMonitor the health and history of your pipeline runs. See Monitor Executions.