Polling Rates
Select the schedule for your pipeline executions.
Polling rates determine how often your pipeline runs to fetch new data from a source integration. Choosing the right polling rate is crucial for creating efficient and reliable data flows. This guide explains the key factors to consider when setting a schedule for your pipeline.
How to Choose a Polling Rate
The ideal polling rate is a balance between data freshness and system efficiency.
Check the Integration Guide: This is the most important step. Every source integration is different. Navigate to the Integration Library, find your source, and look for the “Recommended Polling Rates” section. This will give you the safest and most effective polling interval to use.
Consider the Data Source: How often does your sensor or device actually generate new data? Polling more frequently than new data is available will only result in empty runs. For example, if a sensor reports a new reading every 15 minutes, a 5-minute polling rate is inefficient.
Respect API Limits: Many vendors limit the number of requests you can make in a given period. While IMSURGE is designed to handle these limits gracefully, setting a polling rate that is too aggressive can still lead to warnings or temporary throttling.
General Recommendations
The table below provides general guidelines, but it is not a substitute for the specific recommendations found on each integration’s page.