Share post
Share post
Share post
Blog: Product
An overview of inbound sync configurability
An overview of inbound sync configurability
Snowflake |
Product features
By: Chris Chandler
24 July 2024
Omnata Sync is designed to be fast but also flexible. For ingesting into Snowflake, it comes with simple defaults to get you started quickly, but we make it easy to change these defaults to suit your production pipelines.
The Controls tab is where you can edit and existing sync and manage advanced config settings; rate-limits, sync record state, normalized views, and storage location.
Go to Home, select the sync, go to Controls tab, then scroll down to the sections below.
Adjust rate limits
Omnata Plugins come with pre-configured rate limits for each API endpoint. Some apps like Zendesk and HubSpot, offer different API limits depending on your plan, and we gather this info during the connection steps. Instead of hitting API limits and backing off, we stay under the limit to avoid errors. In some situations, you may want to increase or decrease the default rate-limit for a sync. You can do that here.
Reset inbound sync state
Omnata fetches data incrementally by default, falling back to full-refresh where not supported by the source. You have the option of merging or appending record changes. You can see the latest fetched data by the cursor field of each object and clear the state to trigger a re-sync on the next scheduled run.
View normalized schemas
Omnata always lands the raw data from API endpoints. This makes it easier to handle new fields and schema changes in the source. We also provide normalized views that incorporate the metadata about the field in the source and you can drill into this detail here.
Modify storage location
By default, Omnata lands inbound data in the Omnata Sync Engine app database. This is the simplest way to get started quickly.
However, the app database has some limitations for production pipelines:
It is owned by the application
It cannot be easily replicated or shared
It does not fit with most transformation pipelines
Scrolling down on the Controls tab to Storage Location. You can modify the location for each sync to a separate database, customize the naming convention, change the table type and enable change tracking.
Try it free today
Overall, these features set Omnata apart from basic connector apps and enable you to deploy production-ready data pipelines natively on Snowflake. You can try Omnata for free in minutes on the Snowflake Marketplace.
Omnata Sync is designed to be fast but also flexible. For ingesting into Snowflake, it comes with simple defaults to get you started quickly, but we make it easy to change these defaults to suit your production pipelines.
The Controls tab is where you can edit and existing sync and manage advanced config settings; rate-limits, sync record state, normalized views, and storage location.
Go to Home, select the sync, go to Controls tab, then scroll down to the sections below.
Adjust rate limits
Omnata Plugins come with pre-configured rate limits for each API endpoint. Some apps like Zendesk and HubSpot, offer different API limits depending on your plan, and we gather this info during the connection steps. Instead of hitting API limits and backing off, we stay under the limit to avoid errors. In some situations, you may want to increase or decrease the default rate-limit for a sync. You can do that here.
Reset inbound sync state
Omnata fetches data incrementally by default, falling back to full-refresh where not supported by the source. You have the option of merging or appending record changes. You can see the latest fetched data by the cursor field of each object and clear the state to trigger a re-sync on the next scheduled run.
View normalized schemas
Omnata always lands the raw data from API endpoints. This makes it easier to handle new fields and schema changes in the source. We also provide normalized views that incorporate the metadata about the field in the source and you can drill into this detail here.
Modify storage location
By default, Omnata lands inbound data in the Omnata Sync Engine app database. This is the simplest way to get started quickly.
However, the app database has some limitations for production pipelines:
It is owned by the application
It cannot be easily replicated or shared
It does not fit with most transformation pipelines
Scrolling down on the Controls tab to Storage Location. You can modify the location for each sync to a separate database, customize the naming convention, change the table type and enable change tracking.
Try it free today
Overall, these features set Omnata apart from basic connector apps and enable you to deploy production-ready data pipelines natively on Snowflake. You can try Omnata for free in minutes on the Snowflake Marketplace.
Omnata Sync is designed to be fast but also flexible. For ingesting into Snowflake, it comes with simple defaults to get you started quickly, but we make it easy to change these defaults to suit your production pipelines.
The Controls tab is where you can edit and existing sync and manage advanced config settings; rate-limits, sync record state, normalized views, and storage location.
Go to Home, select the sync, go to Controls tab, then scroll down to the sections below.
Adjust rate limits
Omnata Plugins come with pre-configured rate limits for each API endpoint. Some apps like Zendesk and HubSpot, offer different API limits depending on your plan, and we gather this info during the connection steps. Instead of hitting API limits and backing off, we stay under the limit to avoid errors. In some situations, you may want to increase or decrease the default rate-limit for a sync. You can do that here.
Reset inbound sync state
Omnata fetches data incrementally by default, falling back to full-refresh where not supported by the source. You have the option of merging or appending record changes. You can see the latest fetched data by the cursor field of each object and clear the state to trigger a re-sync on the next scheduled run.
View normalized schemas
Omnata always lands the raw data from API endpoints. This makes it easier to handle new fields and schema changes in the source. We also provide normalized views that incorporate the metadata about the field in the source and you can drill into this detail here.
Modify storage location
By default, Omnata lands inbound data in the Omnata Sync Engine app database. This is the simplest way to get started quickly.
However, the app database has some limitations for production pipelines:
It is owned by the application
It cannot be easily replicated or shared
It does not fit with most transformation pipelines
Scrolling down on the Controls tab to Storage Location. You can modify the location for each sync to a separate database, customize the naming convention, change the table type and enable change tracking.
Try it free today
Overall, these features set Omnata apart from basic connector apps and enable you to deploy production-ready data pipelines natively on Snowflake. You can try Omnata for free in minutes on the Snowflake Marketplace.
subscribe
Deliver the goods to your inbox
subscribe
Deliver the goods to your inbox
subscribe