Share post

Share post

Share post

Blog: Product

An overview of inbound sync configurability

An overview of inbound sync configurability

Snowflake |

Product features

By: Chris Chandler

24 July 2024

Omnata Sync is designed to be fast but also flexible. For ingesting into Snowflake, it comes with simple defaults to get you started quickly, but we make it easy to change these defaults to suit your production pipelines.

The Controls tab is where you can edit and existing sync and manage advanced config settings; rate-limits, sync record state, normalized views, and storage location.

Go to Home, select the sync, go to Controls tab, then scroll down to the sections below.

Adjust rate limits

Omnata Plugins come with pre-configured rate limits for each API endpoint. Some apps like Zendesk and HubSpot, offer different API limits depending on your plan, and we gather this info during the connection steps. Instead of hitting API limits and backing off, we stay under the limit to avoid errors. In some situations, you may want to increase or decrease the default rate-limit for a sync. You can do that here.

Reset inbound sync state

Omnata fetches data incrementally by default, falling back to full-refresh where not supported by the source. You have the option of merging or appending record changes. You can see the latest fetched data by the cursor field of each object and clear the state to trigger a re-sync on the next scheduled run.

View normalized schemas

Omnata always lands the raw data from API endpoints. This makes it easier to handle new fields and schema changes in the source. We also provide normalized views that incorporate the metadata about the field in the source and you can drill into this detail here.

Modify storage location

By default, Omnata lands inbound data in the Omnata Sync Engine app database. This is the simplest way to get started quickly.

However, the app database has some limitations for production pipelines:

  • It is owned by the application

  • It cannot be easily replicated or shared

  • It does not fit with most transformation pipelines

Scrolling down on the Controls tab to Storage Location. You can modify the location for each sync to a separate database, customize the naming convention, change the table type and enable change tracking.


Try it free today

Overall, these features set Omnata apart from basic connector apps and enable you to deploy production-ready data pipelines natively on Snowflake. You can try Omnata for free in minutes on the Snowflake Marketplace.

Omnata Sync is designed to be fast but also flexible. For ingesting into Snowflake, it comes with simple defaults to get you started quickly, but we make it easy to change these defaults to suit your production pipelines.

The Controls tab is where you can edit and existing sync and manage advanced config settings; rate-limits, sync record state, normalized views, and storage location.

Go to Home, select the sync, go to Controls tab, then scroll down to the sections below.

Adjust rate limits

Omnata Plugins come with pre-configured rate limits for each API endpoint. Some apps like Zendesk and HubSpot, offer different API limits depending on your plan, and we gather this info during the connection steps. Instead of hitting API limits and backing off, we stay under the limit to avoid errors. In some situations, you may want to increase or decrease the default rate-limit for a sync. You can do that here.

Reset inbound sync state

Omnata fetches data incrementally by default, falling back to full-refresh where not supported by the source. You have the option of merging or appending record changes. You can see the latest fetched data by the cursor field of each object and clear the state to trigger a re-sync on the next scheduled run.

View normalized schemas

Omnata always lands the raw data from API endpoints. This makes it easier to handle new fields and schema changes in the source. We also provide normalized views that incorporate the metadata about the field in the source and you can drill into this detail here.

Modify storage location

By default, Omnata lands inbound data in the Omnata Sync Engine app database. This is the simplest way to get started quickly.

However, the app database has some limitations for production pipelines:

  • It is owned by the application

  • It cannot be easily replicated or shared

  • It does not fit with most transformation pipelines

Scrolling down on the Controls tab to Storage Location. You can modify the location for each sync to a separate database, customize the naming convention, change the table type and enable change tracking.


Try it free today

Overall, these features set Omnata apart from basic connector apps and enable you to deploy production-ready data pipelines natively on Snowflake. You can try Omnata for free in minutes on the Snowflake Marketplace.

Omnata Sync is designed to be fast but also flexible. For ingesting into Snowflake, it comes with simple defaults to get you started quickly, but we make it easy to change these defaults to suit your production pipelines.

The Controls tab is where you can edit and existing sync and manage advanced config settings; rate-limits, sync record state, normalized views, and storage location.

Go to Home, select the sync, go to Controls tab, then scroll down to the sections below.

Adjust rate limits

Omnata Plugins come with pre-configured rate limits for each API endpoint. Some apps like Zendesk and HubSpot, offer different API limits depending on your plan, and we gather this info during the connection steps. Instead of hitting API limits and backing off, we stay under the limit to avoid errors. In some situations, you may want to increase or decrease the default rate-limit for a sync. You can do that here.

Reset inbound sync state

Omnata fetches data incrementally by default, falling back to full-refresh where not supported by the source. You have the option of merging or appending record changes. You can see the latest fetched data by the cursor field of each object and clear the state to trigger a re-sync on the next scheduled run.

View normalized schemas

Omnata always lands the raw data from API endpoints. This makes it easier to handle new fields and schema changes in the source. We also provide normalized views that incorporate the metadata about the field in the source and you can drill into this detail here.

Modify storage location

By default, Omnata lands inbound data in the Omnata Sync Engine app database. This is the simplest way to get started quickly.

However, the app database has some limitations for production pipelines:

  • It is owned by the application

  • It cannot be easily replicated or shared

  • It does not fit with most transformation pipelines

Scrolling down on the Controls tab to Storage Location. You can modify the location for each sync to a separate database, customize the naming convention, change the table type and enable change tracking.


Try it free today

Overall, these features set Omnata apart from basic connector apps and enable you to deploy production-ready data pipelines natively on Snowflake. You can try Omnata for free in minutes on the Snowflake Marketplace.

subscribe

Deliver the goods to your inbox

subscribe

Deliver the goods to your inbox

subscribe

Deliver the goods to your inbox