Blog > Opinion Product

The future of Reverse ETL - Smart pumps, dumb pipes

Posted by Chris Chandler | May 12, 2021

ELT beats ETL. This is because your data transformations can now happen in-database and you don't lose data in the process. It makes data teams more agile and better able to model data from many systems. A category of tools has formed that simply extract and load data into data warehouses. ETL is basically dead.

If ETL is dead for data going in, shouldn’t it be the same for data going out? 🧐

We think so.

However, ‘Reverse ETL’ has been trending and some of the tools tout the value of transforming data in the last-mile, on the way to applications. If we believe in the above, T should only happen inside the warehouse and should remain decoupled from EL.

Reverse ETL products typically provide smart pipes. They are configured and managed as separate systems to your data warehouse transformations, and are orchestrated as a subsequent step.

But if you're bought into the modern data stack, there's not much value in doing transformations again in the last-mile. Doing so is even a bad thing because it creates a silo. In this case, Reverse ETL tools should really only be used as ‘Reverse EL’ tools.

More than just a semantic tug-o-war?

Yeh!
We think warehouse-to-app integration should be simplified, not another ETL or EL layer stacked on top. Omnata Push is a step in this direction by using the data warehouse as a smart end-point.

Smart pumps
Snowflake paved the way for this pattern by delivering external functions early 2021. Omnata uses these to, as our product name suggests, “push” data down the pipes. It’s a process that is initiated by the warehouse, turning it into a smart pump. If you’re using dbt, you can keep your warehouse-to-app integration logic within the existing ecosystem, rather than adding a subsequent step at the end.

Dumb pipes
The pipes do as little as possible, they're dumb, acting as a conduit between Snowflake and the destination app. They don't extract and load any data, they don't even hold credentials. If you were a little mad, you could even call it Reverse L...😬

Since we're not mad, we think the most logical thing to do is simply add a P for Push onto what's already in place; ELT + P. (Daigo Tanaka used P too, but for Publish instead of Push)

What are the benefits of Push?

Push helps you to simplify and decouple last-mile integration with smart pumps and dumb pipes.

Native to your data pipeline

Push runs from Snowflake so it fits nicely into your data team's workspace. Outbound integrations are managed like any other SQL.

No middle-layer to configure

Omnata’s integration portal helps you to set up your connections and then takes a back seat. There is no middle platform doing extracts and loads or managing schedules which means one less point of failure.

No storage of data or credentials

Omnata doesn’t store data, not even for a moment, and it doesn’t store your credentials. This requires less trust and results in less onerous risk assessment and vendor assurance. Instead, your app credentials can be safely stored within Snowflake, because only Omnata’s APIs have the ability to decrypt and use them.

Monitor tasks centrally

The destination will ping a response from the load job and Omnata will deliver this back to Snowflake like the rest of your data. This means that you can build retry and failure logic centrally and monitor it with your existing BI tools.

Integrate with dbt

Ommata provides pre-built dbt packages for Omnata Push so you can get started quickly and run data loads alongside your other CI/CD tasks.

More destinations on the way

We have just released Salesforce Marketing Cloud to public preview and will be expanding the destinations of Push.

Try Omnata Push free

Let us know

if there’s a destination that you want.

Omnata blog

Expert tips for Salesforce and Snowflake, plus, open-source and community contributions. Read insights about analytics, machine learning, enterprise architecture and data-engineering.

Thanks for signing up!

Error sending please try again

Popular Posts