I want to start talking about migrating from Oracle to Snowflake and the best methods I’ve come up with to do this, so I’m going to use this article to post some background around my journey.

Here goes.

Using Pentaho For Financial Services

Back in 2012 while working for a bank in Central London, UK, I started using Pentaho Data Integration as the second member of a team building a platform for 300 apps to share infrastructure event data.

I went on to grow the data integration platform to synchronise 100s of billions of records weekly from over 30 internal and external sources.

Our pipelines were fully integrated with bank-wide monitoring and alerting systems, and the application itself was fully supported by my colleague and I, from development through production. That’s ONLY TWO members of staff!

We attributed our success to the fact that we built systems that scaled with as few moving parts as possible. We shunned complexity and storing state where it wasn’t required and made sure our data pipelines were always idempotent.

We wanted them to just pick up where they left off, if any of the moving parts failed at any point for any reason. We didn’t want to maintain anything and this meant we were free to grow the platform and innovate around analytics for the business.

Using Pentaho In Retail

Winding forward to 2018, I was given the opportunity to do it again, while working for a retail company in West London, UK.

This time, the task was to migrate data out of an on-prem Oracle database into Snowflake, the cloud warehouse service.

The aim was to give the business renewed access to their data in modern analytics and dashboarding services, starting with SaaS product, Looker.

We wanted to democratise the data and gain the efficiencies of having more people innovating around it, instead of relying on a centralised reporting team.

So I thought, “I know, this’ll be easy, I’ll just use Pentaho!”

After all, I had used it in anger for 5 years…

And this leads me on nicely to my next article about migrating Oracle to Snowflake using Pentaho and why you shouldn’t use it.

Hopefully see you there,

– Richard