Lorem ipsum dolor sit . Proin gravida nibh vel vealiquete sollicitudin, lorem quis bibendum auctonisilin sequat. Nam nec tellus a odio tincidunt auctor ornare.

Stay Connected & Follow us

What are you looking for?

Simply enter your keyword and we will help you find what you need.
PBT GroupGCP Metadata driven ETL

GCP Metadata driven ETL

The Company

A retail organisation that has embraced Google Cloud Platform for their analytics work.

The Business Challenge

Calculating LOST SALES every season in order to do planning for the next season. The legacy calculations ran per department, and took between 30 minutes and 3 hours to complete per department (there are 45 departments).

The Solution

As the data needed for the LOST SALES calculation was already being loaded into the GCP data lake, it was possible to embed the LOST SALES and planning logic into a meta-data-driven ETL solution developed in python 3 and deployable as either a app engine app ,or a cloud function (or both), and automated using a choice of AppEngine(cron) or Cloud Composer(apache airflow). The ETL Component was designed in such a way that it is meta data driven, and can be invoked using Pub/Sub, REST calls, App Engine or gcloud command line.


The Value Proposition

The LOST SALES Calculation now runs for the whole organisation in less than 10 minutes. And analysts can maintain the etl by simply editing SQL That they are familiar with, while the solution can be parameterised in a flexible way to do ‘ad hoc’ runs.  This means planners can add new prices, and redo their planning run at the push of a button.


Related Cases

No posts were found.