Camunda and DBT transforms

What is DBT?

If you haven't encountered DBT before, it serves as a framework designed to assist data analysts and engineers in the transformation and loading of data into their data warehouse. It's a modern take on building database assets that can use all the CI tools we've learned to love over the years.

DBT offers an uncomplicated approach to crafting and overseeing data transformations, encompassing valuable features like testing, documentation, and collaboration.

A key advantage of dbt lies in its capability to enable data transformation through code, eliminating the dependence on manual processes or isolated scripts. This approach simplifies the maintenance of your data pipeline, ensuring the consistency of transformations. Furthermore, dbt facilitates version control for your data models, a crucial aspect for tracking changes and troubleshooting issues.

Data engineering and Camunda data

Camunda is a best-in-class workflow engine. Under the hood, workflow engines are complicated. As anyone who has tried to build one will attest, there are code and data constructs to create which add a lot of complexity to the mix. So how do we measure and report on our workflows?

Luckily, Camunda has Optimize and Operate products, occasionally we want to combine data. For example:

• Measuring how our customer profiles impact our processes by joining workflow data with our CRM

  • An ecommerce company that has certain customers segments that have negative impacts on it's camunda driven support process.

• Employee training required to perform a process based on our time-to-completion for task types

  • A government department measuring passport demand on camunda driven processes and staff profiles.

• Which sales activities lead to worse outcomes for our processes

  • An insurance company receiving leads that didn't pass a camunda driven due diligence.

All these require some sort of data engineering. The practice of bringing data sources together and running transforms and queries to make data-backed decisions.

DBT and Camunda - Part One

Camunda exposes it's data in a few different ways.

• Database History tables

• Java Libraries

• REST APIs

In our example, we use Airbyte, an iPaas (integration platform as a service) to extract the data from Camunda's REST APIs.

We then run DBT transforms to extract powerful views such as:

• Current tasks in progress

• Completed tasks

• Longest processes by duration

These can then be combined with other data sources to enable a 360 view of your processes, CRM and other data sources.

We've used this in the past to combine data from workflow processes, Google Analytics and WIP (work-in-progress) databases to build a detailed operational picture.

This shows our users how they found us, how they used our service, how our staff performed their tasks and whether the user returned.

Show me the code!

We'll show how we build this in the upcoming posts. Stay tuned.