A note on dbt product naming: dbt is an engine, and it now comes in two versions: v1.x (previously called dbt Core) and the next-generation v2.0. You can either self-host dbt or run on dbt platform (formerly dbt Cloud). Many people still search for “dbt Core” and “dbt Cloud”, but those are the previous names, so this article uses the current terms.
Running dbt or Working with dbt?
Self-hosted dbt (formally called dbt Core*) gives data teams considerable flexibility and control. But as an implementation grows, so can the amount of work required to keep everything around it running.
Orchestration, infrastructure, CI/CD, access, monitoring, upgrades and troubleshooting all need to be managed somewhere. Over time, this operational work can consume engineering capacity that could otherwise be spent improving data quality, building models and delivering new data products.
So, how much time is your team spending managing dbt rather than using dbt?
The Infrastructure Around dbt
When teams self host dbt, organisations are responsible for more than the dbt project itself.
Depending on the environment, teams may need to manage:
- Infrastructure and compute
- Job scheduling and orchestration
- CI/CD workflows
- Development and production environments
- Authentication and access
- Monitoring and alerting
- Upgrades and dependencies
- Documentation and troubleshooting
Individually, these tasks may seem manageable. As the environment grows, however, the cumulative workload can become significant.
Operational Work Competing for Engineering Capacity?
Every hour spent maintaining deployment scripts, investigating failed jobs or managing environments is an hour that isn’t being spent on higher-value data work.
For a growing data team, this creates an important question:
Are your engineers spending their time building with dbt, or maintaining everything required to run it?
Complexity Compounds As You Scale
More models mean more dependencies.
More contributors require stronger development processes.
More workloads increase monitoring requirements.
Greater business reliance increases the consequences when something goes wrong.
As each requirement is addressed, additional tooling, scripts and processes can accumulate around the original dbt implementation.
The result can be an environment that still works, but requires increasingly more effort to operate.
When Does Self-Hosting Stop Making Sense?
There isn’t a single point at which every organisation should move away from self-hosted dbt.
The better question is whether maintaining your current operating model remains the best use of your team’s skills and resources.
If operational overhead is increasing faster than the value your engineers can deliver, it may be worth reassessing how dbt is managed and whether the dbt platform could simplify some of that responsibility.
Read on if you’re weighing your options: self-hosting dbt or dbt platform.
Considering the Move to dbt platform?
See how we’ve helped others make the move, or learn how to manage your own upgrade to dbt platform.
See a dbt platform upgrade in action
See how Data Army helped Australian digital home loan platform and financial technology company, Lendi, standardise on dbt platform, creating a trusted data foundation for its AI-native future.
How to upgrade to dbt platform
Our practical guide steps through how to move to dbt platform while managing dependencies, reducing disruption and taking the opportunity to build a more scalable, easier-to-manage environment.
*dbt Labs now refers to what many teams know as dbt Core as self-hosted dbt, while dbt Cloud is dbt platform.
You can read more about the distinction between self-hosted dbt and the dbt platform here.
Helping Organisations Get More From dbt
Data Army is a dbt Visionary Consulting & Services Partner, the highest tier of partnership within the dbt Partner Program. As a dbt-certified organisation, we bring deep expertise in analytics engineering, implementing and optimising dbt for our clients, and helping organisations advance their modern data stack with scalable, trusted data practices.
We also use dbt extensively within Data Army’s own data environment, so our team has first-hand experience of the platform and the same practical considerations our clients encounter when implementing, managing and scaling dbt.
