Skip to main content

Move from dbt Core to the dbt platform: Get started

Back to guides Total estimated time: 3-4 hours
Migration
dbt Core
dbt platform
Intermediate
Menu

    Introduction​

    Moving from dbt Core to dbt streamlines analytics engineering workflows by allowing teams to develop, test, deploy, and explore data products using a single, fully managed software service.

    Explore our 3-part-guide series on moving from dbt Core to dbt. This series is ideal for users aiming for streamlined workflows and enhanced analytics:

    Guide
    Information
    Audience
    Move from dbt Core to dbt platform: What you need to knowUnderstand the considerations and methods needed in your move from dbt Core to dbt platform.Team leads
    Admins
    Move from dbt Core to dbt platform: Get startedLearn the steps needed to move from dbt Core to dbt platform.Developers
    Data engineers
    Data analysts
    Move from dbt Core to dbt platform: Optimization tipsLearn how to optimize your dbt experience with common scenarios and useful tips.Everyone

    Why move to the dbt platform?​

    If your team is using dbt Core today, you could be reading this guide because:

    • You’ve realized the burden of maintaining that deployment.
    • The person who set it up has since left.
    • You’re interested in what dbt could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices.

    Moving from dbt Core to dbt simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With dbt, you gain access to features like cross-team collaboration (dbt Mesh), version management, streamlined CI/CD, dbt Explorer for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently.

    It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.

     What are dbt and dbt Core?
    • dbt is the fastest and most reliable way to deploy dbt. It enables you to develop, test, deploy, and explore data products using a single, fully managed service. It also supports:

    Learn more about dbt features.

    • dbt Core is an open-source tool that enables data teams to define and execute data transformations in a cloud data warehouse following analytics engineering best practices. While this can work well for ‘single players’ and small technical teams, all development happens on a command-line interface, and production deployments must be self-hosted and maintained. This requires significant, costly work that adds up over time to maintain and scale.

    What you'll learn​

    This guide outlines the steps you need to take to move from dbt Core to dbt and highlights the necessary technical changes:

    • Account setup: Learn how to create a dbt account, invite team members, and configure it for your team.
    • Data platform setup: Find out about connecting your data platform to dbt.
    • Git setup: Learn to link your dbt project's Git repository with dbt.
    • Developer setup: Understand the setup needed for developing in dbt.
    • Environment variables: Discover how to manage environment variables in dbt, including their priority.
    • Orchestration setup: Learn how to prepare your dbt environment and jobs for orchestration.
    • Models configuration: Get insights on validating and running your models in dbt, using either the Studio IDE or dbt CLI.
    • What's next?: Summarizes key takeaways and introduces what to expect in the following guides.

    Prerequisites​

    Account setup​

    This section outlines the steps to set up your dbt account and configure it for your team.

    1. Create your dbt account.

    2. Provide user access and invite users to your dbt account and project.

    3. Configure Single Sign-On (SSO) or Role-based access control (RBAC) for easy and secure access. EnterpriseEnterprise +

      • This removes the need to save passwords and secret environment variables locally.

    Additional configuration​

    Explore these additional configurations for performance and reliability improvements:

    1. In Account settings, enable partial parsing to only reparse changed files, saving time.

    2. In Account settings, enable Git repo caching for job reliability & third-party outage protection. EnterpriseEnterprise +

    Data platform setup​

    This section outlines the considerations and methods to connect your data platform to dbt.

    1. In dbt, set up your data platform connections and environment variables. dbt can connect with a variety of data platform providers including:

    2. You can verify your data platform connections by clicking the Test connection button in your deployment and development credentials settings.

    Additional configuration​

    Explore these additional configurations to optimize your data platform setup further:

    1. Use OAuth connections, which enables secure authentication using your data platform’s SSO. EnterpriseEnterprise +

    Git setup​

    Your existing dbt project source code should live in a Git repository. In this section, you will connect your existing dbt project source code from Git to dbt.

    1. Ensure your dbt project is in a Git repository.

    2. In Account settings, select Integrations to connect your Git repository to dbt:

      • (Recommended) Connect with one of the native integrations in dbt (such as GitHub, GitLab, and Azure DevOps).

        This method is preferred for its simplicity, security features (including secure OAuth logins and automated workflows like CI builds on pull requests), and overall ease of use.

      • Import a Git repository from any valid Git URL that points to a dbt project.

    Additional configuration​

    Explore these additional configurations to optimize your Git setup further:

    1. Log into dbt using OAuth connections to integrate with your source code platform. It automatically links to the repository using one of the native integrations set at the account level. EnterpriseEnterprise +

    Set up groups for dbt project access with those configured for repository access to streamline permissions.

    Developer setup​

    This section highlights the development configurations you’ll need for your dbt project. The following categories are covered in this section:

    dbt environments​

    The most common data environments are production, staging, and development. The way dbt Core manages environments is through target, which are different sets of connection details.

    dbt environments go further by:

    • Integrating with features such as job scheduling or version control, making it easier to manage the full lifecycle of your dbt projects within a single platform.
    • Streamlining the process of switching between development, staging, and production contexts.
    • Making it easy to configure environments through the dbt UI instead of manually editing the profiles.yml file. You can also set up or customize target names in dbt.
    • Adding profiles.yml attributes to dbt environment settings with Extended Attributes.
    • Using Git repo caching to protect you from third-party outages, Git auth failures, and more. EnterpriseEnterprise +

    Initial setup steps​

    1. Set up development environment — Set up your development environment and development credentials. You’ll need this to access your dbt project and start developing.

    2. dbt Core version — In your dbt environment, select a release track for ongoing dbt version upgrades. If your team plans to use both dbt Core and dbt for developing or deploying your dbt project, you can run dbt --version in the command line to find out which version of dbt Core you’re using.

      • When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt, leverage release tracks so you don’t have to.

    3. Connect to your data platform — When using dbt, you can connect to your data platform directly in the UI.

      • Each environment is roughly equivalent to an entry in your profiles.yml file. This means you don't need a profiles.yml file in your project.

    4. Development tools — Set up your development workspace with the dbt CLI (command line interface or code editor) or Studio IDE (browser-based) to build, test, run, and version control your dbt code in your tool of choice.

      • If you've previously installed dbt Core, the dbt CLI installation doc has more information on how to install the dbt CLI, create aliases, or uninstall dbt Core for a smooth transition.

    Additional configuration​

    Explore these additional configurations to optimize your developer setup further:

    1. Custom target names — Using custom target.names in your dbt projects helps identify different environments (like development, staging, and production). While you can specify the custom target.name values in your developer credentials or orchestration setup, we recommend using environment variables as the preferred method. They offer a clearer way to handle different environments and are better supported by dbt's partial parsing feature, unlike using {{ target }} logic which is meant for defining the data warehouse connection.

    dbt commands​

    1. Review the dbt commands supported for dbt development. For example, dbt init isn’t needed in dbt as you can create a new project directly in dbt.

    Environment variables​

    This section will help you understand how to set up and manage dbt environment variables for your project. The following categories are covered:

    In dbt, you can set environment variables in the dbt user interface (UI). Read Set up environment variables for more info.

    In dbt Core, environment variables, or the env_var function, are defined manually by the developer or within the external application running dbt.

    Environment variables in dbt​

    • dbt environment variables must be prefixed with DBT_ (including DBT_ENV_CUSTOM_ENV_ or DBT_ENV_SECRET).
    • If your dbt Core environment variables don’t follow this naming convention, perform a “find and replace” in your project to make sure all references to these environment variables contain the proper naming conventions.
    • dbt secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with DBT_ENV_SECRETto obscure them in logs and the UI.
    Setting project level and environment level valuesSetting project level and environment level values

    dbt environment variables order of precedence​

    Environment variables in dbt are managed with a clear order of precedence, allowing users to define values at four levels (highest to lowest order of precedence):

    • The job level (job override) or in the Studio IDE for an individual developer (personal override). Highest precedence
    • The environment level, which can be overridden by the job level or personal override.
    • A project-wide default value, which can be overridden by the environment level, job level, or personal override.
    • The optional default argument supplied to the env_var Jinja function in the code. Lowest precedence
    Environment variables order of precedenceEnvironment variables order of precedence

    Set environment variables in dbt​

    • To set these variables for an entire project or specific environments, navigate to Deploy > Environments > Environment variables tab.
    • To set these variables at the job level, navigate to Deploy > Jobs > Select your job > Settings > Advanced settings.
    • To set these variables at the personal override level, navigate to Profile Settings > Credentials > Select your project > Environment variables.

    Orchestration setup​

    This section outlines the considerations and methods to set up your dbt environments and jobs for orchestration. The following categories are covered in this section:

    dbt environments​

    To use the dbt's job scheduler, set up one environment as the production environment. This is the deployment environment. You can set up multiple environments for different stages of your deployment pipeline, such as development, staging/QA, and production.

    Initial setup steps​

    1. dbt Core version — In your environment settings, configure dbt with the same dbt Core version.

      • Once your full migration is complete, we recommend upgrading your environments to release tracks to always get the latest features and more. You only need to do this once.

    2. Configure your jobs — Create jobs for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or trigger on the completion of another job.

    Additional configuration​

    Explore these additional configurations to optimize your dbt orchestration setup further:

    1. Custom target names — Use environment variables to set a custom target.name for every corresponding dbt job at the environment level.

    2. dbt commands — Add any relevant dbt commands to execute your dbt jobs runs.

    3. Notifications — Set up notifications by configuring email and Slack alerts to monitor your jobs.

    4. Monitoring tools — Use monitoring tools like run history, job retries, job chaining, dashboard status tiles, and more for a seamless orchestration experience.

    5. API access — Create API auth tokens and access to dbt APIs as needed. StarterEnterprise

    6. dbt Explorer — If you use dbt Explorer and run production jobs with an external orchestrator, ensure your production jobs run dbt run or dbt build to update and view models and their metadata in dbt Explorer. Running dbt compile alone will not update model metadata. In addition, features like column-level lineage also requires catalog metadata produced through running dbt docs generate. StarterEnterprise

    CI/CD setup​

    Building a custom solution to efficiently check code upon pull requests is complicated. With dbt, you can enable continuous integration / continuous deployment (CI/CD) and configure dbt to run your dbt projects in a temporary schema when new commits are pushed to open pull requests.

    Workflow of continuous integration in dbtWorkflow of continuous integration in dbt

    This build-on-PR functionality is a great way to catch bugs before deploying to production, and an essential tool for data practitioners.

    1. Set up an integration with a native Git application (such as Azure DevOps, GitHub, GitLab) and a CI environment in dbt.
    2. Create a CI/CD job to automate quality checks before code is deployed to production.
    3. Run your jobs in a production environment to fully implement CI/CD. Future pull requests will also leverage the last production runs to compare against.

    Model development and discovery​

    In this section, you’ll be able to validate whether your models run or compile correctly in your development tool of choice: The Studio IDE or dbt CLI.

    You’ll want to make sure you set up your development environment and credentials.

    1. In your development tool of choice, you can review your dbt project, ensure it's set up correctly, and run some dbt commands:

      • Run dbt compile to make sure your project compiles correctly.
      • Run a few models in the Studio IDE or dbt CLI to ensure you’re experiencing accurate results in development.

    2. Once your first job has successfully run in your production environment, use dbt Explorer to view your project's resources (such as models, tests, and metrics) and their data lineage  to gain a better understanding of its latest production state. StarterEnterprise

    What’s next?​

    Congratulations on completing the first part of your move to dbt 🎉!

    You have learned:

    • How to set up your dbt account
    • How to connect your data platform and Git repository
    • How to configure your development, orchestration, and CI/CD environments
    • How to set up environment variables and validate your models

    For the next steps, you can continue exploring our 3-part-guide series on moving from dbt Core to dbt:

    Guide
    Information
    Audience
    Move from dbt Core to dbt platform: What you need to knowUnderstand the considerations and methods needed in your move from dbt Core to dbt platform.Team leads
    Admins
    Move from dbt Core to dbt platform: Get startedLearn the steps needed to move from dbt Core to dbt platform.Developers
    Data engineers
    Data analysts
    Move from dbt Core to dbt platform: Optimization tipsLearn how to optimize your dbt experience with common scenarios and useful tips.Everyone

    Why move to the dbt platform?​

    If your team is using dbt Core today, you could be reading this guide because:

    • You’ve realized the burden of maintaining that deployment.
    • The person who set it up has since left.
    • You’re interested in what dbt could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices.

    Moving from dbt Core to dbt simplifies workflows by providing a fully managed environment that improves collaboration, security, and orchestration. With dbt, you gain access to features like cross-team collaboration (dbt Mesh), version management, streamlined CI/CD, dbt Explorer for comprehensive insights, and more — making it easier to manage complex dbt deployments and scale your data workflows efficiently.

    It's ideal for teams looking to reduce the burden of maintaining their own infrastructure while enhancing governance and productivity.

    0