In addition to launching a new Copilot AI engine to assist with data analytic workflows, dbt Labs is expanding the capabilities of its popular dbt Cloud platform with cross-platform data mesh capabilities, support for the Apache Iceberg data table format, and new low-code visual editing tools.

Dbt Labs is expanding its dbt Cloud data management and transformation platform with a range of new features and capabilities that the company says will help streamline and automate data development processes.

The platform’s new functionality, launched this week at dbt Lab’s Coalesce 2024 conference, includes dbt Copilot, an AI engine in dbt Cloud that automates data development tasks and accelerates analytics workflows.

The announcements come as the dbt Cloud platform gains traction as an uber data control plane for enterprise data analytics, residing between cloud data platforms and data analysis tools and handling a range of data development tasks including data transformation and preparation.

[Related: Meeting The Exploding Demand For Data: The 2024 CRN Big Data 100]

Over the last five years or so there has been an “explosion” of point data products that handle specific tasks across data management and data analysis processes, said Tristan Handy, dbt Labs founder and CEO, in an interview with CRN. The result is that businesses and organizations find themselves constructing complex stacks of tools to assemble a complete data analytics system.

“They were having to buy 10 or 12 different products and then duct tape them all together, to get them to work together,” Handy (pictured) said. He founded dbt Labs in 2016 to tackle that problem by bringing software engineering practices to data processing and transformation.

“What we are building is a single, integrated control plane that allows you to do all of these different things: [data] orchestration, observability, catalog, semantic layer, transformation – and it actually all works together without so much duct tape and, hopefully, fewer vendor contracts to manage,” Handy said.

“There’s a bunch of capabilities that we think that the control plane has to have. It has to support many different user personas, because data people come in many shapes and sizes. It has to support the interactivity between many different data clouds. And it has to be AI-enabled from beginning to end,” he said.

“We use dbt at every single one of our projects, because we really believe they are the heart of the modern data stack,” said Isabela Blasi, co-founder and chief business development officer at Indicium, a global data services company and leading dbt Labs partner. (Headquartered in Brazil, Indicium has been expanding within the U.S. and this week was named dbt Labs’ Emerging Partner of the Year, Americas.)

“When we talk about data transformation, they revolutionized the extraction and load and transformation methodologies of working with data, and our company was even able to scale because we started applying dbt [with] every single one of our clients,” Blasi said in an interview with CRN.

Today dbt Labs, based in Philadelphia, has more than 4,600 global customers and 100,000 dbt community members. The fast-growing company was valued at $4.2 billion in February 2022 when it raised $222 million in Series D funding. Investors included Snowflake and Databricks – an indication of the pivotal position the company has achieved in the big data industry.

Dbt Labs has been staffing up lately, hiring SentinelOne, Elastic and Informatica veteran Sally Jenkins in September as the company’s new chief marketing officer, and former Rubrik executive Austin Stefani in February as chief revenue officer. The company just hired Shawn Toldo, who held channel management posts at OneTrust, Appian and VMware, as vice president, worldwide partner organization, to oversee the company’s channel operations.

Many of this week’s additions to the dbt Cloud platform are based on the Analytics Development Lifecycle (ADLC) workflow that underlies the company’s technology. The overall theme at this week’s Coalesce was providing a single, unified dbt experience regardless of the underlying IT infrastructure, data platform or cloud system an organization uses.

The suite of new features and capabilities across the dbt Cloud data control plane is designed to scale adoption across a more diverse set of data practitioners; make data development more accessible, streamlined and governed; and build and automate high quality data pipelines, according to the company’s press release.

Topping the list is the new dbt Copilot, which automates tasks that previously required repetitive manual work, significantly improving productivity, data quality, and stakeholder trust, according to the company.

Dbt Copilot’s functions include the ability to auto-generate tests, documentation, and semantic models (all currently in beta), an AI-chatbot that allows business stakeholders to ask natural language questions of their data (now in beta as part of the dbt native app in Snowflake), and the ability for users to bring their own OpenAI API key (now generally available). In the coming months, dbt Copilot will be extended to help automate model code generation.

“AI is very good at writing code and very good at writing dbt code,” Handy said in the interview with CRN.

Dbt Cloud now offers cross-platform dbt Mesh, building on the platform’s existing dbt Mesh feature, for cross-platform references that lets users eliminate data silos while maintaining data governance standards and helping developers see end-to-end data lineage within complex, multi-platform environments.

Key to the Mesh capabilities is dbt Cloud’s new support for the open-source Apache Iceberg data table format that’s supported by leading big data companies including Snowflake, Databricks, Starburst and Dremio.

Handy said Iceberg addresses what he sees as one of the biggest big data challenges companies face today with different data teams using different data platforms – with the result being a lot of duplication of effort.

“There’s pretty widespread support for iceberg. It is, I believe, the most widely supported open file format across the different data platforms,” he said. “It enables vendors like us to build these abstraction layers where a user can…be on a team that uses Snowflake and interact with another team that uses Databricks…seamlessly, transparently, without them having to be a super-technical data engineer that knows where the files are written to.”

Indicium supports dbt’s new products unveiled at Coalesce, including the cross-platform Mesh and the company’s hybrid approach with dbt Cloud, Blasi said. “It aligns perfectly with our shared commitment to driving data transformation and delivering outcomes at scale.

“Our agnostic approach ensures that we always prioritize the unique needs of our customers, and these new product offerings enhance our ability to deliver the flexibility and innovation required to meet those needs. We believe this launch will further enable our customers and prospects to unlock greater value from their data and accelerate their transformation efforts,” she said.

Also new to the dbt platform is a low-code, drag-and-drop visual editing environment, currently in beta, for building and exploring dbt visual models. Now generally available is advanced CI capabilities to compare code changes as part of the continuous integration process to identify issues before code is put into production. Also now generally available are data health tiles that can be embedded into downstream applications to monitor data freshness and quality.

Currently in preview are the ability to incorporate Tableau dashboards into dbt lineage and adapters support for Teradata and Athena. And an upcoming Power BI integration for the dbt Semantic Layer will allow business users in the Microsoft ecosystem to query and analyze consistent metrics, according to dbt.

Dbt also said it is teaming up with cloud application giant Salesforce to integrate Salesforce Data Cloud AI and other salesforce automation and analytics software (including Tableau and Agentforce) with the dbt Cloud and its data transformation pipeline capabilities.