Here’s a look at 10 data science and machine learning tools that solution and service providers should be aware of.

Deep Thoughts

Data science and machine learning technologies have long been important for data analytics tasks and predictive analytical software. But with the wave of artificial intelligence and generative AI development in 2023, the importance of data science and machine learning tools has risen to new heights.

One absolute truth about AI systems is that they need huge amounts of data to be effective.

Data science combines math and statistics, advanced analytics, specialized programming and other skills and tools to help uncover actionable insights within an organization’s data. The global data science tool market reached $8.73 billion last year and will nearly double to $16.85 billion by 2030, according to 24MarketReports.

Machine learning systems make business-outcome decisions and predictions based on algorithms and statistical models that analyze and draw inferences from huge amounts of data. The worldwide machine learning market is expected to reach $79.29 billion this year, according to Statista, and grow at a 36 percent CAGR to $503.40 billion by 2030

Here’s a look at some of the hottest data science and machine learning tools in use today. Some of the following tools are relatively new to the market while others have been around for a while and recently updated. The list also includes both commercial products and open-source software.

Amazon SageMaker

Amazon SageMaker is one of Amazon Web Services’ (AWS) flagship AI and machine learning software tools – and is one of the most prominent machine learning products in the industry.

In November, at the AWS re:Invent extravaganza, AWS expanded SageMaker’s functionalities with five new capabilities that the company said helps accelerate the building, training and deployment of large language models and other foundation machine learning models that power generative AI.

One new capability enhances SageMaker’s ability to scale models by accelerating model training time while another optimizes managed ML infrastructure operations by reducing deployment costs and model latency.

The new SageMaker Clarify tool makes it easier to select the right model based on quality parameters that support responsible use of AI. A new no-code feature in SageMaker Canvas makes it possible to prepare data using natural language instructions. And Canvas continues to “democratize” model building and customization, AWS said, by making it easier to use models to extract insights, make predictions and generate content using an organization’s proprietary data.

AWS also offers Amazon Machine Learning, a more highly automated tool for building machine learning models.

Anaconda Distribution for Python

Python has become the most popular programming language overall, but it has long been used by data scientists for development in data analytics, AI and machine learning. Anaconda’s distribution of the open-source Python system is one of the most widely used data science and AI platforms.

In addition to its distribution of Python, Anaconda offers its Data Science and AI Workbench platform that data science and machine learning teams use for expediting model development and deployment while adhering to security and governance requirements.

Over the last year Anaconda has established alliances with major IT vendors to expand the use of its platform. In April Anaconda announced a partnership to integrate its Anaconda Python Repository with Teradata’s VantageCloud and ClearScape Analytics. A collaboration with IBM announced in February provides watsonx.ai users with access to the Anaconda software repository. And in August 2023 the company unveiled the Anaconda Distribution for Python in Microsoft Excel.

ClearML

ClearML’s platform, designed for data scientists and data engineers, automates and simplifies the development and management of machine learning solutions. The system provides a comprehensive lineup of capabilities spanning data science, data management, MLOps, and model orchestration and deployment.

In March startup ClearML added new orchestration capabilities to its platform to expand control over AI infrastructure management and compute costs while maximizing the use of compute resources and improving model serving visibility.

Also in March introduced an open-source fractional GPU tool to help business utilize their GPU utilization by enabling multi-tenancy for all Nvidia GPUs.

Databricks Mosaic AI

At Databricks’ recent Data + AI Summit the company unveiled a number of new capabilities for its Mosaic AI software for building and deploying production-quality ML and GenAI applications.

Databricks acquired MosaicML in June 2023 in a blockbuster $1.3-billion deal and has been integrating the startup’s technology with its data lakehouse platform. (Databricks has since rebranded the product as Mosaic AI.)

The latest capabilities in Mosaic AI include support for building compound AI systems, new functionality to improve model quality, and AI governance tools. Databricks said the innovations give users “the confidence to build and measure production-quality applications, delivering on the promises of generative AI for their business.”

Dataiku

The Dataiku platform offers a comprehensive lineup of data science, machine learning and AI capabilities including machine learning development, MLOps, data preparation, DataOps, visualization, analytical applications and generative AI.

In September 2023, Dataiku launched LLM Mesh, a new tool for integrating large language models within the enterprise that the company called “the common backbone” for Gen AI applications. LLM Mesh capabilities include universal AI service routing, secure access and auditing for AI services, performance and cost tracking, and safety provisions for private data screening and response moderation.

In April, Dataiku debuted LLM Cost Guard, a new capability within LLM Mesh that creates standards for tracking and optimizing generative AI use cases.

dotData Feature Factory 1.1

dotData’s Feature Factory is an automated feature discovery and engineering platform that helps data scientists find and use data features within large-scale data sets for use in AI and machine learning projects.

In Feature Factory version 1.1, introduced in May, the company provided significant enhancements including new data quality assessment capabilities, support for user-defined features and interactive feature selection, improved support for AutoML through the Python-based PyCaret AutoML library, and preview support for generative AI feature discovery.

Hopsworks MLOps Platform

The Hopsworks platform is used to develop, deploy and monitor AI/ML models at scale.

The core of the serverless system is its machine learning feature store for storing data for ML models running on AWS, Azure and Google Cloud platforms and in on-premises systems. The Hopsworks platform also provides machine learning pipelines and a comprehensive development toolkit.

Hopsworks 3.7, what the company called the “GenAI release,” became generally available in March with new capabilities to support GenAI and large language model use cases. It also introduced feature monitoring, a new notification service to track changes to specific features, and support for the Delta Lake data storage format.

Founded in Sweden in 2016, Hopsworks has offices in Stockholm, London and Palo Alto, Calif.

Obviously AI

A problem faced by many businesses is the shortage of people with data science and machine learning expertise. Obviously AI looks to close that gap with its no-code AI/ML platform that allows people without technical backgrounds to build and train machine learning models.

The platform helps quickly build models that run predictions on historical data, everything from sales and revenue forecasting to predictions about energy consumption and population growth.

“Because data science shouldn’t feel like rocket science,” the company’s web site says.

PyTorch

PyTorch is a powerful open-source framework and deep learning library for data scientists who are building and training deep learning models.

PyTorch is popular for such applications as computer vision, natural language processing, image classification and text generation. It can be used for a variety of algorithms including convolutional neural networks, recurrent neural networks and generative adversarial networks, according to a LinkedIn posting by data scientist and analysis expert Vitor Mesquita.

PyTorch 2.3 was released on April 24.

PyTorch was created out of the Lua-based Torch framework that came out of Facebook’s AI research lab in 2017. Today PyTorch is part of the Linux Foundation and is available through the pytorch.org website.

PyTorch and TensorFlow are generally seen as the top alternative – even competing – open-source data science and machine learning systems, according to a Projectpro.com comparison. PyTorch is often considered better for smaller-scale research projects while TensorFlow is more widely used for production-scale projects.

TensorFlow

TensorFlow is a popular open-source, end-to-end machine learning platform and library for building ML models that can run in any environment. The system handles data preprocessing, model building and model training tasks.

TensorFlow, generally seen as an alternative to PyTorch, was originally developed by the Google Brain team for internal research and production tasks, particularly around machine learning and “deep leaning” neural networks. It was originally released as open-source software under the Apache License 2.0 in November 2015.

Google continues to own and maintain TensorFlow, which is available through the tensorflow.org community website. A major update, TensorFlow 2.0, was released in September 2019.

TensorFlow and PyTorch are generally seen as the top alternative – even competing – open-source data science and machine learning systems, according to a Projectpro.com comparison. PyTorch is often considered better for smaller-scale research projects while TensorFlow is more widely used for production-scale projects.