Here’s a look at 10 open-source software tools – many for building AI applications or managing huge volumes of data – that are either already widely used or are gaining in popularity.

A Software Free-For-All

Open-source software tools continue to increase in popularity because of the multiple advantages they provide including lower upfront software and hardware costs, lower total-cost-of-ownership, lack of vendor lock-in, simpler license management and support from active communities.

In the following slides, as part of the CRN 2024 Year In Review (So Far) project, we take a look at some of the most popular open-source software products that have caught our attention in the first half of the year. Some of these have been around for some time and are already widely used while others are relatively new – a couple just making their debut in the last year or so – but show early signs of momentum.

Not surprisingly, the wave of AI and generative AI application development is a major driver for open-source software adoption. Many of the products on this list are in the software development space or help answer the need to manage the huge volumes of data that feed AI systems.

These products are available under open-source licenses such as the MIT License, Apache 2.0 License, GNU GPL and others. Several are products developed by startups that have received financial investments from Y Combinator, the startup accelerator and venture capital firm.

Airbyte

Airbyte is a fast-growing data integration and data movement platform for ETL/ELT data pipelines that connect applications, APIs, databases and files to data warehouses, data lakes and other destinations. Airbyte can also be used to move unstructured and semi-structured data into vector databases and large language model frameworks for AI applications.

The core Airbyte Open Source is already used by more than 40,000 companies. The software is available under multiple open-source licenses including the MIT License and Elastic License 2.0.

Airbyte, headquartered in San Francisco, also provides a number of commercial products and services around the platform. The company launched a partner program in May, including a certification course, to help technology service providers and resellers work with the Airbyte software.

Anaconda Distribution for Python

Python has become the most popular programming language overall, but it has long been used by data scientists for development in data analytics, AI and machine learning. Anaconda’s distribution of the open-source Python system is one of the most widely used data science and AI platforms.

In addition to its distribution of Python, Anaconda offers its Data Science and AI Workbench platform that data science and machine learning teams use for expediting model development and deployment while adhering to security and governance requirements.

Over the last year Anaconda has established alliances with major IT vendors to expand the use of its platform. In April Anaconda announced a partnership to integrate its Anaconda Python Repository with Teradata’s VantageCloud and ClearScape Analytics. A collaboration with IBM announced in February provides watsonx.ai users with access to the Anaconda software repository. And in August 2023 the company unveiled the Anaconda Distribution for Python in Microsoft Excel.

Apache DataFusion

The Apache Software Foundation describes DataFusion as “a fast, extensible query engine for building high quality, data-centric systems” such as database, dataframe libraries, machine learning, and streaming applications.

DataFusion can be used as an embedded SQL engine or customized and used as a foundation for building new systems with a focus on high-throughput, low-latency analytical, streaming and transaction workloads.

DataFusion leverages the technology capabilities of Apache Arrow, a language-agnostic framework for building data analytics applications that process columnar data, and the Rust programming language.

In June the Apache Software Foundation, which has been developing DataFusion since 2019 as part of the Apache Arrow project, said DataFusion is now designated as a Top-Level Project “to provide more focused governance capacity for continued growth.”

DataFusion is available for download from the Apache Software Foundation website, GitHub, and other sites under the Apache 2.0 License. The latest source release is 37.0.0.

DuckDB

DuckDB is a high-performance, in-process database that’s designed to support online analytical processing (OLAP) query workloads.

The relational (table-oriented) database supports SQL and utilizes a columnar-vectorized query execution engine that can process large batches of values in one operation as a vector, according to the Database of Databases website. The database is designed to run embedded within a host process – there is no server database to install.

DuckDB was originally developed at the Centrum Wiskunde & Informatica, the national research institute for mathematics and computer science in the Netherlands, in 2018.

DuckDB and its core extensions are open sourced under the MIT License and the entire source code is freely available on GitHub. DuckDB version 1.0.0 was just released in June and is available through the DuckDB.org website and GitHub.

One reason DuckDB has been gaining attention is the cloud analytics software developed by startup MotherDuck that runs on DuckDB.

GIMP

GIMP is an open-source photo and image editing tool that its many fans, including photographers, designers and illustrators, tout as a free alternative to the Adobe Photoshop application.

The widely popular application is used for a range of tasks including photo editing and retouching, image composition and authoring, painting, logo designing and more, according to the javapoint.com website and a Wikipedia profile.

GIMP’s functionality is generally seen as meeting the basic image editing requirements of everyday users, but lacks some advanced tools and capabilities found in Photoshop and other commercial products used by people with more intermediate and advanced graphic skills.

The cross-platform GIMP (GNU Image Manipulation Program) has been around since 1995 and is available for GNU/Linux, Windows and macOS operating systems, among others. The software is available under the GNU General Public License through the gimp.org website. The current stable version of the software is 2.10.38 (DMG revision 1) released on May 2, 2024.

Fun Fact: GIMP’s original developers included Spencer Kimball as a project at the University of California, Berkeley in 1995. Today Kimball is co-founder and CEO of next-generation database developer Cockroach Labs.

Grafana Observability Tools

Grafana is an open-source observability and data visualization platform used to collect and visualize metric, trace and log data from many data sources. It is frequently used as a component in IT/OT monitoring systems.

Grafana is developed by Grafana Labs and is available under the AGPL-3.0 open-source license. In April the company debuted Grafana 11.0 with a new Explore Metrics root cause analysis feature, improved visualizations, simpler alerting and support for additional data sources.

In addition to its flagship software, Grafana Labs develops additional open-source software including Grafana Loki, a multi-tenant log aggregation system; Grafana Tempo, back-end software for high-scale distributed tracing; and Grafana Mimir, a scalable backend metrics storage and analysis tool. Grafana Labs also sells commercial enterprise editions of its software.

MindsDB

MindsDB is an open-source virtual database and development platform that automates workflows that connect real-time data to AI systems. The software makes it easier to build, train and deploy machine learning models using SQL queries.

The software’s developer, MindsDB, was founded in 2017 and is based in San Francisco. The company says its mission with its open-source software is to democratize machine learning, according to the company’s website.

With that goal in mind, in September 2023 the company launched the MindsDB AI Collective, a network of AI startups and developers that are advancing opens-source machine learning and AI projects and providing connections to investors, technical assistance and talent.

The company is one of many open-source technology startups funded by the Y Combinator, including several on this list.

The MindsDB software is available under the open-source MIT License while MindsDB Core, the core component of the software, specifically uses the Elastic License v2.

OpenFoundry

The OpenFoundry platform provides developer infrastructure for open-source AI projects. The technology helps engineers build, deploy and scale their open-source AI “stack” 10-times faster and ship open-source, AI-powered products more quickly, according to the company’s website.

OpenFoundry was just launched this year by CEO Tyler Lehman, previously a product manager at Meta, and CTO Arthur Chi, a software engineer at Stack. The company is another open-source technology startup funded by the Y Combinator.

The OpenFoundry page on the Y Combinator website pitches the startup as an open-source alternative to the Hugging Face machine learning and data science platform. OpenFoundry is available on GitHub under the MIT License.

PyTorch

PyTorch is a powerful open-source framework and deep learning library for data scientists who are building and training deep learning models.

PyTorch is popular for such applications as computer vision, natural language processing, image classification and text generation. It can be used for a variety of algorithms including convolutional neural networks, recurrent neural networks and generative adversarial networks, according to a LinkedIn posting by data scientist and analysis expert Vitor Mesquita.

PyTorch 2.3 was released on April 24.

PyTorch was created out of the Lua-based Torch framework that came out of Facebook’s AI research lab in 2017. Today PyTorch is part of the Linux Foundation and is available through the pytorch.org website under the modified BSD license.

PyTorch and TensorFlow are generally seen as the top alternative – even competing – open-source data science and machine learning systems, according to a Projectpro.com comparison. PyTorch is often considered better for smaller-scale research projects while TensorFlow is more widely used for production-scale projects.

TensorFlow

TensorFlow is a popular open-source, end-to-end machine learning platform and library for building ML models that can run in any environment. The system handles data preprocessing, model building and model training tasks.

TensorFlow, generally seen as an alternative to PyTorch, was originally developed by the Google Brain team for internal research and production tasks, particularly around machine learning and “deep leaning” neural networks. It was originally released as open-source software under the Apache License 2.0 in November 2015.

Google continues to own and maintain TensorFlow, which is available through the tensorflow.org community website. A major update, TensorFlow 2.0, was released in September 2019.

TensorFlow and PyTorch are generally seen as the top alternative – even competing – open-source data science and machine learning systems, according to a Projectpro.com comparison. PyTorch is often considered better for smaller-scale research projects while TensorFlow is more widely used for production-scale projects.