BentoML

Overview

BentoML is an open-source framework designed to streamline the process of taking machine learning models from development to production. It provides a structured approach for packaging ML models and their dependencies into production-ready services, often referred to as 'Bentos'. These Bentos are self-contained units that can be deployed as API endpoints, microservices, or batch inference jobs. The framework is built to be agnostic to the underlying machine learning framework, supporting popular libraries such as scikit-learn, TensorFlow, PyTorch, and XGBoost, among others.

Developers use BentoML to define their prediction services in Python, specifying the model to be served and the API endpoints for inference. The framework then handles the creation of Docker images or other deployment artifacts, encapsulating all necessary code, models, and environments. This containerization approach ensures consistency across different deployment environments and simplifies scaling. For example, a data scientist can train a model using PyTorch, then use BentoML to wrap it into a service that can be invoked via a REST API, without needing extensive DevOps expertise.

Beyond packaging, BentoML offers tools for managing the lifecycle of models, including versioning and orchestration. It integrates with various deployment targets, from local machines and virtual machines to Kubernetes clusters and serverless platforms. For organizations seeking a managed solution, BentoML also offers BentoCloud, a commercial platform that provides a fully managed service for deploying and monitoring Bentos. BentoCloud handles infrastructure provisioning, scaling, and operational tasks, allowing teams to focus on model development and iteration. This managed offering includes features like deployment dashboards, performance monitoring, and compliance certifications like BentoCloud's SOC 2 Type II compliance, addressing security and operational requirements for enterprise use cases.

BentoML is particularly well-suited for machine learning engineers and developers who need to quickly deploy and iterate on AI applications. Its Pythonic interface simplifies the creation of production-grade services from trained models, reducing the manual effort associated with dependency management and API exposure. The framework is designed to handle various inference patterns, including real-time online inference, batch processing, and asynchronous tasks, making it adaptable to diverse ML application requirements. The ability to define custom pre-processing and post-processing logic within the service further enhances its flexibility, allowing for complex data transformations before and after model inference.

The open-source nature of BentoML fosters community contributions and provides transparency in its development. The project's extensive documentation resources include quickstarts, API references, and conceptual guides, supporting developers in adopting and utilizing the framework. For comparative context, similar model serving platforms like KServe focus on Kubernetes-native deployments, offering a different architectural approach to serving models, as detailed in the KServe architecture overview. BentoML aims to provide a more opinionated and integrated developer experience for packaging and serving, abstracting some of the underlying infrastructure complexities.

Key features

• Model Packaging and Versioning: Create 'Bentos' – self-contained, deployable units that bundle models, code, and dependencies. Includes robust support for model versioning and dependency management.
• API Generation: Automatically generates production-ready REST APIs for ML models, supporting various data types and inference patterns.
• Framework Agnostic: Compatible with major machine learning frameworks such as scikit-learn, TensorFlow, PyTorch, XGBoost, and Hugging Face Transformers.
• Deployment Flexibility: Supports diverse deployment targets including local environments, Docker, Kubernetes, serverless functions, and the managed BentoCloud platform.
• Batch Inference: Provides capabilities for efficient batch processing of data for offline inference tasks, optimizing resource utilization.
• Adaptive Micro-batching: Optimizes inference throughput by dynamically grouping requests into mini-batches, reducing overhead for GPU/CPU inference.
• Model Observability: Offers tools and integrations for monitoring model performance, drift, and service health in production.
• Scalability: Designed for horizontal scalability, allowing services to handle increased inference loads by adding more instances.

Pricing

BentoML is available as an open-source framework at no cost. For managed deployments and advanced features, BentoCloud offers several tiers.

For detailed pricing and current offers, refer to the BentoCloud pricing page.

Common integrations

• Machine Learning Frameworks: Integrates directly with TensorFlow, PyTorch, scikit-learn, XGBoost, and more for model serialization and inference.
• Containerization: Built-in support for generating Docker images for service deployment.
• Orchestration: Can be deployed to Kubernetes clusters, leveraging container orchestration capabilities.
• Cloud Platforms: Integrates with major cloud providers for deployment, including AWS, Google Cloud, and Azure.
• API Gateways: Services can be exposed through various API gateways for production access and management.

Alternatives

• Seldon: An open-source MLOps platform for deploying machine learning models on Kubernetes, focusing on explainability, monitoring, and A/B testing.
• KServe: A Kubernetes-native platform for serving machine learning models, providing performant and scalable serverless inference.
• Ray Serve: A scalable, framework-agnostic model serving library built on Ray, designed for deploying ML models and arbitrary business logic.

Getting started

To begin using BentoML, install the library and create a simple service. This example demonstrates how to serve a scikit-learn model.

# -*- coding: utf-8 -*-
import bentoml
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

# Prepare a simple scikit-learn model
iris = load_iris()
X, y = iris.data, iris.target
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X, y)

# Save the model with BentoML
# The 'iris_classifier' tag will be used to reference this model
saved_model = bentoml.sklearn.save(
    "iris_classifier", model,
    # Specify framework and version for better organization
    metadata={
        "framework": "sklearn",
        "version": "1.0",
        "dataset": "iris"
    }
)

print(f"Model saved: {saved_model}")

# Define a BentoML service
# This service will expose an API endpoint for inference
# It references the saved model by its tag

# Create a new BentoML Service instance
svc = bentoml.Service("iris_classifier_service", models=[saved_model])

# Define an API runner that will be used by the service
# This runner can handle concurrent requests and batching
iris_runner = bentoml.sklearn.get("iris_classifier").to_runner()
svc.add_runner(iris_runner)

@svc.api(input=bentoml.io.NumpyNdarray(), output=bentoml.io.NumpyNdarray())
def classify(input_series: "np.ndarray") -> "np.ndarray":
    # Use the runner to make predictions
    result = iris_runner.run(input_series)
    return result

# To run this service locally:
# 1. Save the above code as service.py
# 2. Run from your terminal: bentoml serve service.py:svc --reload
#    The '--reload' flag enables live reloading during development.
# 3. Access the API at http://localhost:3000/classify and send NumPy array data.

This Python code first trains a simple scikit-learn RandomForestClassifier on the Iris dataset. It then uses bentoml.sklearn.save to persist the trained model, assigning it a tag for future reference. Next, it defines a bentoml.Service named "iris_classifier_service", linking it to the saved model through a runner. The @svc.api decorator defines an API endpoint that accepts NumPy arrays as input and returns NumPy arrays as output, using the model runner for inference. To test this locally, save the code as service.py and execute bentoml serve service.py:svc --reload from your terminal. Requests can then be sent to the generated API endpoint, typically at http://localhost:3000/classify.