KServe – Kubernetes Native Model Inference Platform

Overview

KServe (formerly KFServing) is an open-source project that provides a Kubernetes-native platform for deploying and managing machine learning (ML) models. It aims to standardize the serving of ML models across various frameworks by providing a consistent, serverless inference experience. KServe leverages Knative for its serverless capabilities, allowing inference services to scale dynamically, including scaling to zero when not in use, which can optimize resource consumption on Kubernetes clusters. This approach makes it suitable for workloads with fluctuating demand, where resources need to be provisioned and de-provisioned efficiently.

KServe is designed for developers and MLOps engineers who need to deploy trained ML models into production environments on Kubernetes. It abstracts away much of the underlying infrastructure complexity, allowing users to focus on the model itself. The platform supports a wide range of ML frameworks, including TensorFlow, PyTorch, scikit-learn, XGBoost, and ONNX, through its pre-built model servers. This multi-framework support reduces the effort required to integrate models developed using different tools into a single serving infrastructure.

The core of KServe's functionality revolves around the InferenceService Custom Resource Definition (CRD) in Kubernetes. This CRD allows users to define how their models should be deployed, specifying details such as the model's location, resource requirements, and traffic routing rules. KServe then handles the creation of necessary Kubernetes resources, including deployments, services, and ingresses, to make the model accessible for inference requests. This declarative approach aligns with Kubernetes best practices, enabling GitOps workflows for model deployment and management.

Beyond basic model deployment, KServe offers advanced features for managing the lifecycle of ML models in production. These include A/B testing, canary rollouts, and blue/green deployments, which are crucial for safely introducing new model versions and evaluating their performance in real-world scenarios. For example, a new model version can be deployed to a small percentage of user traffic (canary rollout) to monitor its stability and accuracy before a full rollout. KServe's integration with Kubernetes' native networking and traffic management capabilities, often through Istio, facilitates these advanced deployment strategies.

KServe's architecture also includes components like ModelMesh, which is optimized for high-density, low-latency serving of many models. ModelMesh reduces the overhead of running numerous individual inference services by sharing resources and efficiently loading/unloading models based on demand. This is particularly beneficial for applications requiring personalized models or a catalog of many models. Overall, KServe aims to provide a comprehensive, scalable, and resilient platform for ML model inference on Kubernetes, addressing common challenges in MLOps such as scalability, multi-framework support, and advanced deployment strategies.

Key features

• Kubernetes-native deployment: Utilizes Kubernetes Custom Resource Definitions (CRDs) for defining and managing ML inference services, integrating with Kubernetes' declarative management model.
• Multi-framework support: Provides pre-built model servers for popular ML frameworks such as TensorFlow, PyTorch, Scikit-learn, XGBoost, and ONNX, simplifying deployment across diverse ecosystems.
• Serverless inference: Leverages Knative Serving to enable automatic scaling of inference services, including scaling to zero when idle, and rapid scaling up during peak loads.
• Advanced deployment strategies: Supports A/B testing, canary rollouts, and blue/green deployments for safely introducing and validating new model versions in production.
• ModelMesh: A sub-component designed for high-density model serving, optimizing resource utilization and latency for scenarios involving many models.
• Pluggable explainers and transformers: Allows integration of custom pre-processing, post-processing, and model explainability components into the inference pipeline.
• Payload logging and metrics: Provides capabilities for logging inference requests and responses, as well as exposing standard metrics for monitoring model performance and service health.

Pricing

KServe is an open-source project released under the Apache 2.0 license, meaning it is free to use and modify. The primary costs associated with running KServe stem from the underlying Kubernetes infrastructure and cloud resources consumed for deploying and serving models.

For detailed information on cloud provider costs for Kubernetes, refer to specific provider documentation, such as Google Kubernetes Engine pricing.

Common integrations

• Kubernetes: KServe is built natively on Kubernetes, using its API and resource model for deployment and management. See the KServe installation guide for Kubernetes prerequisites.
• Knative Serving: KServe leverages Knative Serving for serverless capabilities, enabling automatic scaling and traffic management for inference services.
• Istio: Often used with KServe for advanced traffic management, M.L.B., and secure communication between services. KServe's documentation often assumes an Istio installation.
• Cloud Storage (S3, GCS, Azure Blob Storage): Models are typically stored in object storage services, which KServe can access directly for model loading. Refer to KServe's model management documentation.
• Prometheus/Grafana: For monitoring the health and performance of inference services, KServe exposes metrics that can be scraped by Prometheus and visualized in Grafana.
• MLflow: Can be integrated for tracking experiments and managing model artifacts, with KServe deploying models registered in MLflow.
• Kubeflow: KServe is an integral part of the Kubeflow MLOps platform, providing the model serving component within the broader ecosystem.

Alternatives

• Seldon Core: An open-source MLOps platform for deploying machine learning models on Kubernetes, providing similar capabilities for serving, scaling, and lifecycle management.
• Kubeflow: A broader MLOps platform for machine learning on Kubernetes, which includes serving components (KServe is often used within Kubeflow).
• Cortex Labs: An open-source platform for deploying and managing machine learning models in production, focusing on simplicity and ease of use on AWS.
• PaddlePaddle Serving: A high-performance serving framework for PaddlePaddle models, offering flexible deployment options.
• OpenVINO Model Server: A high-performance serving solution optimized for Intel hardware, focusing on efficient inference with OpenVINO models.

Getting started

To get started with KServe, you typically need a running Kubernetes cluster and Knative installed. This example demonstrates deploying a simple scikit-learn model.

First, ensure KServe is installed on your Kubernetes cluster. Refer to the KServe installation guide for detailed instructions.

Once KServe is ready, you can define an InferenceService YAML file, for instance, sklearn-isvc.yaml:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
spec:
  predictor:
    sklearn:
      storageUri: gs://kserve-samples/models/sklearn/iris
      resources:
        limits:
          memory: 2Gi
        requests:
          memory: 1Gi

This YAML defines an InferenceService named sklearn-iris that uses a pre-trained scikit-learn model stored in a Google Cloud Storage bucket. It also specifies resource limits and requests for the predictor pod.

Apply this configuration to your Kubernetes cluster:

kubectl apply -f sklearn-isvc.yaml

Monitor the status of your InferenceService:

kubectl get inferenceservice sklearn-iris

Once the service is ready, you can get the ingress URL to send inference requests:

MODEL_URL=$(kubectl get inferenceservice sklearn-iris -o jsonpath='{.status.address.url}')
echo $MODEL_URL

Finally, send an inference request using curl:

curl -v \
  -H "Content-Type: application/json" \
  -d '@./iris-input.json' \
  "${MODEL_URL}/v1/predict"

Where iris-input.json contains your input data, for example:

{
  "instances": [[6.8, 2.8, 4.8, 1.4]]
}

This will return the prediction from your deployed scikit-learn model.