Cortex Labs

Overview

Cortex Labs is a component of the Databricks platform designed to facilitate the deployment and serving of large language models (LLMs) and foundation models. It is positioned as a managed service within the Databricks ecosystem, aiming to simplify the operational aspects of bringing LLM-powered applications to production. The service provides APIs for common LLM tasks, allowing developers to integrate pre-trained or fine-tuned models into their applications without managing underlying infrastructure.

Cortex Labs targets organizations that require a streamlined approach to LLM inference, particularly those already utilizing the Databricks Lakehouse Platform for data and machine learning workloads. Its primary use cases include serving custom LLMs trained on proprietary data, integrating third-party foundation models, and enabling low-latency responses for interactive AI applications such as chatbots or content generation tools. The platform handles model scaling, endpoint management, and security, abstracting these concerns from the developer. This approach aligns with managed service offerings from other providers, which aim to reduce the operational overhead associated with complex model deployments, as discussed in a survey on LLM serving systems on arXiv. By providing a managed environment, Cortex Labs enables teams to concentrate on model development and application logic rather than infrastructure provisioning and maintenance.

Developers interact with Cortex Labs through standard HTTP APIs, allowing for language-agnostic integration into various application backends. The service supports a range of tasks, including text generation, summarization, and embeddings, typically relying on the underlying capabilities of the deployed LLM. For instance, a developer building a customer support chatbot could deploy a fine-tuned LLM on Cortex and interact with it via a REST API call to generate responses based on user queries. Similarly, a content platform could use Cortex to generate article drafts or summarize long documents. The integration within Databricks means that data scientists can transition models from experimentation and training within Databricks notebooks directly to a production serving environment with reduced configuration overhead.

Key features

• Managed LLM Endpoints: Provides fully managed infrastructure for deploying and serving LLMs, abstracting away server management, scaling, and load balancing.
• Standardized LLM APIs: Offers unified API interfaces for text generation, embeddings, and other common LLM tasks, regardless of the underlying model architecture.
• Foundation Model Access: Includes access to pre-trained foundation models directly callable via API, reducing the need for extensive in-house model development.
• Low-latency Inference: Optimized for quick response times, making it suitable for real-time applications requiring immediate LLM outputs.
• Integration with Databricks Platform: Seamlessly integrates with Databricks Machine Learning and Data Lakehouse, allowing for direct deployment of models trained within the platform.
• Security and Compliance: Adheres to industry compliance standards including SOC 2 Type II, GDPR, HIPAA, ISO 27001, and PCI DSS, supporting enterprise requirements for data governance.
• Cost Optimization: Aims to optimize resource utilization and provides cost-effective serving options through its managed infrastructure.

Pricing

Cortex Labs is part of Databricks' broader offering, and its pricing is typically integrated into custom enterprise agreements. Specific details on pricing for Cortex LLM APIs and Foundation Models are available by contacting Databricks sales. The cost structure generally accounts for factors such as the type of model served, the volume of inference requests, and the computational resources consumed.

For detailed and up-to-date pricing information, customers are directed to the Databricks product pricing page or to engage with their sales team.

Common integrations

• Databricks MLflow: For tracking experiments, packaging models, and deploying them directly to Cortex endpoints (Databricks MLflow integration guidance).
• Databricks Notebooks: To develop, train, and test LLMs before deploying them via Cortex (Databricks Notebooks documentation).
• RESTful APIs: Any application capable of making HTTP requests can integrate with Cortex endpoints for inference (Cortex LLM API reference).
• Python applications: Common for data science and AI-driven applications, utilizing standard HTTP client libraries.

Alternatives

• Anyscale: Offers a platform for deploying and scaling AI applications, including LLMs, built on Ray.
• Replicate: Provides a platform for running and hosting AI models via a cloud API, with a focus on ease of use.
• LlamaIndex: A data framework for LLM applications, assisting with data ingestion, indexing, and retrieval augmented generation (RAG) capabilities.
• AWS Bedrock: A fully managed service that makes foundation models from Amazon and leading AI startups available via an API.
• Google Cloud Vertex AI: A comprehensive platform for building, deploying, and scaling machine learning models, including MLOps tools for LLMs.

Getting started

To get started with Cortex Labs, you typically interact with its APIs after deploying a model within your Databricks workspace. The following Python example demonstrates how to call a deployed Cortex LLM API endpoint for text generation.

import requests
import json

# Replace with your actual Databricks workspace URL and personal access token
databricks_workspace_url = "https://<your-workspace-id>.databricks.com"
databricks_token = "dapi<your-personal-access-token>"

# Replace with the name of your deployed Cortex LLM endpoint
endpoint_name = "my-llm-endpoint"

# Construct the API URL for the deployed endpoint
api_url = f"{databricks_workspace_url}/serving-endpoints/{endpoint_name}/invocations"

headers = {
    "Authorization": f"Bearer {databricks_token}",
    "Content-Type": "application/json",
}

# Define the payload for text generation
payload = {
    "dataframe_split": {
        "columns": ["prompt"],
        "data": [["Write a short, compelling paragraph about the benefits of cloud computing."]]
    }
}

try:
    response = requests.post(api_url, headers=headers, json=payload)
    response.raise_for_status() # Raise an exception for bad status codes

    result = response.json()
    print("Generated Text:")
    for prediction in result.get("predictions", []):
        print(prediction)

except requests.exceptions.RequestException as e:
    print(f"API request failed: {e}")
    if response is not None:
        print(f"Response Status Code: {response.status_code}")
        print(f"Response Body: {response.text}")

This Python code snippet outlines the process of authenticating with the Databricks workspace using a personal access token, constructing the API request URL for a specific Cortex LLM endpoint, and sending a JSON payload for text generation. The response contains the model's generated text. Before executing this, users must ensure they have a Databricks workspace, an active personal access token with appropriate permissions, and an LLM deployed as an endpoint within Cortex (Databricks documentation on deploying LLM endpoints). The specific structure of the payload might vary based on the deployed model and the expected input format, but dataframe_split is a common format for Databricks model serving. Further details on API invocation and payload structures are available in the official Cortex LLM API documentation.