Weaviate

Overview

Weaviate is an open-source vector database designed for high-performance indexing and querying of vector embeddings. Founded in 2019, it functions as a database that stores both objects and their vector representations, enabling efficient semantic search and other AI-driven functionalities. It is suitable for developers and technical buyers building applications that require understanding the meaning and context of data, rather than just keyword matching. Weaviate supports various data types, including text, images, and audio, by converting them into vectors using integrated machine learning models or external embedding services.

The platform is optimized for use cases such as semantic search, where users can query based on conceptual meaning rather than exact terms. It also supports recommendation systems by identifying items with similar vector embeddings, and generative AI applications by providing a knowledge base for Retrieval Augmented Generation (RAG) tasks. For instance, in a RAG pipeline, Weaviate can store document chunks as vectors, allowing a large language model (LLM) to retrieve relevant information before generating a response, thereby improving accuracy and reducing hallucinations. This approach is similar to how other vector databases, such as Qdrant, handle vector indexing for semantic similarity searches.

Weaviate offers both a self-managed open-source version (Weaviate OSS) and a managed cloud service (Weaviate Cloud). The self-managed option provides flexibility for deployment in various environments, including Kubernetes clusters, while the cloud service simplifies operations with managed infrastructure. Weaviate's architecture is designed for scalability, allowing it to handle billions of objects and deliver low-latency query responses. It provides a GraphQL API and a RESTful API, alongside client libraries in multiple programming languages, facilitating integration into existing application stacks. These features aim to reduce the complexity of developing AI-powered applications that rely on dense vector representations of data.

Key features

• Vector Indexing and Storage: Weaviate stores data objects and their corresponding vector embeddings, generated from various data types. This allows for semantic search and similarity matching based on contextual meaning rather than keywords.
• Semantic Search: Enables search queries where results are based on the conceptual similarity of content, even if exact keywords are not present. This is achieved by comparing vector embeddings of the query and the stored data.
• Generative AI & RAG Support: Functions as a knowledge base for Retrieval Augmented Generation (RAG) applications by storing vectorized data. This allows Large Language Models (LLMs) to retrieve relevant information to enhance response generation, as described in the Weaviate developers' documentation for RAG with generative models.
• Real-time Data Indexing: Capable of indexing new data continuously, ensuring that search results and recommendations are based on the most current information available.
• Module System: Extensible architecture supporting various modules for vectorization (e.g., OpenAI, Cohere, Hugging Face), question answering, and other data processing tasks.
• GraphQL and RESTful APIs: Provides flexible API interfaces for interacting with the database, allowing developers to query and manage data using either GraphQL or REST principles, detailed in the Weaviate GraphQL API reference and Weaviate RESTful API reference.
• Multi-Tenancy: Supports isolating data for different applications or users within a single Weaviate instance, which can be useful for managing resources in shared environments.
• Data Schema Management: Allows defining and managing data schemas, including properties and their data types, to structure the stored objects.
• Backup and Restore: Includes functionality for backing up and restoring data, crucial for disaster recovery and data migration.

Pricing

Weaviate offers a free sandbox environment for initial exploration, with paid plans based on resource consumption for managed cloud instances. Custom enterprise pricing is available for larger deployments.

For detailed and current pricing information, refer to the Weaviate pricing page.

Common integrations

• LangChain: Integrates with LangChain for building LLM applications, enabling Weaviate to serve as a vector store for RAG pipelines. Documentation is available on Weaviate's LangChain integration guide.
• LlamaIndex: Connects with LlamaIndex for indexing and querying data with LLMs, using Weaviate as a vector index.
• OpenAI: Supports integration with OpenAI embedding models for vectorizing data and with OpenAI's large language models for generative tasks. See Weaviate's OpenAI generative module documentation.
• Cohere: Compatible with Cohere's embedding and generative models for advanced natural language processing functionalities.
• Hugging Face: Allows use of models from the Hugging Face ecosystem for various tasks, including text embeddings and generative AI. Weaviate provides a Hugging Face generative module.
• Docker: Can be deployed using Docker containers, simplifying local development and self-hosting. Instructions are provided in the Weaviate Docker Compose guide.

Alternatives

• Pinecone: A managed vector database service focused on performance and scalability for AI applications.
• Qdrant: An open-source vector similarity search engine, offering HTTP API for searching embeddings.
• Milvus: An open-source vector database built for scalable similarity search and AI applications.
• Zilliz Cloud: A fully managed vector database service based on Milvus, designed for enterprise AI applications.
• Elasticsearch with Vector Search: While primarily a search engine, Elasticsearch can be configured for vector search using dense vector fields.

Getting started

To get started with Weaviate using its Python client, you can set up a local instance or connect to a Weaviate Cloud sandbox. The following example demonstrates how to initialize the client, create a schema, import data, and perform a semantic search.

import weaviate
import json

# 1. Connect to a Weaviate instance (e.g., local or cloud sandbox)
# For a local instance:
# client = weaviate.Client("http://localhost:8080")
# For a Weaviate Cloud sandbox:
client = weaviate.Client(
    url="YOUR_WEAVIATE_CLUSTER_URL",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="YOUR_WEAVIATE_API_KEY"),
    additional_headers={
        "X-OpenAI-Api-Key": "YOUR_OPENAI_API_KEY"  # If using OpenAI for embeddings
    }
)

# 2. Define a schema for your data
class_obj = {
    "class": "Question",
    "vectorizer": "text2vec-openai",  # Or another vectorizer like 'text2vec-transformers'
    "properties": [
        {
            "name": "question",
            "dataType": ["text"],
        },
        {
            "name": "answer",
            "dataType": ["text"],
        },
        {
            "name": "category",
            "dataType": ["text"],
        },
    ],
}

# Ensure the class doesn't already exist, then create it
if client.schema.exists("Question"):
    client.schema.delete("Question")
client.schema.create(class_obj)

# 3. Import data
data = [
    {"question": "What is the capital of France?", "answer": "Paris is the capital of France.", "category": "Geography"},
    {"question": "How many states are in the U.S.?", "answer": "There are 50 states in the U.S.", "category": "Geography"},
    {"question": "What is the largest moon of Saturn?", "answer": "Titan is the largest moon of Saturn.", "category": "Astronomy"},
    {"question": "Who wrote 'Don Quixote'?", "answer": "Miguel de Cervantes wrote 'Don Quixote'.", "category": "Literature"},
]

with client.batch as batch:
    for item in data:
        batch.add_data_object(item, "Question")

print("Data imported successfully.")

# 4. Perform a semantic search (e.g., GraphQL Get query)
response = (
    client.query
    .get("Question", ["question", "answer", "category"])
    .with_near_text({"concepts": ["French capital"]})
    .with_limit(1)
    .do()
)

print("\nSemantic search results:")
print(json.dumps(response, indent=2))

# Expected output for the example search:
# {
#   "data": {
#     "Get": {
#       "Question": [
#         {
#           "answer": "Paris is the capital of France.",
#           "category": "Geography",
#           "question": "What is the capital of France?"
#         }
#       ]
#     }
#   }
# }

This Python example initializes the Weaviate client, defines a simple schema for a 'Question' class, imports several data objects, and then executes a semantic search using with_near_text to find relevant questions based on the concept "French capital". This demonstrates the core capability of retrieving information based on semantic similarity. For more detailed getting started guides and advanced usage, refer to the Weaviate Quickstart documentation.