构建语义搜索引擎

本教程将让您熟悉 LangChain 的文档加载器、嵌入和 vector store 抽象。这些抽象旨在支持从（矢量）数据库和其他来源检索数据，以便与 LLM 工作流集成。它们对于获取数据作为模型推理的一部分进行推理的应用程序非常重要，例如检索增强生成或 RAG（请参阅此处的 RAG 教程）。

在这里，我们将在 PDF 文档上构建一个搜索引擎。这将允许我们在 PDF 中检索类似于输入查询的段落。

概念

本指南重点介绍文本数据的检索。我们将介绍以下概念：

文档和文档加载器;
文本拆分器;
嵌入;
向量存储和检索器。

设置

Jupyter 笔记本

本教程和其他教程可能在 Jupyter 笔记本中运行最方便。有关如何安装的说明，请参阅此处。

安装

本教程需要langchain-community和pypdf包：

果仁
康达

pip install langchain-community pypdf

conda install langchain-community pypdf -c conda-forge

有关更多详细信息，请参阅我们的安装指南。

LangSmith

您使用 LangChain 构建的许多应用程序将包含多个步骤，其中包含多次调用 LLM 调用。随着这些应用程序变得越来越复杂，能够检查您的链条或代理内部到底发生了什么变得至关重要。最好的方法是使用 LangSmith。

在上面的链接中注册后，请确保设置环境变量以开始记录跟踪：

export LANGSMITH_TRACING="true"
export LANGSMITH_API_KEY="..."

或者，如果在笔记本中，您可以通过以下方式设置它们：

import getpass
import os

os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = getpass.getpass()

文档和文档加载器

LangChain 实现了一个 Document 抽象，它旨在表示一个文本单元和关联的元数据。它有三个属性：

page_content：表示内容的字符串;
metadata：包含任意元数据的 dict;
id：（可选）文档的字符串标识符。

这metadata属性可以捕获有关文档来源、文档与其他文档的关系以及其他信息的信息。请注意，单个Documentobject 通常表示较大文档的一个块。

我们可以根据需要生成示例文档：

from langchain_core.documents import Document

documents = [
    Document(
        page_content="Dogs are great companions, known for their loyalty and friendliness.",
        metadata={"source": "mammal-pets-doc"},
    ),
    Document(
        page_content="Cats are independent pets that often enjoy their own space.",
        metadata={"source": "mammal-pets-doc"},
    ),
]

API 参考：文档

但是，LangChain 生态系统实现了与数百个常见源集成的文档加载器。这使得将这些来源的数据整合到您的 AI 应用程序中变得容易。

加载文档

让我们将 PDF 加载到Document对象。此处的 LangChain 存储库中有一个示例 PDF — 2023 年为 Nike 提交的 10-k 文件。我们可以查阅 LangChain 文档以获取可用的 PDF 文档加载器。让我们选择 PyPDFLoader，它是相当轻量级的。

from langchain_community.document_loaders import PyPDFLoader

file_path = "../example_data/nke-10k-2023.pdf"
loader = PyPDFLoader(file_path)

docs = loader.load()

print(len(docs))

API 参考：PyPDFLoader

提示

有关 PDF 文档加载器的更多详细信息，请参阅本指南。

PyPDFLoader加载一个Document对象。对于每个，我们都可以轻松访问：

页面的字符串内容;
包含文件名和页码的元数据。

print(f"{docs[0].page_content[:200]}\n")
print(docs[0].metadata)

Table of Contents
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
Washington, D.C. 20549
FORM 10-K
(Mark One)
☑ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934
FO

{'source': '../example_data/nke-10k-2023.pdf', 'page': 0}

分裂

对于信息检索和下游问答目的，页面的表示形式可能过于粗糙。我们最终的目标是找回Document对象，并进一步拆分我们的 PDF 将有助于确保文档相关部分的含义不会被周围的文本“冲淡”。

为此，我们可以使用文本拆分器。在这里，我们将使用一个简单的文本拆分器，它根据字符进行分区。我们将文档拆分为 1000 个字符的块块之间有 200 个字符的重叠。重叠有助于减少将 statement 与 important 分开的可能性上下文。我们使用 RecursiveCharacterTextSplitter，它将使用常见的分隔符递归地拆分文档，例如新行，直到每个块的大小都合适。这是推荐用于通用文本使用案例的文本拆分器。

我们设置add_start_index=True这样，字符索引中每个 split Document 在初始 Document 中开始，将保留为元数据属性 “start_index”。

请参阅本指南以了解有关使用 PDF 的更多详细信息，包括如何从特定部分和图像中提取文本。

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=200, add_start_index=True
)
all_splits = text_splitter.split_documents(docs)

len(all_splits)

API 参考：RecursiveCharacterTextSplitter

嵌入

矢量搜索是存储和搜索非结构化数据（如非结构化文本）的常用方法。这个想法是存储与文本关联的数字向量。给定一个查询，我们可以将其嵌入为相同维度的向量，并使用向量相似度指标（例如余弦相似度）来识别相关文本。

LangChain 支持来自数十个提供商的嵌入。这些模型指定如何将文本转换为数值向量。我们选择一个模型：

选择嵌入模型：

pip install -qU langchain-openai

import getpass
import os

if not os.environ.get("OPENAI_API_KEY"):
  os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter API key for OpenAI: ")

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

vector_1 = embeddings.embed_query(all_splits[0].page_content)
vector_2 = embeddings.embed_query(all_splits[1].page_content)

assert len(vector_1) == len(vector_2)
print(f"Generated vectors of length {len(vector_1)}\n")
print(vector_1[:10])

Generated vectors of length 1536

[-0.008586574345827103, -0.03341241180896759, -0.008936782367527485, -0.0036674530711025, 0.010564599186182022, 0.009598285891115665, -0.028587326407432556, -0.015824200585484505, 0.0030416189692914486, -0.012899317778646946]

有了生成文本嵌入的模型，我们接下来可以将它们存储在支持高效相似性搜索的特殊数据结构中。

矢量存储

LangChain VectorStore 对象包含用于添加文本和Document对象添加到 store 中，并使用各种相似性指标查询它们。它们通常使用嵌入模型进行初始化，嵌入模型确定如何将文本数据转换为数字向量。

LangChain 包括一套与不同向量存储技术的集成。某些矢量存储由提供商（例如，各种云提供商）托管，需要特定的凭证才能使用;一些（例如 Postgres）在单独的基础设施中运行，这些基础设施可以在本地或通过第三方运行;其他 CPU 可以在内存中运行轻量级工作负载。让我们选择一个 vector store：

选择载体存储：

pip install -qU langchain-core

from langchain_core.vectorstores import InMemoryVectorStore

vector_store = InMemoryVectorStore(embeddings)

实例化我们的 vector store 后，我们现在可以为文档编制索引。

ids = vector_store.add_documents(documents=all_splits)

请注意，大多数 vector store 实现都允许您连接到现有的 vector store —— 例如，通过提供 client、索引名称或其他信息。有关更多详细信息，请参阅特定集成的文档。

一旦我们实例化了VectorStore，我们可以查询它。VectorStore 包括用于查询的方法：

同步和异步;
按字符串查询和按向量;
返回和不返回相似性分数;
通过相似性和最大边际相关性（以平衡相似性与查询与检索结果中的多样性）。

这些方法的输出中通常包括 Document 对象的列表。

用法

嵌入向量通常将文本表示为“密集”向量，以便具有相似含义的文本在几何上接近。这样，我们只需传入一个问题即可检索相关信息，而无需了解文档中使用的任何特定关键术语。

根据与字符串查询的相似性返回文档：

results = vector_store.similarity_search(
    "How many distribution centers does Nike have in the US?"
)

print(results[0])

page_content='direct to consumer operations sell products through the following number of retail stores in the United States:
U.S. RETAIL STORES NUMBER
NIKE Brand factory stores 213 
NIKE Brand in-line stores (including employee-only stores) 74 
Converse stores (including factory stores) 82 
TOTAL 369 
In the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.
2023 FORM 10-K 2' metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}

异步查询：

results = await vector_store.asimilarity_search("When was Nike incorporated?")

print(results[0])

page_content='Table of Contents
PART I
ITEM 1. BUSINESS
GENERAL
NIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this "Annual Report"), the terms "we," "us," "our,"
"NIKE" and the "Company" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.
Our principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is
the largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores
and sales through our digital platforms (also referred to as "NIKE Brand Digital"), to retail accounts and to a mix of independent distributors, licensees and sales' metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}

返回分数：

# Note that providers implement different scores; the score here
# is a distance metric that varies inversely with similarity.

results = vector_store.similarity_search_with_score("What was Nike's revenue in 2023?")
doc, score = results[0]
print(f"Score: {score}\n")
print(doc)

Score: 0.23699893057346344

page_content='Table of Contents
FISCAL 2023 NIKE BRAND REVENUE HIGHLIGHTS
The following tables present NIKE Brand revenues disaggregated by reportable operating segment, distribution channel and major product line:
FISCAL 2023 COMPARED TO FISCAL 2022
•NIKE, Inc. Revenues were $51.2 billion in fiscal 2023, which increased 10% and 16% compared to fiscal 2022 on a reported and currency-neutral basis, respectively.
The increase was due to higher revenues in North America, Europe, Middle East & Africa ("EMEA"), APLA and Greater China, which contributed approximately 7, 6,
2 and 1 percentage points to NIKE, Inc. Revenues, respectively.
•NIKE Brand revenues, which represented over 90% of NIKE, Inc. Revenues, increased 10% and 16% on a reported and currency-neutral basis, respectively. This
increase was primarily due to higher revenues in Men's, the Jordan Brand, Women's and Kids' which grew 17%, 35%,11% and 10%, respectively, on a wholesale
equivalent basis.' metadata={'page': 35, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}

根据与嵌入式查询的相似性返回文档：

embedding = embeddings.embed_query("How were Nike's margins impacted in 2023?")

results = vector_store.similarity_search_by_vector(embedding)
print(results[0])

page_content='Table of Contents
GROSS MARGIN
FISCAL 2023 COMPARED TO FISCAL 2022
For fiscal 2023, our consolidated gross profit increased 4% to $22,292 million compared to $21,479 million for fiscal 2022. Gross margin decreased 250 basis points to
43.5% for fiscal 2023 compared to 46.0% for fiscal 2022 due to the following:
*Wholesale equivalent
The decrease in gross margin for fiscal 2023 was primarily due to:
•Higher NIKE Brand product costs, on a wholesale equivalent basis, primarily due to higher input costs and elevated inbound freight and logistics costs as well as
product mix;
•Lower margin in our NIKE Direct business, driven by higher promotional activity to liquidate inventory in the current period compared to lower promotional activity in
the prior period resulting from lower available inventory supply;
•Unfavorable changes in net foreign currency exchange rates, including hedges; and
•Lower off-price margin, on a wholesale equivalent basis.
This was partially offset by:' metadata={'page': 36, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}

了解更多信息：

Retriever

LangChain 语言链VectorStoreobjects 不子类 Runnable。LangChain Retriever 是 Runnables，因此它们实现了一组标准的方法（例如，同步和异步invoke和batch作）。尽管我们可以从向量存储构建检索器，但检索器也可以与非向量存储数据源（例如外部 API）交互。

我们可以自己创建一个 this 的简单版本，而无需子类化Retriever.如果我们选择希望使用什么方法来检索文档，我们可以轻松地创建一个 runnable。下面我们将围绕similarity_search方法：

from typing import List

from langchain_core.documents import Document
from langchain_core.runnables import chain


@chain
def retriever(query: str) -> List[Document]:
    return vector_store.similarity_search(query, k=1)


retriever.batch(
    [
        "How many distribution centers does Nike have in the US?",
        "When was Nike incorporated?",
    ],
)

API 参考：文档 | Chains

[[Document(metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}, page_content='direct to consumer operations sell products through the following number of retail stores in the United States:\nU.S. RETAIL STORES NUMBER\nNIKE Brand factory stores 213 \nNIKE Brand in-line stores (including employee-only stores) 74 \nConverse stores (including factory stores) 82 \nTOTAL 369 \nIn the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.\n2023 FORM 10-K 2')],
 [Document(metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}, page_content='Table of Contents\nPART I\nITEM 1. BUSINESS\nGENERAL\nNIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this "Annual Report"), the terms "we," "us," "our,"\n"NIKE" and the "Company" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\nOur principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\nthe largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\nand sales through our digital platforms (also referred to as "NIKE Brand Digital"), to retail accounts and to a mix of independent distributors, licensees and sales')]]

Vectorstores 实现了一个as_retriever方法，它将生成一个 Retriever，特别是 VectorStoreRetriever。这些检索器包括特定的search_type和search_kwargs属性，这些属性标识要调用的基础向量存储的方法，以及如何参数化它们。例如，我们可以用下面的代码来复制上面的内容：

retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 1},
)

retriever.batch(
    [
        "How many distribution centers does Nike have in the US?",
        "When was Nike incorporated?",
    ],
)

[[Document(metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}, page_content='direct to consumer operations sell products through the following number of retail stores in the United States:\nU.S. RETAIL STORES NUMBER\nNIKE Brand factory stores 213 \nNIKE Brand in-line stores (including employee-only stores) 74 \nConverse stores (including factory stores) 82 \nTOTAL 369 \nIn the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.\n2023 FORM 10-K 2')],
 [Document(metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}, page_content='Table of Contents\nPART I\nITEM 1. BUSINESS\nGENERAL\nNIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this "Annual Report"), the terms "we," "us," "our,"\n"NIKE" and the "Company" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\nOur principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\nthe largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\nand sales through our digital platforms (also referred to as "NIKE Brand Digital"), to retail accounts and to a mix of independent distributors, licensees and sales')]]

VectorStoreRetriever支持的搜索类型"similarity"（默认）、"mmr"（最大边际相关性，如上所述）和"similarity_score_threshold".我们可以使用后者通过相似性分数对检索器输出的文档进行阈值限制。

检索器可以很容易地合并到更复杂的应用程序中，例如检索增强生成（RAG）应用程序，这些应用程序将给定问题与检索到的上下文合并到 LLM 的提示中。要了解有关构建此类应用程序的更多信息，请查看 RAG 教程教程。

了解更多信息：

检索策略可能很丰富，也很复杂。例如：

我们可以从查询中推断出硬规则和过滤器（例如，“使用 2020 年之后发布的文档”）;
我们可以以某种方式返回链接到检索到的上下文的文档（例如，通过某些文档分类法）;
我们可以为每个上下文单元生成多个 embedding;
我们可以集成来自多个检索器的结果;
我们可以为文档分配权重，例如，将最近的文档的权重提高。

操作指南的检索器部分介绍了这些策略和其他内置检索策略。

扩展 BaseRetriever 类以实现自定义检索器也很简单。在此处查看我们的操作指南。

后续步骤

您现在已经了解了如何在 PDF 文档上构建语义搜索引擎。

有关文档加载器的更多信息：

有关嵌入的更多信息：

有关载体存储的更多信息：

有关 RAG 的更多信息，请参阅：