如何在执行提取时处理长文本

在处理文件（如 PDF）时，您可能会遇到超出语言模型上下文窗口的文本。要处理此文本，请考虑以下策略：

1. 更改 LLM选择支持更大上下文窗口的其他 LLM。
2. 蛮 力对文档进行分块，并从每个分块中提取内容。
3. 抹布对文档进行分块，为数据块编制索引，并且仅从看起来“相关”的数据块子集中提取内容。

请记住，这些策略有不同的权衡，最佳策略可能取决于您正在设计的应用程序！

本指南演示了如何实施策略 2 和 3。

设置

首先，我们将安装本指南所需的依赖项：

%pip install -qU langchain-community lxml faiss-cpu langchain-openai

Note: you may need to restart the kernel to use updated packages.

现在我们需要一些示例数据！让我们从 wikipedia 下载一篇关于 cars 的文章，并将其作为 LangChain 文档加载。

import re

import requests
from langchain_community.document_loaders import BSHTMLLoader

# Download the content
response = requests.get("https://en.wikipedia.org/wiki/Car")
# Write it to a file
with open("car.html", "w", encoding="utf-8") as f:
    f.write(response.text)
# Load it with an HTML parser
loader = BSHTMLLoader("car.html")
document = loader.load()[0]
# Clean up code
# Replace consecutive new lines with a single new line
document.page_content = re.sub("\n\n+", "\n", document.page_content)

API 参考：BSHTMLLoader

print(len(document.page_content))

78865

定义架构

在提取教程之后，我们将使用 Pydantic 来定义我们希望提取的信息架构。在这种情况下，我们将提取一份 “关键发展”（例如，重要的历史事件）列表，其中包括年份和描述。

请注意，我们还包含一个evidence键，并指示模型逐字提供文章中的相关文本句子。这允许我们将提取结果与原始文档中的文本（模型的重建）进行比较。

from typing import List, Optional

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from pydantic import BaseModel, Field


class KeyDevelopment(BaseModel):
    """Information about a development in the history of cars."""

    year: int = Field(
        ..., description="The year when there was an important historic development."
    )
    description: str = Field(
        ..., description="What happened in this year? What was the development?"
    )
    evidence: str = Field(
        ...,
        description="Repeat in verbatim the sentence(s) from which the year and description information were extracted",
    )


class ExtractionData(BaseModel):
    """Extracted information about key developments in the history of cars."""

    key_developments: List[KeyDevelopment]


# Define a custom prompt to provide instructions and any additional context.
# 1) You can add examples into the prompt template to improve extraction quality
# 2) Introduce additional parameters to take context into account (e.g., include metadata
#    about the document from which the text was extracted.)
prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an expert at identifying key historic development in text. "
            "Only extract important historic developments. Extract nothing if no important information can be found in the text.",
        ),
        ("human", "{text}"),
    ]
)

API 参考：ChatPromptTemplate | MessagesPlaceholder

创建提取器

我们选择一个 LLM。由于我们使用的是工具调用，因此需要一个支持工具调用功能的模型。有关可用的 LLM，请参阅此表。

选择聊天模式：

OpenAI

Anthropic

天蓝色

谷歌双子座

谷歌顶点

AWS

Groq

Cohere

英伟达

烟花 AI

Mistral AI

携手 AI

IBM 沃森

Databricks

xAI

困惑

pip install -qU "langchain[openai]"

import getpass
import os

if not os.environ.get("OPENAI_API_KEY"):
  os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter API key for OpenAI: ")

from langchain.chat_models import init_chat_model

llm = init_chat_model("gpt-4o", model_provider="openai", temperature=0)

extractor = prompt | llm.with_structured_output(
    schema=ExtractionData,
    include_raw=False,
)

蛮力方法

将文档拆分为多个块，以便每个块都适合 LLM 的上下文窗口。

from langchain_text_splitters import TokenTextSplitter

text_splitter = TokenTextSplitter(
    # Controls the size of each chunk
    chunk_size=2000,
    # Controls overlap between chunks
    chunk_overlap=20,
)

texts = text_splitter.split_text(document.page_content)

API 参考：TokenTextSplitter

使用 batch 功能在每个 chunk 中并行运行提取！

提示

您通常可以使用 .batch（） 来并行化提取！.batch在后台使用 ThreadPool 来帮助您并行化工作负载。

如果您的模型是通过 API 公开的，这可能会加快您的提取流程！

# Limit just to the first 3 chunks
# so the code can be re-run quickly
first_few = texts[:3]

extractions = extractor.batch(
    [{"text": text} for text in first_few],
    {"max_concurrency": 5},  # limit the concurrency by passing max concurrency!
)

合并结果

从各个数据块中提取数据后，我们需要将提取合并在一起。

key_developments = []

for extraction in extractions:
    key_developments.extend(extraction.key_developments)

key_developments[:10]

[KeyDevelopment(year=1769, description='Nicolas-Joseph Cugnot built the first steam-powered road vehicle.', evidence='The French inventor Nicolas-Joseph Cugnot built the first steam-powered road vehicle in 1769, while the Swiss inventor François Isaac de Rivaz designed and constructed the first internal combustion-powered automobile in 1808.'),
 KeyDevelopment(year=1808, description='François Isaac de Rivaz designed and constructed the first internal combustion-powered automobile.', evidence='The French inventor Nicolas-Joseph Cugnot built the first steam-powered road vehicle in 1769, while the Swiss inventor François Isaac de Rivaz designed and constructed the first internal combustion-powered automobile in 1808.'),
 KeyDevelopment(year=1886, description='Carl Benz invented the modern car, a practical, marketable automobile for everyday use, and patented his Benz Patent-Motorwagen.', evidence='The modern car—a practical, marketable automobile for everyday use—was invented in 1886, when the German inventor Carl Benz patented his Benz Patent-Motorwagen.'),
 KeyDevelopment(year=1901, description='The Oldsmobile Curved Dash became the first mass-produced car.', evidence='The 1901 Oldsmobile Curved Dash and the 1908 Ford Model T, both American cars, are widely considered the first mass-produced[3][4] and mass-affordable[5][6][7] cars, respectively.'),
 KeyDevelopment(year=1908, description='The Ford Model T became the first mass-affordable car.', evidence='The 1901 Oldsmobile Curved Dash and the 1908 Ford Model T, both American cars, are widely considered the first mass-produced[3][4] and mass-affordable[5][6][7] cars, respectively.'),
 KeyDevelopment(year=1885, description='Carl Benz built the original Benz Patent-Motorwagen, the first modern car.', evidence='The original Benz Patent-Motorwagen, the first modern car, built in 1885 and awarded the patent for the concept'),
 KeyDevelopment(year=1881, description='Gustave Trouvé demonstrated a three-wheeled car powered by electricity.', evidence='In November 1881, French inventor Gustave Trouvé demonstrated a three-wheeled car powered by electricity at the International Exposition of Electricity.'),
 KeyDevelopment(year=1888, description="Bertha Benz undertook the first road trip by car to prove the road-worthiness of her husband's invention.", evidence="In August 1888, Bertha Benz, the wife and business partner of Carl Benz, undertook the first road trip by car, to prove the road-worthiness of her husband's invention."),
 KeyDevelopment(year=1896, description='Benz designed and patented the first internal-combustion flat engine, called boxermotor.', evidence='In 1896, Benz designed and patented the first internal-combustion flat engine, called boxermotor.'),
 KeyDevelopment(year=1897, description='The first motor car in central Europe and one of the first factory-made cars in the world was produced by Czech company Nesselsdorfer Wagenbau (later renamed to Tatra), the Präsident automobil.', evidence='The first motor car in central Europe and one of the first factory-made cars in the world, was produced by Czech company Nesselsdorfer Wagenbau (later renamed to Tatra) in 1897, the Präsident automobil.')]

基于 RAG 的方法

另一个简单的想法是将文本分块，但不是从每个块中提取信息，而是只关注最相关的块。

谨慎

可能很难确定哪些数据块是相关的。

例如，在car文章，文章的大部分内容都包含关键的开发信息。因此，通过使用 RAG，我们可能会丢弃大量相关信息。

我们建议您尝试您的用例并确定此方法是否有效。

要实施基于 RAG 的方法，请执行以下作：

1. 对文档进行分块并为其编制索引（例如，在 vectorstore 中）;
2. 在extractorchain 替换为 VectorStore 的检索步骤。

下面是一个依赖于FAISSvectorstore 中。

from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document
from langchain_core.runnables import RunnableLambda
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import CharacterTextSplitter

texts = text_splitter.split_text(document.page_content)
vectorstore = FAISS.from_texts(texts, embedding=OpenAIEmbeddings())

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 1}
)  # Only extract from first document

API 参考：FAISS | 文档 | RunnableLambda | OpenAI 嵌入 | CharacterTextSplitter（字符文本拆分器）

在这种情况下，RAG 提取器仅查看顶部文档。

rag_extractor = {
    "text": retriever | (lambda docs: docs[0].page_content)  # fetch content of top doc
} | extractor

results = rag_extractor.invoke("Key developments associated with cars")

for key_development in results.key_developments:
    print(key_development)

year=2006 description='Car-sharing services in the US experienced double-digit growth in revenue and membership.' evidence='in the US, some car-sharing services have experienced double-digit growth in revenue and membership growth between 2006 and 2007.'
year=2020 description='56 million cars were manufactured worldwide, with China producing the most.' evidence='In 2020, there were 56 million cars manufactured worldwide, down from 67 million the previous year. The automotive industry in China produces by far the most (20 million in 2020).'

常见问题

不同的方法在成本、速度和准确性方面各有优缺点。

请注意这些问题：

• 对内容进行分块意味着，如果信息分布在多个数据块中，则 LLM 可能无法提取信息。
• 大块重叠可能会导致相同的信息被提取两次，因此请准备好删除重复数据！
• LLM 可以组成数据。如果在大型文本中查找单个事实并使用蛮力方法，您最终可能会获得更多虚构的数据。