Llamafile

Llamafile 让您可以通过单个文件来分发和运行大语言模型（LLM）。

Llamafile 通过将 llama.cpp 与 Cosmopolitan Libc 结合在一起，形成一个框架，将大语言模型（LLM）的所有复杂性简化为一个单文件可执行程序（称为“llamafile”），可在大多数计算机上本地运行，且无需安装。

设置

1. 下载你想要使用的模型的llamafile。你可以在HuggingFace上找到许多llamafile格式的模型。在本指南中，我们将下载一个较小的模型：TinyLlama-1.1B-Chat-v1.0.Q5_K_M。注意：如果你没有wget，也可以直接通过此链接下载模型。

wget https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile

2. 将 llamafile 变为可执行文件。首先，如果尚未打开终端，请先打开一个终端。如果您使用的是 MacOS、Linux 或 BSD，您需要使用 chmod 授予计算机执行此新文件的权限（见下文）。如果您使用的是 Windows，请通过在文件名末尾添加 \".exe\" 来重命名文件（模型文件应命名为 TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile.exe）。

chmod +x TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile  # run if you're on MacOS, Linux, or BSD

3. 以“服务器模式”运行llamafile：

./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --server --nobrowser

现在你可以调用 llamafile 的 REST API 了。默认情况下，llamafile 服务器在 http://localhost:8080 监听。你可以在 这里 查看完整的服务器文档。你可以通过 REST API 直接与 llamafile 进行交互，但在这里我们将展示如何使用 LangChain 与其交互。

用法

from langchain_community.llms.llamafile import Llamafile

llm = Llamafile()

llm.invoke("Tell me a joke")

API 参考:Llamafile

'? \nI\'ve got a thing for pink, but you know that.\n"Can we not talk about work anymore?" - What did she say?\nI don\'t want to be a burden on you.\nIt\'s hard to keep a good thing going.\nYou can\'t tell me what I want, I have a life too!'

要流式传输令牌，请使用 .stream(...) 方法：

query = "Tell me a joke"

for chunks in llm.stream(query):
    print(chunks, end="")

print()

.
- She said, "I’m tired of my life. What should I do?"
- The man replied, "I hear you. But don’t worry. Life is just like a joke. It has its funny parts too."
- The woman looked at him, amazed and happy to hear his wise words. - "Thank you for your wisdom," she said, smiling. - He replied, "Any time. But it doesn't come easy. You have to laugh and keep moving forward in life."
- She nodded, thanking him again. - The man smiled wryly. "Life can be tough. Sometimes it seems like you’re never going to get out of your situation."
- He said, "I know that. But the key is not giving up. Life has many ups and downs, but in the end, it will turn out okay."
- The woman's eyes softened. "Thank you for your advice. It's so important to keep moving forward in life," she said. - He nodded once again. "You’re welcome. I hope your journey is filled with laughter and joy."
- They both smiled and left the bar, ready to embark on their respective adventures.

有关 LangChain 表达式语言以及 LLM 上可用方法的更多信息，请参阅 LCEL 接口

相关

• 大语言模型 概念指南
• LLM 操操作指南