LLMonitor
LLMonitor 是一个开源可观测性平台，提供成本和使用情况分析、用户跟踪、跟踪和评估工具。
设置
在 llmonitor.com 上创建一个帐户，然后复制新应用的tracking id.
拥有它后，通过运行以下命令将其设置为环境变量：
export LLMONITOR_APP_ID="..."
如果您不想设置环境变量，则可以在初始化回调处理程序时直接传递 key：
from langchain_community.callbacks.llmonitor_callback import LLMonitorCallbackHandler
handler = LLMonitorCallbackHandler(app_id="...")
API 参考：LLMonitorCallbackHandler
与 LLM/Chat 模型的使用
from langchain_openai import OpenAI
from langchain_openai import ChatOpenAI
handler = LLMonitorCallbackHandler()
llm = OpenAI(
callbacks=[handler],
)
chat = ChatOpenAI(callbacks=[handler])
llm("Tell me a joke")
与链和代理一起使用
确保将回调处理程序传递给run方法，以便正确跟踪所有相关的链和 LLM 调用。
也建议通过agent_name在元数据中，以便能够区分仪表板中的代理。
例：
from langchain_openai import ChatOpenAI
from langchain_community.callbacks.llmonitor_callback import LLMonitorCallbackHandler
from langchain_core.messages import SystemMessage, HumanMessage
from langchain.agents import OpenAIFunctionsAgent, AgentExecutor, tool
llm = ChatOpenAI(temperature=0)
handler = LLMonitorCallbackHandler()
@tool
def get_word_length(word: str) -> int:
"""Returns the length of a word."""
return len(word)
tools = [get_word_length]
prompt = OpenAIFunctionsAgent.create_prompt(
system_message=SystemMessage(
content="You are very powerful assistant, but bad at calculating lengths of words."
)
)
agent = OpenAIFunctionsAgent(llm=llm, tools=tools, prompt=prompt, verbose=True)
agent_executor = AgentExecutor(
agent=agent, tools=tools, verbose=True, metadata={"agent_name": "WordCount"} # <- recommended, assign a custom name
)
agent_executor.run("how many letters in the word educa?", callbacks=[handler])
API 参考：ChatOpenAI | LLMonitorCallbackHandler | 系统消息 | HumanMessage （人工消息） | OpenAIFunctionsAgent | 代理执行程序 | 工具
另一个例子：
from langchain.agents import load_tools, initialize_agent, AgentType
from langchain_openai import OpenAI
from langchain_community.callbacks.llmonitor_callback import LLMonitorCallbackHandler
handler = LLMonitorCallbackHandler()
llm = OpenAI(temperature=0)
tools = load_tools(["serpapi", "llm-math"], llm=llm)
agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, metadata={ "agent_name": "GirlfriendAgeFinder" }) # <- recommended, assign a custom name
agent.run(
"Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?",
callbacks=[handler],
)
用户跟踪
用户跟踪允许您识别您的用户、跟踪他们的成本、对话等。
from langchain_community.callbacks.llmonitor_callback import LLMonitorCallbackHandler, identify
with identify("user-123"):
llm.invoke("Tell me a joke")
with identify("user-456", user_props={"email": "user456@test.com"}):
agent.run("Who is Leo DiCaprio's girlfriend?")
API 参考：LLMonitorCallbackHandler | 识别