ObservabilityPython

Manual integration

In the previous section, we explored the quick integration of Maxim observability using decorators. However, there are scenarios where you might need finer control over your code's tracing. This section will guide you through the process of manually integrating Maxim observability into your codebase, providing you with more flexibility and granular control over your observability implementation.

Tracing one shot logs

Creating a trace

from maxim.logger import TraceConfig
 
trace = logger.trace(TraceConfig(id="trace-id",name="trace-name",tags={"key":"value"}))

Tracing multi-turn logs

Creating a session and adding a trace to it

from maxim.logger import SessionConfig, TraceConfig
 
session = logger.session(SessionConfig(id="session-id",name="session-name",tags={"key":"value"}))
trace = session.trace(TraceConfig(id="trace-id"))

Using custom ids for Traces/Sessions

Using custom ids allows you to fetch a trace in any function using the same id. This can be useful in updating trace across your workflow.

Trace

Using custom ids for Traces

from maxim.logger import TraceConfig
# ... in function 1
trace = logger.trace(TraceConfig(id="trace-id"))
# ... in function 2
trace = logger.trace(TraceConfig(id="trace-id"))
# it returns the same trace object
trace.add_tag({"key":"value"})

Session

Using custom ids for Sessions

from maxim.logger import SessionConfig
# ... in function 1
session = logger.session(SessionConfig(id="session-id"))
# ... in function 2
session = logger.trace(SessionConfig(id="session-id"))
# it returns the same session object
session.add_tag({"key":"value"})

Elements of Traces

Once you have a Trace object, you can add

Span

A Span (short for timespan), groups a bunch of items (Events, Retrieval, Generation, Feedback).

Adding a Span to a Trace

from maxim.logger import SpanConfig
 
span = trace.span(SpanConfig(id=str(uuid4()), name="Test Span"))
span.event("event-id","event-name")
span.end()

Event

An event is a point in time when something happened in the system.

Adding an Event to a Trace

trace.event("event-id", "event name", {})

Retrieval

A Retrieval is a special type of Span in Maxim, which represents a retrieval query to a knowledge base or vector database.

Adding a retrieval to a span

from maxim.logger import RetrievalConfig
 
retrieval = span.retrieval(RetrievalConfig(id="retrieval-id", name="Test Retrieval"))
retrieval.input("How many PTO days do I have?")
retrieval.output(["doc1", "doc2"])
retrieval.end()

Generation

A Generation is a special type of Span in Maxim, which represents a call to an LLM.

generation.result expects result to be in OpenAI response format. Here is the reference to the OpenAI response format

Adding a Generation to a Span

from maxim.logger import GenerationConfig
 
generationConfig = GenerationConfig(
					id=str(uuid4()),
					name="gen1",
					provider="openai",
					model="gpt-3.5-turbo-16k",
					model_parameters={"temperature": 3},
					messages=[{
						"role": "user",
						"content": "Hello, how can I help you today?"
					}])
generation = trace.generation(generationConfig)
generation.result({
	"id": "cmpl-uN1k3lnZkTlZg8GHt4Vtd1aB",
	"object": "text_completion",
	"created": 1718393286,
	"model": "gpt-3.5-turbo-16k",
	"choices": [
		{
			"index": 0,
			"text": "\n Here is the response\n",
			"logprobs": None,
			"finish_reason": "stop",
		},
	],
	"usage": {
		"prompt_tokens": 7,
		"completion_tokens": 105,
		"total_tokens": 112,
	},
})

Feedback

A Feedback is a special type of event in Maxim, which is a point in time when Feedback was given in the system.

Adding a Feedback to a Trace

from maxim.logger import Feedback
 
trace.feedback(feedback=Feedback(score=5, comment="Great job!"))

Tool Call

A Tool Call is a special type of Span in Maxim, which represents an external system or service call done based on an LLM response.

Adding a Tool Call to a Trace

from maxim.logger import ToolCallConfig
 
tool_call = completion.choices[0].message.tool_calls[0]
tool_call_config = ToolCallConfig(
    id=tool_call.id,
    name=tool_call.function.name,
    description="Get current temperature for a given location.",
    args=tool_call.function.arguments,
    tags={ "location": toolCall.function.arguments["location"] }
)
trace_tool_call = trace.tool_call(tool_call_config)
 
result = call_external_service(tool_call.function.name, tool_call.function.arguments)
 
trace_tool_call.result(result)

Node-Level Evaluation (Agentic Evaluation)

Maxim supports node-level evaluation, which allows you to evaluate each individual node in your whole trace. This evaluation can be used to measure the quality of each node's output, and can be used to improve the performance of the node.

To evaluate a node, you can use the evaluate method of the node.

Evaluating a node

# for this example we are evaluating a particular generation
# but you can evaluate any node in your trace similarly
 
# ...receive user input and process it
 
generation.evaluate()
    .with_evaluators("clarity", "toxicity")
    .with_variables({
        "input": user_input
    })
 
# ...generate llm response
 
generation.evaluate().with_variables(
    { "output": llm_response.choices[0].message.content },
    ["clarity", "toxicity"]
)
 
# ...code continues

More infomation about Agentic Evaluation can be found in the Agentic Evaluation section under Observability -> Evaluating Logs.

Tracing Litellm with Maxim SDK

Changelog

Manual integration

On this page