ObservabilityJS/TS

Getting started

Before you start logging you will have to create a LogRepository on the Maxim dashboard. To create a log repository,click on Logs and the click on + icon on the sidebar. You can use the ID of that repository to push logs.

Initializing SDK

import { Maxim } from "@maximai/maxim-js";
const maxim = new Maxim({ apiKey: "" });
const logger = await maxim.logger({ id: "" });

Tracing single shot logs

Creating a Trace

const trace = logger.trace({
	id: "trace-id", // required
	name: "trace-name", // optional
	tags: { key: "value" }, // optional, you can filter logs based on metadata
});

Tracing multi-turn logs

Creating a Session and adding a Trace to it

const session = logger.session({
	id: "session-id", // required
	name: "session-name", // optional
	tags: { key: "value" }, // optional, you can filter logs based on metadata
});

Once session object is created, you can add multiple Traces across the lifecycle of the conversation.

Adding a Trace to a Session

const session = logger.session({ id: "session-id" });
 
// Here if the trace with the same id is available, it will return the same trace object else create a new one.
const trace = session.trace({ id: "trace-id" });

Using custom ids for Traces/Sessions

Using custom ids allows you to fetch a trace in any function using the same id. This can be useful in updating trace across your workflow.

Trace

Creating a Trace

// ... in function 1
const trace = logger.trace({ id: "trace-id" });
 
// ... in function 2
const trace = logger.trace({ id: "trace-id" });
// it returns the same trace object
trace.addTag({ key: "value" });

Session

Creating a Session and adding a Trace to it

// ... in function 1
const session = logger.session({ id: "session-id" });
 
// ... in function 2
const session = logger.session({ id: "session-id" });
// it returns the same session object
session.addTag({ key: "value" });

Elements of Traces

Once you have a Trace object, you can add

Span

A Span (short for timespan), groups a bunch of items (Events, Retrieval, Generation, Feedback).

Adding a Span to a Trace

const span = trace.span({
	id: "span-id", // optional
	name: "name", // optional
	tags: { key: "value" }, // optional
});
span.event({ id: "event-id"... });
span.end();

Event

An Event is a point in time when something happened in the system.

Adding an Event to a Trace

await trace.event({
	id: "event-id", // optional
	name: "name", // optional
	tags: { key: "value" }, // optional
});

Retrieval

A Retrieval is a special type of Span in Maxim, which represents a retrieval query to a knowledge base or vector database.

Adding a Retrieval to a Trace

const retrieval = trace.retrieval({
	id: "retrieval-id", // optional
	name: "name", // optional
	metadata: { key: "value" }, // optional
});
retrieval.input("How many PTO days do I have?");
retrieval.output(["doc1", "doc2"]);
retrieval.end();

Generation

A Generation is a special type of Span in Maxim, which represents a call to an LLM.

generation.result expects result to be in OpenAI response format. Here is the reference to the OpenAI response format

Adding a Generation to a Trace

const generation = await trace.generation({
	id: "generation-id", // optional
	name: "name", // optional
	maximPromptId: "prompt-id", // optional
	provider: "openai",
	model: "gpt-3.5-turbo-16k",
	messages: [{ role: "system", content: "This is the main prompt" }], // optional
	modelParamters: { key: "value" },
	tags: { key: "value" }, // optional
});
generation.addMessage({
	role: "user",
	content: "This is the main prompt",
	tool_calls: [{}],
});
// We mark generation ended when you call completionResult method
generation.result({
	id: "",
	choices: [
		{
			role: "",
			content: "",
			tool_call: [
				{
					type: "function",
					name: "function-name",
					parameters: [
						{
							key: "value",
						},
					],
				},
			],
		},
	],
	usage: {
		prompt_tokens: 100,
		completion_tokens: 50,
		total_tokens: 150,
	},
});

Feedback

A Feedback is a special type of Event in Maxim, which is a point in time when feedback was given in the system.

Adding a Feedback to a Trace

trace.feedback({
	score: 0.5, // optional (0-1)/(scale)
	feedback: "string feedback", // optional
});

Tool Call

A Tool Call is a special type of Span in Maxim, which represents an external system or service call done based on an LLM response.

Adding a Tool Call to a Trace

const toolCall = completion.choices[0].message.tool_calls[0];
const traceToolCall = trace.toolCall({
	id: toolCall.id,
	name: toolCall.function.name,
	description: "Get current temperature for a given location.",
	args: toolCall.function.arguments,
	tags: { location: toolCall.function.arguments["location"] },
});
 
const result = callExternalService(toolCall.function.name, toolCall.function.arguments);
 
traceToolCall.result(result);

Node-Level Evaluation (Agentic Evaluation)

Maxim supports node-level evaluation, which allows you to evaluate each individual node in your whole trace. This evaluation can be used to measure the quality of each node's output, and can be used to improve the performance of the node.

To evaluate a node, you can use the evaluate method of the node.

Evaluating a node

// for this example we are evaluating a particular generation
// but you can evaluate any node in your trace similarly
 
// ...receive user input and process it
 
generation.evaluate.withEvaluators("clarity", "toxicity").withVariables({
	input: userInput,
});
 
// ...generate llm response
 
generation.evaluate.withVariables({ output: llmResponse.choices[0].message.content }, ["clarity", "toxicity"]);
 
//...code continues

More infomation about Agentic Evaluation can be found in the Agentic Evaluation section under Observability -> Evaluating Logs.

Overview

Langchain

Getting started

On this page