This is an interactive notebook. You can run it locally or use the links below:
Structured Outputs for Multi-Agent Systems
OpenAI relased Structured Outputs to enable users to ensure the model will always generate responses that adhere to your supplied JSON Schema without strongly worded prompts. With Structured Outputs, we don’t need to validate or retry incorrectly formatted responses. By using the new parameterstrict: true
, we are able to guarantee the response abides by a provided schema.
The use of structured outputs in a multi-agent system enhances communication by ensuring consistent, easily processed data between agents. It also improves safety by allowing explicit refusals and boosts performance by eliminating the need for retries or validations. This simplifies interactions and increases overall system efficiency.
This tutorial demonstrates how we can utilize structured outputs in multi-agent system and trace them with Weave.
Source: This cookbook is based on sample code from OpenAI’s structured outputs, with some modifications added for improved visualization using Weave.
Installing the Dependencies
We need the following libraries for this tutorial:- OpenAI to create multi-agent system.
- Weave to track our LLM workflow and evaluate our prompting strategies.
WANDB_API_KEY
in our env so that we may easily login with wandb.login() (this should be given to the colab as a secret).
We set the project in W&B we want to log this into in name_of_wandb_project
.
NOTE: name_of_wandb_project
may also be in the format of {team_name}/{project_name}
to specify a team to log the traces into.
We then fetch a weave client by calling weave.init()
Since we’ll be using OpenAI API, we will also need an OpenAI API key. You can sign up on the OpenAI platform to get your own API key. (this should be given to the colab as a secret too.)
Agents set up
The use case we will tackle is a data analysis task. Let’s first set up our 4-agents system:- Triaging agent: Decides which agent(s) to call
- Data pre-processing Agent: Prepares data for analysis - for example by cleaning it up
- Data Analysis Agent: Performs analysis on the data
- Data Visualization Agent: Visualizes the output of the analysis to extract insights We will start by defining the system prompts for each of these agents.
Enable tracking of multi-agent using Weave
We need to write the code logic to:- handle passing the user query to the multi-agent system
- handle the internal workings of the multi-agent system
- execute the tool calls
clean_data
, start_analysis
and use_line_chart
.
We will begin by defining the execution function responsible for running tool calls.
By decorating Python functions with @weave.op()
, we can log and debug language model inputs, outputs, and traces.
When creating a multi-agent system, many functions will appear, but it’s sufficient to simply add @weave.op()
on top of them.
Execute multi-agent systems and visualization in Weave
Finally, we execute the primaryhandle_user_message
function using the user’s input and observe the results.

analysis_agent
, we can see that it is in a structured output format. OpenAI’s structured output facilitates collaboration between agents, but as the system becomes more complex, it becomes harder to grasp the format in which these interactions are taking place. Using Weave allows us to understand these intermediate processes and their inputs and outputs as if we were holding them in your hand.
