Tutorial: Use custom charts

Tutorial of using the custom charts feature in the W&B UI

Use custom charts to control the data you’re loading in to a panel and its visualization.

1. Log data to W&B

First, log data in your script. Use wandb.Run.config for single points set at the beginning of training, like hyperparameters. Use wandb.Run.log() for multiple points over time, and log custom 2D arrays with wandb.Table(). We recommend logging up to 10,000 data points per logged key.

with wandb.init() as run: 

  # Logging a custom table of data
  my_custom_data = [[x1, y1, z1], [x2, y2, z2]]
  run.log(
    {"custom_data_table": wandb.Table(data=my_custom_data, columns=["x", "y", "z"])}
  )

Try a quick example notebook to log the data tables, and in the next step we’ll set up custom charts. See what the resulting charts look like in the live report.

2. Create a query

Once you’ve logged data to visualize, go to your project page and click the + button to add a new panel, then select Custom Chart. You can follow along in the custom charts demo workspace.

Add a query

1. Click summary and select historyTable to set up a new query pulling data from the run history.
2. Type in the key where you logged the wandb.Table(). In the code snippet above, it was my_custom_table . In the example notebook, the keys are pr_curve and roc_curve.

Set Vega fields

Now that the query is loading in these columns, they’re available as options to select in the Vega fields dropdown menus:

• x-axis: runSets_historyTable_r (recall)
• y-axis: runSets_historyTable_p (precision)
• color: runSets_historyTable_c (class label)

3. Customize the chart

Now that looks pretty good, but I’d like to switch from a scatter plot to a line plot. Click Edit to change the Vega spec for this built in chart. Follow along in the custom charts demo workspace.

I updated the Vega spec to customize the visualization:

• add titles for the plot, legend, x-axis, and y-axis (set “title” for each field)
• change the value of “mark” from “point” to “line”
• remove the unused “size” field

To save this as a preset that you can use elsewhere in this project, click Save as at the top of the page. Here’s what the result looks like, along with an ROC curve:

Bonus: Composite Histograms

Histograms can visualize numerical distributions to help us understand larger datasets. Composite histograms show multiple distributions across the same bins, letting us compare two or more metrics across different models or across different classes within our model. For a semantic segmentation model detecting objects in driving scenes, we might compare the effectiveness of optimizing for accuracy versus intersection over union (IOU), or we might want to know how well different models detect cars (large, common regions in the data) versus traffic signs (much smaller, less common regions). In the demo Colab, you can compare the confidence scores for two of the ten classes of living things.

To create your own version of the custom composite histogram panel:

1. Create a new Custom Chart panel in your Workspace or Report (by adding a “Custom Chart” visualization). Hit the “Edit” button in the top right to modify the Vega spec starting from any built-in panel type.
2. Replace that built-in Vega spec with my MVP code for a composite histogram in Vega. You can modify the main title, axis titles, input domain, and any other details directly in this Vega spec using Vega syntax (you could change the colors or even add a third histogram :)
3. Modify the query in the right hand side to load the correct data from your wandb logs. Add the field summaryTable and set the corresponding tableKey to class_scores to fetch the wandb.Table logged by your run. This will let you populate the two histogram bin sets (red_bins and blue_bins) via the dropdown menus with the columns of the wandb.Table logged as class_scores. For my example, I chose the animal class prediction scores for the red bins and plant for the blue bins.
4. You can keep making changes to the Vega spec and query until you’re happy with the plot you see in the preview rendering. Once you’re done, click Save as in the top and give your custom plot a name so you can reuse it. Then click Apply from panel library to finish your plot.

Here’s what my results look like from a very brief experiment: training on only 1000 examples for one epoch yields a model that’s very confident that most images are not plants and very uncertain about which images might be animals.