iiinova — where your digital self keeps evolving

Interactive Data Lab

@idl.uw.edu

Visualization & data analysis research at the University of Washington. In a prior life was the Stanford Vis Group. idl.uw.edu

Posts Replies Media

Interactive Data Lab @idl.uw.edu · 26/11/2024

We applied DracoGPT using experimental data from Younghoon Kim et al. (EuroVis'18) that spans a variety of plots, data distributions, and tasks (comparing individual values vs. aggregate properties). The plots show chart "scores" from two GPT-4 Turbo prompting methods vs. human performance data.

Chart costs conditioned on task type, comparing Kim et al.’s experimental results to DracoGPT-Rank (A) and DracoGPT-Recommend Vega-Lite (B). For value tasks, results from both GPT4-Turbo models moderately correlate with a model fit to human performance data, but do not significantly correlate for summary tasks.

100

Interactive Data Lab @idl.uw.edu · 26/11/2024

People are now using LLMs to create charts and graphs. How might we assess the quality and consistency of the results? DracoGPT is a method that fits a visualization knowledge base (Draco) to LLM responses, enabling comparison across models, prompts, and results from human subjects experiments.

Overview of the DracoGPT-Rank pipeline. (1) User provides prompt templates for an LLM to rank chart pairs; (2) Draco featurizes charts and produces feature vectors consisting of constraint counts; (3) Draco learns constraint weights over LLM-labeled chart pairs by fitting a RankSVM model; (4) The fitted Draco model can be applied to score charts. Results at each stage of the pipeline afford insight into LLM ranking preferences.

1192