If you ever want to see data represented in a certain way, such as a chart or a graph, to better understand what's going on in the world, you can use AI's to grab the most recent data and then convert it into any visual format you want. This is enabled in AI studio with Gemini through what's called grounding from web search, structured output formatting data, and code execution, which can run Python and generate charts. This lesson is going to walk you through the challenges of using grounding, especially when working with numbers which need to be accurate. How you can use image recognition and convert that to structured output. And then the challenges of using code execution. Especially when AI's are confident that the code ran even though the code failed. At the end, we still get to generating a chart through the online editor and then we take it also to set up in Cursor in a local environment where we can run all the way through YOLO mode and just click around some buttons and get our chart.
Transcript
[00:00] Using GeminiFlash2Experimental, we'll turn on grounding and ask it to find all of the stats of the Lakers roster from yesterday's game against the Warriors. Command-Enter will make this run and we'll get more of a prose-based summary of what was given. So let's ask explicitly for create a markdown table of the stats of the Lakers roster from yesterday's game to at least try and grab us a data set to use. So this did fail pretty miserably and I just wanted to show that since I have been trying all morning to give variations of prompts to get proper data back from current events. For example, we could start with, please create a markdown table of the stats of the Lakers roster from yesterday's game against the Warriors.
[00:39] Make sure grounding is turned on, run it. And this has worked a little bit before. For example, these LeBron stats look correct, but Torian Prince is not on the team, Christian Wood is hurt, Anthony Davis did not score, so do not rely on this data. The way I've had the most success is to create a prompt, go and take a screenshot that already has the data, so it just has to OCR what's going on. Drop that in here and let's use the structured output.
[01:02] We'll say extract the stats from this image, command enter to run it, and now we have an accurate data set of everything from that image. And now that we have proper data we can turn off the structured output, turn on code execution, and ask it using Python, please use a chart library to create a radar chart with each player, their points, their rebounds, and their assists. We'll let this run and it looks like the code execution failed here. And often when working with AIs when I run into an error like this I'll copy the entire conversation and I kind of think of this as a branching point where I take everything and I go start another chat to essentially solve what's going on in that branch. So I'll paste that in and ask it please find why this code errored out, then let this run, and then once this has solved it I'll bring it back into the other chat rather than asking the main chat to go down a different thinking path.
[01:55] So I'm just going to copy this, the key changes, and the explanation, scroll back down, paste this in. Please use the changes above to fix the code execution error, then allow this to run. And something else might be off. I'll just try it one more time. Copy everything, paste it.
[02:11] I'm still getting an error even with your suggested changes. And we can even try in this essentially chain of thought, let's just run it from here. Please generate the radar chart. All right so the image popped up. We'll open this in a new tab and you can see our chart displayed.
[02:27] I really like how a triple double kind of stands out. Where you can see Austin Reeves had 10 rebounds, 10 assists, and over 10 points. So let's try again from here, just take the proper solution this time, try and bring it back to our main chat, paste this in. I found these errors too, please correct it and generate the radar chart. At this point I'm not sure what's going on without just trying to rerun it, but without digging into the code and debugging.
[02:50] Again this is kind of why I like branching conversations, because AIs definitely get very confident even when they're wrong. So just out of curiosity I'm going to copy and paste this over into our other one. I also tried this code but this code failed. What's the difference between this code I tried and the code that you provided that worked? And this is calling out there was only a difference in variable names, so I'm going to assume that something is either going on with the API and code execution is just not working at the moment.
[03:15] Now just out of sheer curiosity, I just want to select the whole thing. And I've yet to find a way of selecting or exporting the entire conversation. These little copy blocks are limited, so without going into Chrome DevTools and looping through DOM elements, I just took the lazy way. And I'm going to open Cursor and try our YOLO mode. Where I go to Composer, I paste in everything, I turn on the agent, and give it vague instructions of, I was trying to accomplish this task in an online editor but it kept failing, please set this up locally and run it, I really need this radar chart.
[03:45] So hit submit and let this go to town see what happens. It's installing the libraries, adding the Python script, setting up a virtual environment. Let me zoom in a little bit here. Seems like this command is hanging. I'm just getting a blinking dot here and no other progress indicators.
[03:59] So I'm going to cancel and accept all and just ask it to please continue. I'm not sure if the output or a log of this is available somewhere. It's prompting me to run the command so I'll allow it to this time and I'm not sure if that's because it disabled the YOLO mode or if it switched to more of an interactive mode because I canceled. So now let's run this command. So yeah it looks like YOLO mode is disabled so I'll be accepting commands from here on out.
[04:24] We'll run this and I'm a bit confused because now it's pulsing like it's doing work but there's also a run command. I'm gonna click run command. Maybe I'm just confused by the UX because I would expect this button to be blue if it wanted me to do it. So I'll click run command. Maybe that was it.
[04:40] We'll click run command again. Looks like we have our PNG. Let's check it out. All right, looks like we're back and business here.