Aside: Stop Sequence

[00:00] Let's take a look at the stop sequence, a way of stopping the model from generating tokens. So, what exactly is it and how does it work? Well, it's an optional setting that tells the API to stop generating tokens at a given point. The completion will never contain the stop sequence.

[00:18] And the stop sequence is an array, and that's because we can have multiple stops in a stop sequence. Now, all of that is quite theoretical, so let's just jump straight into an example and you'll see what I mean. Right here, I've got a basic API request set up using the TextDaVinci 003 model,

[00:37] and I'm just asking it to list some great books to read on the topic of coding. So, I will uncomment this function call and hit save, and let's open up the console and see what we get. And there we are, we get 10 good book recommendations.

[00:52] Now, I'm going to come in here underneath the max tokens and I'm going to add a stop sequence, and the property that we want is just stop. The stop sequence is an array, and each individual stop will be a string. Now, we can have up to four stops in this array, I'm only going to use one.

[01:10] And if we look at the formatting of what's in the console, each list item starts with a number and a dot. So, it's one dot. Code, the hidden language of computer hardware and software by Charles Petzold. Two dot. Python crash course by Eric Maths.

[01:26] So, the stop that I put in here just to experiment with is going to be two dot. Let's see what happens when I press save. We get a list with only one item. So, what happened here? Well, if you'll remember, the completion will not contain the stop sequence,

[01:43] so the model was prevented from writing two dot. And because it won't write a stop sequence, at that point, the completion is cut off. So, if we wanted a list of five books, we could change this to six dot. And there we are, we have got a list of five books. Now, you can use anything as a stop.

[02:03] You can put whatever characters in there you might find in your completion, but you just need to know what you're doing with it. Obviously, if you put something like the letter A, well, that's going to cause all sorts of problems. There we are, the completion has actually been cut off on the fourth letter of the book title, which was obviously an A.

[02:21] Now, when we're working with chatbots, it's very common to use the newline value as a stop sequence. Why? Because often, you want the bot to give you one paragraph as an answer, and then you want the user to respond.

[02:36] If you don't use a stop sequence with a chatbot, you run the risk of the chatbot answering and asking questions and having a conversation with itself that becomes ever more bizarre and illogical until it hits the token limit and that stops it.

[02:50] Now, with the general purpose chatbot we built with GPT-4, we didn't need to add a stop property because GPT-4 has actually got that specific syntax, the object with the role and the content properties. DaVinci and other models don't have that, so we can use the newline character as a stop

[03:09] to stop the bot from continuing the conversation on its own. Okay, let's go back to the app and add a stop property.