During a design meeting, a marketing video was shown which would suggest to our users that the search bar on our client’s website might be able to somehow answer their questions. For example, one might type in “How do I start a career in accounting?” and the search results would populate with the relevant pages. The problem with this is that our search does not currently work this way. Instead, it’s a rudimentary text search that checks for a direct match between what was typed in the search bar and partial matches in page titles. At first we assumed that the video would have to be changed, but then we thought for a bit and wondered, “what if that could work?”
Testing the capabilities of LLMs
We’ve been pretty into AI here at Alipes, so anytime a seemingly impossible technical idea pops up, we turn to LLMs and test their capabilities. We already know that ChatGPT appears to have impressive evaluation abilities since we built an AI interview simulator with it. Thus, the most obvious implementation would be asking ChatGPT the question that the user posed in the search bar, but coaxing it to return a usable answer in terms of site navigation.
Here was the idea for the prompt: Given the input “[insert search query here]” and the following page titles:
- Article Title 1
- Article Title 2
- Article Title 3
which URL slug sounds like the best match for the input? Answer with only a single option from the provided list of slugs.
By giving it a list of answers to choose from, I can pretty much guarantee that the AI will return a usable answer. Also, because the list is clearly formatted and bulleted, ChatGPT can pick up distinct options well (ironically, this was a prompt crafting suggestion provided to me by Gemini).
Evaluating the results
Okay, time for our very first attempt. I completed the prompt with “Is accounting right for me?” and plugged every page title used by our site into the list.
The result? Within a moment ChatGPT hit me with “What’s so great about accounting?”
Wow, perfect answer on the very first attempt. Admittedly I give it a pretty easy input. Let’s try something a little weirder and see what it gives me.
Let’s try “What’s the best way to grind for a high paying accounting job as a high school student?” Seems trickier, right? It’s got some bad grammar, it’s kind of long, and it’s a vague question.
ChatGPT returned: “Your August Grind Will Help You Shine: Take Your Next Steps Toward Success.” Wow again, I didn’t even know we had an article like that. So far we’re two for two.
Let’s try something that’s not even a question: “Making an accounting resume.” It returned: “College resume tips.” Three out of three.
Taking action
It’s been perfect so far with my limited testing and I’m impressed. Clearly this is going to be the future of web search. So now that I know the AI works, how do I handle the implementation?
I’ll still keep the pattern matching text search aspect, but if zero results are returned I could try adding an “Ask our AI for the right page” option to the results dropdown. When that’s clicked, the user’s query will be inserted into our prompt and passed on to ChatGPT. When ChatGPT answers, we match the chosen page title to a slug and pop the result into the results dropdown. Making the AI option a conscious user choice rather than an automatic lookup will keep the API costs low.
What do I do if ChatGPT fails one in 1000 times and gives me an unusable page title? I could simply keep the result dropdown empty and inform the user that nothing could be found by AI, either. Relying on LLMs is perfect for these low stake value add scenarios.
I still have to make sure there are no cost or legal concerns and put it through additional testing, but all systems are go to take the next step. AI has blown my mind once again.