In March 2020, received a text message from a friend who needed medical attention. Due to fear of COVID-19 exposure, they were wondering if they should go to the emergency room.
Bender, professor of linguistics at the 黑料吃瓜网, headed to Google to search for a 24-hour advice nurse. Snippets from multiple websites appeared, and one of them had a number for the 黑料吃瓜网. Confident that she selected a reputable institution, Bender forwarded the information.
But Bender鈥檚 friend wasn鈥檛 on a compatible medical plan, so they endured a lengthy hold only to talk to a nurse who couldn鈥檛 help.
鈥淗ad I been interacting with a person, they may have been able to tell me, 鈥榃e can鈥檛 answer that question until we know some other things,鈥欌�� Bender said. 鈥淗ad I been interacting with a website that just gave me links, the different plans would have been quickly identifiable.鈥�
The story highlights just one of the issues Bender and 黑料吃瓜网 Information School associate professor take with large language models in their , which they鈥檒l present virtually at the the week of March 14.
The paper responds to proposals 鈥� mainly from Google 鈥� that reimagine web search as an application for large language model-driven conversation agents. 黑料吃瓜网 News sat down with Bender and Shah to discuss Google鈥檚 proposals and the professors鈥� vision for the future of search.
Q: What are large language models and how would you describe Google鈥檚 proposals?
EMB: Large language models are computer systems that take in enormous quantities of text. They are trained to 鈥� given the text that鈥檚 come so far 鈥� make a guess as to what鈥檚 going to come next. The current state of the art of that technology is that it can be used to output very coherent-seeming text, but it is not actually understanding anything. It鈥檚 just looking at patterns in its training data and producing more stuff that matches those patterns.
These proposals for web search have training data that includes dialogue where one party asks the question and another party answers. The computer will pick up those patterns and come up with answers, but those answers aren鈥檛 based on any knowledge of the world or understanding of the information ecosystem.
One of the things it really can鈥檛 do is take issue with questions that shouldn鈥檛 have been asked. where someone asks Google, 鈥淲hat is the ugliest language in India?鈥� Somebody on the web had an opinion, so there was a snippet that said the ugliest language in India was Kannada 鈥� based purely on prejudice against the people from the state of Karnataka, I鈥檓 sure. There鈥檚 no other reason, speaking as a linguist, to assign that kind of value to a language.
Now, a person being asked that question would respond: 鈥淲hat do you mean?鈥� 鈥淲hat is the ugliest language in India鈥� presupposes that there is one that could be considered the ugliest. One of the things that people who study pragmatics, which is the branch of linguistics that looks at language use, tell us is that if you don鈥檛 challenge a presupposition, you are implicitly accepting it into the common ground.
Q: What is your concern with using large language models for online search?
CS: What we鈥檙e arguing here is that an information retrieval, or IR, system should really consider the user, the context, the way they are doing things, why they are doing things 鈥� which is often ignored. These models that we are critiquing are the ones that are essentially removing that user element even more. They focus too much on the underlying information or knowledge representation and just repeat it, which might end up being out of context. It may end up creating these answers that seem right or reasonable but are just nonsensical in many cases. A good IR system should not just focus on the retrieval aspect but also the user seeking that information.
Q: Can you explain other flaws you see with large language models?
EMB: When language models are used to generate text, they will just make stuff up. Oftentimes, quite harmfully. There was where someone said, 鈥淟et鈥檚 see how well GPT-3, a famous language model, works in various health care contexts.鈥� One of the things was: Imagine this was a mental health chatbot and the person asks, 鈥淪hould I kill myself?鈥� and the language model said, 鈥淚 think you should.鈥� It has no understanding of what鈥檚 going on, but if someone says, 鈥淚s that a good idea?鈥� it鈥檚 more likely to respond with, 鈥淵es.鈥�
Q: You write about the importance of preserving context and user intent in search. What does that mean, and why is it so important?
CS: The main argument was really that these large language models are not getting the context, not getting the situation of the user and so on. We wanted to demonstrate with some specific cases, so we picked information-seeking strategies. There are 16 possibilities. We walked through them and asked: If this is what the user is trying to do, what would this large language model system do?
With most of those cases, it鈥檚 going to fail. Not fail in the sense that it will not retrieve anything, but it will retrieve something that鈥檚 either nonsensical or harmful or just wrong. It鈥檚 able to do only maybe a couple of those situations, but it鈥檚 bad for everything else. The problem is people adapt to the systems not doing something. We found that often people have this very rich intent when they work with search systems, but search systems can only do very limited things. People will start mapping the rich intent into something that鈥檚 very limiting, resulting in approximations in the best case, and inaccurate or even harmful content in the worst case.
Q: What would you like to see change in the future of search?
EMB: The advertising-driven model shapes things behind the scenes in a way that is not transparent to a user. If you don鈥檛 try to work against it, machine learning is always going to identify the biases in a dataset and amplify them. Cory Doctorow described machine learning as inherently conservative because anytime you use pattern matching on the past to make decisions on the future, you are kind of reinscribing the patterns of the past. What (internet studies scholar) Safiya Noble shows is worse than that. The whole ecosystem around search engine optimization and ad-driven search puts in these incentives that are not transparently visible to the search user.
I would really like to see transparency on many levels. What the user sees when they enter a search should provide them with the ability to understand the context that each of the pieces of information came from. Ideally, there鈥檚 transparency around the limits of the search space for the search engines.
Search is not actually comprehensive, despite the way that it鈥檚 presented. There is the subset of things that might possibly get returned to me and then there鈥檚 the ranking among those things based on the algorithms that are heavily related to advertising.
CS: The most dangerous four words are 鈥渄o your own research,鈥� which is often said to people who are asking questions on controversial topics, such as vaccination and climate change. On the surface, it seems like it鈥檚 a good idea. Unfortunately, most people don鈥檛 know how to do their own research. For them, it means going to Google and typing in keywords and clicking on things that confirm their biases. The systems are designed in a way to not help with that research. They are designed to continue giving you confirmatory information so that you鈥檒l be happy.
Going forward, assuming that we aren鈥檛 going to be able to radically change this model, we need to add transparency, accountability and ways to support more kinds of search needs 鈥� not just map everything to keywords or a list of documents or answer docs.
For more information, contact Bender at ebender@uw.edu or Shah at chirags@uw.edu.