The first analysis of how Americans are using AI for politics, leveraging Anthropic's Economic Index data. Users get info from Claude, not recommendations; and they seek more of it near elections.
To me the main question is whether analysis/information actually changes preferences or vote choice — ie voters presumably ask about stuff they are not yet very confident about, so the pass-through to preferences might be stronger when voters themselves seek out info (compared to settings where such info is exogenously shown to voters). In that case, if we assume preferences are a function of some beliefs and that function upweighs more certain beliefs, then the informational signal from Claude might actually have nontrivial preference effects.
(however this may not hold under different assumptions)
Yes, this is a great point -- I wonder how we could study it. I can imagine a couple different ways to go about it. One would be an experiment where people are nudged to use AI, and then overall shifts in preferences are measured. The other would be some kind of experiment where people are invited to ask whatever questions they want, and the nature of the output is altered? Not sure just spitballing...
An example would be design where people are given a set of topics, pick one, then are randomized to use either Google (AI summary disabled) or Claude/GPT, then asked about beliefs/preferences on that topic. Though to get at the selection channel directly, this could be combined w/ randomization of assigned vs. self-chosen topics.
Relatedly, the google AI summary might be the more consequential treatment in the aggregate, even if the per-person effect is smaller. People select into using Claude for information acquisition, which makes the set of exposed people smaller. The AI summary exposes people to LLM-generated content "against their will" — at least conditional on the query — so the set of exposed people is much larger.
I think the simplest way to do this would be to give people a subset of ballot items that are easier and harder (e.g., high-profile national partisan choices as the easy ones, nonpartisan judicial races and ballot measures as the hard ones) then randomly grant them access to different tools, track what they do with the tool, and ask them who they choose. That would get at vote choice. It wouldn't tell us much yet about whether the information actually changed preferences. There's a related paper from Elliott Ash and some coauthors, but I think it turned out to not have anywhere near enough power to pick up or rule out interesting effects. https://elliottash.com/papers/Ash-Galletta-Opocher-BallotBot-AI-Strengthen-Democracy.pdf
I use chatbots for downballot elections/ballot questions that are totally off the news cycle for ~99% of people. Like a quick and dirty on county council elections or something like that.
Share of conversations is good, but since a single user might start many conversations on the same topic, it seems to be a poor proxy for number of people using AI for political information.
Does anything in the Anthropic data indicate what share of Claude *users* used AI for political conversations?
Also, I'm not sure how we'd be able to tell if this is representative of typical voters.
I just searched and the numbers I'm getting show OpenAI still has the lion's share of "consumer" users that might best represent the typical voter (or then again, maybe that's not a good proxy either). Gemini is next, and Claude is cited as somewhere between 9% and 21%.
My own experience suggests that Claude tends to be preferred by both more engaged and more aware users. That would suggest that these numbers might be inflated from what typical voters are doing. Without data from OpenAI and/or Google, I think it will be hard to generalize this; even the monthly change is likely to be different for engaged voters.
I've seen some policy proposals (h/t Matt Stoller) for requiring AI vendors to share more information about the uses of their technology, because we're flying blind right now trying to figure out how to respond to the technology without the information we need to respond properly. Something like that would be very helpful here.
Yes you’re absolutely right! I don’t think we know how much of this is repeat-user vs. different users, and it’s definitely specific to the population of people who use Claude.
A good first step would be to repeat this analysis for Gemini, ChatGPT, etc. OpenAI and Google have both setup similar data-sharing programs to Anthropic’s, but neither contains quite the same info about politics, yet. Hopefully they will soon.
And then it will take probably a bit more effort to get the data released in a way that lets us tell apart user-level analyses from content-level. I agree that we may need policies to get this going. There was lots of discussion of that for social media, it never quite went anywhere, but there may be more momentum for AI.
To me the main question is whether analysis/information actually changes preferences or vote choice — ie voters presumably ask about stuff they are not yet very confident about, so the pass-through to preferences might be stronger when voters themselves seek out info (compared to settings where such info is exogenously shown to voters). In that case, if we assume preferences are a function of some beliefs and that function upweighs more certain beliefs, then the informational signal from Claude might actually have nontrivial preference effects.
(however this may not hold under different assumptions)
Yes, this is a great point -- I wonder how we could study it. I can imagine a couple different ways to go about it. One would be an experiment where people are nudged to use AI, and then overall shifts in preferences are measured. The other would be some kind of experiment where people are invited to ask whatever questions they want, and the nature of the output is altered? Not sure just spitballing...
An example would be design where people are given a set of topics, pick one, then are randomized to use either Google (AI summary disabled) or Claude/GPT, then asked about beliefs/preferences on that topic. Though to get at the selection channel directly, this could be combined w/ randomization of assigned vs. self-chosen topics.
Relatedly, the google AI summary might be the more consequential treatment in the aggregate, even if the per-person effect is smaller. People select into using Claude for information acquisition, which makes the set of exposed people smaller. The AI summary exposes people to LLM-generated content "against their will" — at least conditional on the query — so the set of exposed people is much larger.
I think the simplest way to do this would be to give people a subset of ballot items that are easier and harder (e.g., high-profile national partisan choices as the easy ones, nonpartisan judicial races and ballot measures as the hard ones) then randomly grant them access to different tools, track what they do with the tool, and ask them who they choose. That would get at vote choice. It wouldn't tell us much yet about whether the information actually changed preferences. There's a related paper from Elliott Ash and some coauthors, but I think it turned out to not have anywhere near enough power to pick up or rule out interesting effects. https://elliottash.com/papers/Ash-Galletta-Opocher-BallotBot-AI-Strengthen-Democracy.pdf
I use chatbots for downballot elections/ballot questions that are totally off the news cycle for ~99% of people. Like a quick and dirty on county council elections or something like that.
How have you found the answers to be? Useful?
Share of conversations is good, but since a single user might start many conversations on the same topic, it seems to be a poor proxy for number of people using AI for political information.
Does anything in the Anthropic data indicate what share of Claude *users* used AI for political conversations?
Also, I'm not sure how we'd be able to tell if this is representative of typical voters.
I just searched and the numbers I'm getting show OpenAI still has the lion's share of "consumer" users that might best represent the typical voter (or then again, maybe that's not a good proxy either). Gemini is next, and Claude is cited as somewhere between 9% and 21%.
My own experience suggests that Claude tends to be preferred by both more engaged and more aware users. That would suggest that these numbers might be inflated from what typical voters are doing. Without data from OpenAI and/or Google, I think it will be hard to generalize this; even the monthly change is likely to be different for engaged voters.
I've seen some policy proposals (h/t Matt Stoller) for requiring AI vendors to share more information about the uses of their technology, because we're flying blind right now trying to figure out how to respond to the technology without the information we need to respond properly. Something like that would be very helpful here.
Yes you’re absolutely right! I don’t think we know how much of this is repeat-user vs. different users, and it’s definitely specific to the population of people who use Claude.
A good first step would be to repeat this analysis for Gemini, ChatGPT, etc. OpenAI and Google have both setup similar data-sharing programs to Anthropic’s, but neither contains quite the same info about politics, yet. Hopefully they will soon.
And then it will take probably a bit more effort to get the data released in a way that lets us tell apart user-level analyses from content-level. I agree that we may need policies to get this going. There was lots of discussion of that for social media, it never quite went anywhere, but there may be more momentum for AI.