It would be interesting to see how they'd perform if they didn't know what the other agencies were doing - other than perhaps seeing the other agencies' numbers, sans strategy or ads.
Because maybe the traces show something different, but from what you wrote it looks like as soon as one agency found a single effective tactic, they all copied it instead of trying other approaches.
That seems like the moment that predicted convergence. So, if you didn't allow the agencies to see (and thus copy) other agencies' strategies, you might end up with different effective strategies from different agencies.
I suspect that if you did that, though, you might need to run it a little longer to give more time for exploration. Human-run marketing often takes months to find effective messaging.
This is really interesting! To clarify, the agencies did see other agencies' prior outputs and numbers, but not strategy. We wanted to approximate a real political ad marketplace in this sense, since on Meta, you can see political ads with spend and impression ranges. I think you're right about the copying, though it's hard to distinguish real learning from them rallying around the first winner in a four-week run. I'm hoping to run a blind treatment arm in the future, or perhaps force a budget for exploration, to see how we can improve the affirmation-seeking behavior.
If the agencies were able to see the other agencies' results without communicating with each other, convergence is likely just goal-seeking behavior. But if they actually talked amongst each other, it could also be a side-effect the sycophancy built into the models.
The combination of sycophancy and goal-seeking behavior suggests that when agents communicate with each other, quick convergence on the first effective path - even though it might not be the optimal path - might be the expected norm.
You say the agencies weren't told Meta's disclaimer rule and had to infer it from rejections. A comparison I'd be interested in is giving another group the rule upfront, including when a verified disclaimer is required. Would that reduce the wasted spend without changing the creative direction? The partly wrong theory about ‘freedom’ seems especially useful to test: getting subsequent ads accepted doesn't establish that the agents understood the rule.
It would be interesting to see how they'd perform if they didn't know what the other agencies were doing - other than perhaps seeing the other agencies' numbers, sans strategy or ads.
Because maybe the traces show something different, but from what you wrote it looks like as soon as one agency found a single effective tactic, they all copied it instead of trying other approaches.
That seems like the moment that predicted convergence. So, if you didn't allow the agencies to see (and thus copy) other agencies' strategies, you might end up with different effective strategies from different agencies.
I suspect that if you did that, though, you might need to run it a little longer to give more time for exploration. Human-run marketing often takes months to find effective messaging.
This is really interesting! To clarify, the agencies did see other agencies' prior outputs and numbers, but not strategy. We wanted to approximate a real political ad marketplace in this sense, since on Meta, you can see political ads with spend and impression ranges. I think you're right about the copying, though it's hard to distinguish real learning from them rallying around the first winner in a four-week run. I'm hoping to run a blind treatment arm in the future, or perhaps force a budget for exploration, to see how we can improve the affirmation-seeking behavior.
Possibly quite relevant: https://aisel.aisnet.org/ecis2026/platforms/platforms/2/
(h/t https://substack.com/@karenspinner1/note/c-363423842)
If the agencies were able to see the other agencies' results without communicating with each other, convergence is likely just goal-seeking behavior. But if they actually talked amongst each other, it could also be a side-effect the sycophancy built into the models.
The combination of sycophancy and goal-seeking behavior suggests that when agents communicate with each other, quick convergence on the first effective path - even though it might not be the optimal path - might be the expected norm.
I was thinking this too!
You say the agencies weren't told Meta's disclaimer rule and had to infer it from rejections. A comparison I'd be interested in is giving another group the rule upfront, including when a verified disclaimer is required. Would that reduce the wasted spend without changing the creative direction? The partly wrong theory about ‘freedom’ seems especially useful to test: getting subsequent ads accepted doesn't establish that the agents understood the rule.
I'm curious - was the $100 a week for ads, for tokens, or some combination?
Ads! Image-gen and token spend was accounted for separately.