Can AI Fix Bad Rules?

We built an AI that reads contracts and successfully predicts which ones will end up disputed, just from the clarity of the text. A small first step towards using AI to write better rules everywhere.

“Whence does it come to pass that our common language, so easy for all other uses, becomes obscure and unintelligible in wills and contracts? and that he who so clearly expresses himself in whatever else he speaks or writes, cannot find in these any way of declaring himself that does not fall into doubt and contradiction?”

–Michel de Montaigne, “Of Experience” (1587)

In 1863, William Winter Raffles, a Liverpool cotton merchant, agreed to sell 125 bales of Surat cotton to the Liverpool trading firm of Daniel Wichelhaus and Gustav Busch. The written contract specified that the cotton would “arrive ex Peerless from Bombay.”

But there was a problem. Unbeknownst to both parties—but beknownst to posterity and to many 1L Contract Law students—there were two different ships named Peerless sailing from Bombay to Liverpool that season!

One of the ships called Peerless

Raffles intended to sell the cotton arriving on the Peerless that departed Bombay in December, while Wichelhaus and Busch meant to receive their order from the Peerless that had departed in October. When Raffles prepared to deliver cotton from the December ship, Wichelhaus and Busch refused to accept it, arguing that they had contracted for the October shipment.

Raffles sued for breach of contract, but the Court of Exchequer held that the contract failed because the phrase “ex Peerless” was latently ambiguous: each side had attached a different, equally reasonable meaning to the same words, so there had never been a true agreement. If only the parties had written the contract to uniquely identify the ship—by specifying its departure date, or using the unique identifier each ship was assigned—the whole dispute could have been avoided, as A.W. Brian Simpsons explains in his magnum opus on the history of the case.

Unanticipated problems of wording, like this one, happen all the time. Just recently, ambiguous wording in America’s “Memorandum of Understanding” with Iran threatened to break a fragile cease fire. When I was an advisor at Meta, the issue cropped up constantly—teams spent enormous effort to write the rules that governed content moderation, only to then face pieces of public content that intentionally or accidentally exposed loopholes or gaps. From local and national politics to private contracts and online platform policies, the issue of writing clear and unambiguous rules is a perennial challenge to all forms of governance.

I’ve been working on what I call political superintelligence—the idea that, as AI makes intelligence more cheaply available, we can use it to make our democracy function better. Could AI help us to avoid some of the ambiguities in important written agreements? Writing better agreements won’t solve all of the challenges of governance—even perfectly worded contracts won’t anticipate major future shifts, obviously, or solve the challenges of imperfect enforcement—but it could certainly improve the way we govern our societies, our private business relationships, our online platforms, and much more.

We decided to put this to the test. To start, we needed somewhere we could quickly harvest lots of written contracts and observe which ones ended up in disputes. We didn’t have to look far: prediction markets are a favorite testbed here at Free Systems, and Kalshi and Polymarket publish resolution rules explaining how each contract will be decided. We’ve previously explored the very problem that those rules can be ambiguous, so it presents a perfect opportunity for us to revisit and see if we can do better.

Working with Claude, we developed a 10-point rubric for spotting flaws in contract rules that might lead to disputes—including things like failing to specify a source of truth, not naming an entity specifically enough (like the Peerless!), and so on. Then, we used those 10 scores for each resolution rule to train a model to predict which ones would actually be contested, in practice, using data on actual disputes from both platforms.

It worked surprisingly well! Just from our grades based on the text of the resolution rules, our tool is able to predict which contracts are more likely to result in a dispute at a rate far better than chance.

In the realm of prediction markets, our scores can hopefully be the start of a “grading system” for contracts. And they can also be used to generate specific recommendations on how to design better rules for future contracts.

More broadly, the results point to an exciting way we can use AI to improve governance. Could we unleash powerful AI models to help us improve all kinds of governance and contract language—asking it to find ambiguities, unanticipated issues, miswordings, and more before we release the language into the wild? If done well, the result will be clearer contracts and better outcomes. A small but valuable step in building political superintelligence for the world.

Here’s how we did our study and more about what we learned.

Teaching AI to evaluate contract language

First, the data: we collected a sample of 10,000 contract resolution rules from Kalshi and Polymarket, combined with information on which contracts ultimately wound up disputed. For Polymarket, we turned to UMA’s governance oracle. This is the process by which Polymarket receives and handles disputes, and all the data is openly available online. We divided these disputes into two categories: material disputes, which are those that weren’t dismissed for being premature (a technicality in how UMA works), and confirmed disputes, which are the subset of material disputes that continue through the dispute process and receive a second dispute request (again, a technicality of the way UMA works).

Because Kalshi’s “disputes” are more ambiguous in nature, we focus our main analyses on Polymarket, but patterns are generally similar across the two.

We developed our rubric for grading contract quality by reading a bunch of contract rules ourselves, and conversing with our buddy Claude. First, we came up with ten dimensions that we and Claude thought captured common failures we saw in contracts. Then, we worked iteratively to develop a system of scores from 0-3, where a 0 represents clarity or no concern on a given dimension, and 3 represents severe ambiguity. Last, we put a large sample of the contracts—weighted to include more of the disputed ones, because disputes are pretty rare in prediction markets—through the finalized LLM grader (using GPT-4.1 to save money for now) and got back our final grades.

Here’s how this process plays out for the famous “Zelensky suit” blow-up that we discussed in a previous piece as a good example of how contract rules can go awry. (Note: we worry that the AI could bring in its knowledge of a disputed case when doing the grading in some cases—in the conclusion we discuss this a bit more.)

The most challenging part of improving our prompt was the tradeoff between specificity and generalizability. Giving examples of 0-3 rankings for the LLM to use only worked within the contract type for a specific market, not more broadly. For example, in the first version of the prompt, we tried to enumerate many specific oversight institutions such as the Fed, CDC, FIFA, the Grammys, Spotify charts, ADP, Billboard as examples for the LLM to learn from. But we found that when we enumerated, the LLM anchored to only these examples, failing to develop similar, analogous institutions for contracts in other domains.

To counter this, we removed specific examples and replaced them with generalizable context that would allow the model to reason. Additionally, the LLM scored poorly on Polymarket contracts that resorted to “a census of credible reporting” for resolution. In early iterations of the prompt, the LLM graded these leniently. We changed the prompt to evaluate the weakest resolution path that the contract permitted, meaning we graded the most dangerous resolution sources for each contract.

Results: our AI can predict which prediction-market contracts will get disputed, just from the text of the resolution rules

To see whether our grades predicted disputes, we used Fable in Claude Code to throw a bunch of different machine-learning methods at the problem.

As the table below shows, a standard ML approach, Gradient-boosted trees, turns out to perform the best when tested against “out-of-sample” cases, and it performs quite well—the standard measure of predictive accuracy we use, AUC, roughly gives the rate at which the model can guess which of a pair of cases (one that ended up disputed, one that didn’t) is the one that will go to a dispute. Our ML model is able to do this on 3 out of every 4 tries, roughly, which is much better than chance. It’s not a crystal ball, by any means, but it’s a remarkably strong signal to draw purely from the text of the contract.

The figure shows this accuracy another way. Each point in this plot represents an average of a number of underlying contracts. The x-axis shows the predicted risk of those contracts based on their language; the y-axis shows the percentage of those contracts that ended up disputed. As the plot shows, the contracts predicted to be riskier go to disputes at a much higher rate than those predicted to be less risky.

(Technical note: these are raw probabilities on the sample that is weighted towards disputed cases – in work not shown we evaluate how this accuracy looks like when re-calibrated to a random sample, finding the same kind of pattern.)

What text features predict disputed contracts?

A neat feature of this approach is that we can also extract which features tend to be most predictive of contract disputes, using the rubric that we developed with Claude. Three features really stand out: when the core question is vaguely defined; when it’s unclear which entity counts; and when the settlement source is not identified.

It might be that these are common mistakes made in hastily written contracts, and that platforms should focus on defining questions more clearly, specifying entities more explicitly, and identifying specific settlement sources in all the contracts they write. But it’s also possible that there is a reverse causation here—maybe events that are hard to call and likely to be disputed are also inherently harder to write contracts for. We’ll need to do more research down the line to really understand this, but in the meantime, our model can certainly suggest potential weaknesses in contracts so that the authors can at least consider whether they have ways to address them beforehand or not.

A grading system for prediction-market contracts

To see how our approach can help, consider the prediction market example. Investors often like to consider assets by looking at “grades” of their quality. Using our predicted probabilities of disputes, we can divide the prediction-market contracts into bins reflecting the risk they’ll end up disputed. Contracts graded “A” are the least likely to be disputed; each subsequent lower grade increases the risk of dispute steadily, with CCC contracts 3.4x as likely to be disputed as A contracts.

Here are a few examples of what the different contracts and their grades look like.

From prediction markets to political superintelligence

When Raffles and Wichelhaus signed their cotton deal, their best defense against an ambiguously written contract was the gimlet eye of a human expert—the same method we still rely on today, for everything from the resolution rules for strange prediction-market contracts to the laws that shape society, the rules that govern global online platforms, and business transactions taking place all over the world. But humans, we know, are fallible. Rules get written ambiguously, or fail to anticipate issues that could have been anticipated in advance. AI is fallible, too, but it could have a very useful role to play in helping us improve the way we write rules all across society.

There are a whole host of limitations to our experiment. First, we’ll be subjecting our method to a truly “out of sample” test by having it evaluate contracts that haven’t yet been resolved, in case the grader LLM is somehow bringing in knowledge of contract disputes when creating the grades. We also need to replicate it for more contracts in more domains. And we need to take the further step of having the AI then help re-write contracts to make them better. All of this lies in the future.

But the direction of travel seems clear. The core act of governance—writing rules that bind us, and adjudicating what they mean when reality gets complicated—has depended for all of human history on scarce, expensive, fallible human judgment, and that constraint has quietly shaped what kinds of institutions we could build. AI loosens it. If a model reading resolution rules can spot the disputes coming three times out of four, then the same intelligence, pointed at statutes, platform policies, and treaties, can help us write agreements that fail less often and govern better. That is the promise of political superintelligence, and this is one small, concrete step toward it.

Like Free Systems? Consider becoming a paid subscriber — all proceeds go to supporting Free Systems community events and programs. By subscribing, you help us to further our research, keep all our content free and open, and gain reach on Substack (which only promotes newsletters with paid subscribers.)