GPT-5 outperforms human judges in legal competition

A recent study conducted by University of Chicago law professor Eric Posner and researcher Shivam Saran has shed light on the capabilities of OpenAI’s GPT-5 model in the realm of judicial decision-making. The study aimed to expand upon previous research involving GPT-4o by testing GPT-5 against 61 US federal judges in legal case scenarios.

In the original study with GPT-4o, the AI model demonstrated a formalistic approach by strictly following the letter of the law without being influenced by external factors. This behavior aligned with findings from prior research that highlighted differences in decision-making between law students and judges – with judges being more prone to considering non-legal factors.

Posner and Saran’s follow-up study with GPT-5 focused on more routine legal questions, such as determining which state law applies in a car accident scenario. The results of the study revealed that GPT-5 exhibited a high level of legal reasoning accuracy, outperforming human judges in strictly following legal outcomes without any evidence of logical errors or hallucinations.

Interestingly, GPT-5 was not the only model to achieve perfect compliance in legal decision-making. Google’s Gemini 3 Pro also demonstrated a 100% formalistic approach, while other models varied in their compliance rates, with GPT-4.1 and Llama 4 Scout aligning closely with human judges at 50% adherence to the law.

While AI models like GPT-5 show promise in legal applications due to their consistent adherence to legal principles, questions arise regarding the implications of relying solely on AI for consequential legal decisions. As AI technology becomes more prevalent in legal settings, the issue of whether AI should transition from a supporting role to a decision-making role must be carefully considered by legal experts, lawmakers, and society as a whole.

Posner and Saran’s research underscores the contrast between AI models and human judges in legal decision-making. While AI models excel in formalistic adherence to the law, human judges have the advantage of applying discretion in cases where strict adherence may lead to unfavorable outcomes from a moral, social, or policy perspective.

As the debate continues over the role of AI in legal decision-making, the study highlights the need for a balanced approach that considers both the strengths and weaknesses of AI models compared to human judges. Ultimately, the question remains as to whether society is willing to accept AI judgments that may lack the nuanced considerations and human biases inherent in human decision-making processes.