Papers
arxiv:2610.25159

Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations

Published on Oct 18
· Submitted by
Amir Mohseni
on Oct 20
Authors:
,

Abstract

The study develops a taxonomy to identify legal guidance requests in conversational AI, annotates thousands of chats, and evaluates ModernBERT models against a baseline for detecting such queries.

People ask conversational AI models about their legal rights, obligations, procedures, and personal situations that may have legal consequences. To study this behavior, we first need to distinguish requests for legal guidance from other legally themed discussions and unrelated conversations. We contribute a multi-axis taxonomy of legal queries capturing whether legal guidance is sought, how a request is formulated, and which legal domains it concerns; apply it to 2,122 naturally occurring WildChat-1M conversations; and create a human-reviewed gold evaluation set of 200 conversations. All annotations, model checkpoints, and evaluation code are publicly available. In this enriched sample, 71.1% of conversations containing legal content seek personalized or practical guidance. Rights-and-options advice is the most common primary request type among legal-content conversations, and 25.3% of guidance-positive conversations span multiple legal topics. The gold evaluation set supports a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF cascade. Our framework supports large-scale studies of legal guidance in conversational AI, and the public release of the annotated data facilitates reproducible research and future model development.

Community

We study how people seek legal guidance in real-world LLM conversations. We introduce a taxonomy, annotate 2,122 WildChat conversations, and release a human-reviewed evaluation set of 200 conversations. We also train and evaluate smaller classifiers to detect guidance-seeking requests and classify their legal topics.

The datasets, model checkpoints, and evaluation code are publicly available. We hope these resources support further research into legal guidance seeking in conversational AI.

Happy to discuss the work and hear your thoughts!

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2610.25159
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 8

Browse 8 models citing this paper

Datasets citing this paper 2

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.25159 in a Space README.md to link it from this page.

Collections including this paper 1