Report 2024

The State of User Experience in LLM Products

We surveyed 200 AI leaders, who have LLMs in production to understand their biggest challenges, how they evaluate LLMs, measure user experience, and meet executive expectations. The report with 20 pages full of insights is available for free.

Get the report

report

LLM User Experience Survey Report 2024 - Highlights

Number of LLM-powered products is ramping up, with an average increase of 21% expected by mid-2025

‍The number of companies that have 11+ applications in production, is going to more than double. With an average increase of 21% in the number of user facing LLM products by mid-2025, this indicates a clear shift towards scaling up, reflecting growing investment in, and deployment of these technologies.

97% of companies collect LLM user feedback, but methods are mostly manual.

Most of the methods companies use to gather user feedback for their LLM-powered products are manual in nature. The ones that automate conversational analysis are the two lowest adopted methods reported by respondents – sentiment analysis tools (12%) and analyzing conversational flow (7%).

79% of companies find it challenging to integrate user feedback into LLM development.

This finding suggests that while many companies recognize the value of user feedback, they also face significant challenges in effectively incorporating it into their AI development processes. The shortage of skilled AI developers and the low adoption of tools that simplify this process, are likely contributing factors.

Get the full report

Free access to LLM Evaluation Insights, AI Success Metrics, and Executive Expectations from 200+ Leaders Across 12 Countries.

Thank you.
Please check your email for the report.

Oops! Something went wrong while submitting the form.