Why Qualitative Data Analysis is Taking Off
Traditionally, while analysts have effectively measured quantitative data, qualitative data analysis, such as evaluating customer dialogues, remained predominantly manual. Old methods were limited to extracting predefined keywords and detecting sentiments like "disappointed" or "frustrated."
However, they could not analyze entire conversations comprehensively. Enter Large Language Models (LLMs). These models are transforming qualitative analysis by enabling it on a scale comparable to quantitative methods.
Conversational interfaces, which allow users to ask open-ended questions, have always been recognized as rich data sources. Previously, the sheer volume of data meant that only a small fraction of dialogues could be analyzed manually. Unlike basic sentiment analysis tools that simply categorize conversations into positive, negative, or neutral, LLMs can delve deeper, extracting actionable insights and critical information in a way that mimics human analysis but on a much larger scale.
As LLM-powered applications and conversational AI interfaces see a surge in adoption, they also herald a new era in qualitative analysis. Through prompt engineering, these models can respond to specific questions from analysts or identify new user intents, allowing for nuanced and contextually aware analysis that facilitates direct insight extraction from dialogue.
Consider this example of how to use an LLM for analyzing customer conversations (from ‘Transforming Conversational AI’ by Michael McTear and Marina Ashurkina):
PROMPT: Analyze the conversation and reply YES or NO to the following questions:
- Was the conversation completed?
- Does the conversation require follow-up?
- Was the customer satisfied with the experience?
- Was the customer complaining about the service?
- Has the customer suggested new features?
- Was the customer asking to call them back?
- Did the customer want to speak with a human?
A Leap in Qualitative Analytics
Getting an LLM to answer a few analytical questions from your dataset is an intriguing start to realizing its possibilities. The real power, however, lies in deploying tools specifically built for conversation analysis.
Here, an LLM user intelligence tool like Nebuly proves invaluable. It processes every single user interaction, capturing subtle cues and combining qualitative understanding with quantitative analytics to provide actionable insights. This gives a comprehensive overview that helps discover common user intents and experiences, and identifies conversations leading to suboptimal outcomes and their reasons.
Example customer Question: “What is the best product for doing X?”
Chatbot Answer: A very long, elaborate, and polite answer that users don't have the patience to read.
Result:
- User drops out, potentially losing a customer.
- User follows up with requests like, “Please be more precise,” “Just list the products,” etc., indicating that the initial response was insufficient. This shows that while the answer may have been accurate, it wasn't user-friendly.
In real-time, Nebuly categorizes these insights from user interactions, highlighting issues such as “Lack of personalization,” and helps you understand the frequency and context of these issues.
It is advantageous not just to focus on individual conversations but to monitor these interactions over time. The integration of intuitive dashboards and automated insights into a single platform empowers organizations to fully grasp customer engagement, address recurring issues effectively, and observe changes in user satisfaction over time.
Selecting an Insights Engine Provider
- Privacy and Security: With great power comes great responsibility. Analyzing conversations to extract business insights is powerful but requires a high standard of privacy and security. Ensure your provider meets the privacy and data security expectations of your customers.
- Quality of the Tool: Automated qualitative insights can be somewhat opaque. Different analysts (and different AIs) might interpret conversations differently. It's crucial to ensure the data you receive is of high quality. Features like auditability, which allow reviewing the source of every derived insight, are essential to build trust in the analysis.
The insights must be compiled flexibly, according to each specific use case, and categorized in a manner useful for example for product analysts, customer success specialists, and marketing teams. As interactions with conversational AI applications increase, the scalability of categorization and insights becomes crucial.
If you'd like to learn more about Nebuly please book a meeting and let's chat.
