Headlight is completing a three-part series breaking down one way we’re incorporating emergent AI technology to solve problems and improve marketing efficiency across our organization.
In Part 1: The Problem, we explored the costly effects of misunderstanding customer needs. In Part 2: The Process, we introduced T.E.A., our step-by-step method to turn qualitative data into valuable customer insights for marketing, creative and product decisions.
As we wrap this series with Part 3: The Pitfalls and The Promise, we unveil essential do’s and don’ts derived from extensive testing to ensure accurate, actionable insights. Highlighting top pitfalls not only showcases AI’s current capabilities but also outlines opportunities for marketers to propel their usage of AI tools further. Subscribe to stay informed on innovative strategy and relevant trends for your business.
Without further ado, here are the do’s and don’ts to unlock better results from AI-driven customer insights.
Get Growth Insights Delivered to Your Inbox
Thank you for signing up!
There was an error
The Pitfalls: Mistakes to Avoid
1. Don’t Trust, Verify
At each step include multiple verification mechanisms to reduce issues. A good practice is to request specific review citations to leave a breadcrumb that can be easily followed back to the source and then spot check random samples. This is necessary because sometimes even plausible sounding sources can be made up! Additionally, you can run other independent analyses from 1P data, user surveys or independent analyses to create a fuller picture. 
2. Don’t Mix-in Hypotheticals
Separate the analysis of current realities from potential solutions. We ran into these issues in early attempts where we tried to get to an actionable insight too soon and clouded our analysis by mixing ‘what is’ with ‘what could be’ scenarios. For example, our initial analysis of Strengths and Weaknesses jumped straight into “how can we improve” and the LLM began to act as if its recommendations had already been implemented.
3. Don’t Introduce Bias
Ensure your dataset is large and unbiased to avoid skewed results. In our testing we have used up to 500 reviews. Sometimes bias can be introduced accidentally since every input and output becomes context and this can inadvertently introduce noise that biases the analysis. Additionally, we have seen cases where lighthearted jokes, like you might make to a coworker lead to unintended results. For example, in one instance we referenced an inside joke about pizza and the AI incorporated pizza references and cheese puns into copy. Mama Mia! So keep the delivery on track by steering with the right context. While this may be less tempting in an off the shelf model you still need to be a good role model for the results you wish to see.
4. Garbage In, Garbage Out
It is critical to use high quality data. Beware of fake reviews and make sure your analysis separates between valuable signal and noise. What if the reviews themselves are fake? A UC Berkeley study of fake reviews showed one telltale sign of fake reviews is a “higher rating dispersion”, meaning a wide split between high 5-star reviews and low 1-star reviews. This method can help identify potential issues if there are biased or disingenuous reviews that may be incentivized. In some instances reviewers themselves will even question the legitimacy of positive reviews that do not match their lived experience.
5. Don’t Oversimplify the Customer Experience
When using AI tools, tailor to the specific situation to avoid oversimplifying with a one dimensional reflection of reviews. For nuanced tasks or sensitive situations, use human judgment to avoid unforced errors. If you need a solution that can stand the test of time, make sure you are building flexibility into your solution. In your analysis ask why a solution may work but also critically how it could be wrong. Lastly, don’t use AI to ghostwrite key content that could undermine IP protection. Similar to privacy regulations, there are specific requirements when using AI that will likely pass EU and expand to the rest of world.
Unlock Hidden Performance
1. Take a Human-Centered Approach
Using a human-centered approach is crucial. Carelessly automating personalization at scale can backfire without a thorough understanding of AI capabilities or verifying outputs with a critical eye. Prioritizing quality over quantity respects customer trust and can improve your brand reputation. This also aligns with findings from the 2023 Salesforce study which underlies the importance of human verification and trust.
“80% of customers say it's important for a human to validate the output of AI.”~Salesforce study, 2023
Our testing has found a human-centered approach works well because it builds on existing strengths and overcomes deficiencies of both the AI model and human side of the process. By using AI for brainstorming we have found eureka moments faster and identified ways build off of existing ideas that marketers can apply within a larger strategic vision.
2. Use Clear and Simple Prompts
How do you get the most out of AI? Bit by bit. As of this writing, ChatGPT excels with focused tasks but struggles with complex, multi-step analysis. Begin with narrow prompts to keep the AI on the right track. AI improvements continue to move quickly across all LLM solutions and recent releases are more powerful. So soon multi-step analyses may be able to happen more quickly without spoon feeding through each step. In the meantime, a safer approach is to use tight, focused prompts to keep AI on the right track.
3. Read the Fine Print
Beware of weasel words indicating big caveats or hallucinations. For example, “hypothetical”, “illustrative”, “mock” etc. In our testing we have found this is most common when there are “multi-modal” requests to go from text to visual representations. Though sometimes there’s a useful proof of concept included even if the data is fictitious when we asked for the real thing.
4. Test Systematic Prompt Iterations
GPT is rapidly improving and is getting better quickly. Continue iterating on prompts so the outputs improve as well. In ChatGPT if at first you don’t succeed you can re-prompt by editing your initial prompt. IMPORTANT: Don’t do this after you have continued a stream of analyses as it can clear the following outputs. Moreover, even if you use the same prompts you can sometimes see different results since the model uses a probabilistic approach with multiple possible answers and then ranks which response is most likely to succeed. This means seemingly inconsequential changes to prompts can have larger changes, though newer more powerful models will undoubtedly improve and may have less deviations.
5. Ask your AI to criticize its solution
Asking the AI to self-critique may seem odd. After all, if you ask your employees to self-critique you might hear “I think it’s perfect” even if its not. However, AI will often give a thoughtful response and improve its results upon self evaluation. Still be sure to include guardrails to keep the output on track. This is one very impactful way to improve results. Though you still need human verification as even basic questions can sometimes yield an incorrect responses.
“The best way to predict the future is to create it.”~Peter Drucker
The Future of AI
In the realm of AI, we’re at the start, not the summit. When we began this series we mentioned that AI is changing so rapidly it might change by the time you read this. Well, it did. On May 13, 2024, OpenAI made a massive update with the announcement GPT-4o, an emotive omni-model with a multi-modal approach of text, video, and voice reasoning. This model includes actual emotional intelligence, that prior models could only emulate. Moreover, Google Gemini also released a host of updates from Google I/O 2024.
It is impossible to fully know what this means in the grand scheme of this technology. What we do know is that AI offers real opportunities to innovate, solve problems pragmatically, and unlock real-time insights beyond unaided human capacity alone. We believe in harnessing AI’s capabilities thoughtfully, with human stewardship and clear ethical standards. This series has focused mainly on ChatGPT due to it being the most widely known LLM. However, others notable models to watch are Anthropic’s Claude and Google’s Gemini model who has long made clear its mission is to organize the world’s information for the past 25+ years. There are also likely many other AI solutions that will not survive this pace of innovation.
Like any new technology, the initial demo of V1 is not indicative of its mainstream impacts. Many of the early iterations of new technology seem quaint in retrospect. Just look at internet – it took 14 years for consumer search engines to gain widespread use and access previously buried information. Or this demo of the first iPhone by Steve Jobs. AI is clearly on a different trajectory, moving exponentially faster. Though at the same time many of its essential features are still in development. You must look to where the technology is headed, not where it is today and have good partners help light the way.
How and when you use AI is as important as using the right tool with the correct expertise and process. This is very much a situation where there is no instruction manual. As Peter Drucker once said “The best way to predict the future is to create it“. So, test out AI to improve, challenge and innovate your marketing strategies and processes. Find the initial wins in your test cases, then tackle the hard problems where a powerful solution can unlock the most value.
At Headlight we enjoy constantly challenging our own processes and discovering creative ways to use AI and other innovations to improve our workflows and deliver better work. We believe each individual customer has perspectives that need a bespoke approach to unlock and activate for our client partners. To learn more about how we work, reach out below.
