Originally Posted on Search Engine Journal by Matt G. Southern
How does Google know if its search results are improving?
As Google rolls out algorithm updates and claims to reduce “unhelpful” content, many wonder about the true impact of these changes.
In an episode of Google’s Search Off The Record podcast, Google Search Directer, Product Management, Elizabeth Tucker discusses how Google measures search quality.
This article explores Tucker’s key revelations, the implications for marketers, and how you can adapt to stay ahead.
Multifaceted Approach To Measurement
Tucker, who transitioned to product management after 15 years as a data scientist at Google, says it’s difficult to determine whether search quality is improving.
“It’s really hard,” she admitted, describing a comprehensive strategy that includes user surveys, human evaluators, and behavioral analysis.
Tucker explained
“We use a lot of metrics where we sample queries and have human evaluators go through and evaluate the results for things like relevance.”
She also noted that Google analyzes user behavior patterns to infer whether people successfully find the information they seek.
The Moving Target Of User Behavior
Tucker revealed that users make more complex queries as search quality improves.
This creates a constantly shifting landscape for Google’s teams to navigate.
Tucker observed:
“The better we’re able to do this, the more interesting and difficult searches people will do.”
Counterintuitive Metrics
Tucker shared that in the short term, poor search performance might lead to increased search activity as users struggle to find information.
However, this trend reverses long-term, with sustained poor performance resulting in decreased usage.
Tucker cautioned:
“A measurement that can be good in the long term can be misleading in the short term.”
Quantifying Search Quality
To tackle the challenge of quantifying search quality, Google relies on an expansive (and expanding) set of metrics that gauge factors like relevance, accuracy, trustworthiness, and “freshness.”
But numbers don’t always tell the full story, Tucker cautioned:
“I think one important thing that we all have to acknowledge is that not everything important is measurable, and not everything that is measurable is important.”
For relatively straightforward queries, like a search for “Facebook,” delivering relevant results is a comparatively simple task for modern search engines.
However, more niche or complex searches demand rigorous analysis and attention, especially concerning critical health information.
The Human Element
Google aims to surface the most helpful information for searchers’ needs, which are as diverse as they are difficult to pin down at the scales Google operates at.
Tucker says:
“Understanding if we’re getting it right, where we’re getting it right, where needs focus out of those billions of queries – man, is that a hard problem.”
As developments in AI and machine learning push the boundaries of what’s possible in search, Tucker sees the “human element” as a key piece of the puzzle.
From the search quality raters who assess real-world results to the engineers and product managers, Google’s approach to quantifying search improvements blends big data with human insight.
Looking Ahead
As long as the web continues to evolve, Google’s work to refine its search quality measurements will be ongoing, Tucker says:
“Technology is constantly changing, websites are constantly changing. If we just stood still, search would get worse.”
What Does This Mean?
Google’s insights can help align your strategies with Google’s evolving standards.
Key takeaways include:
- Quality over quantity: Given Google’s focus on relevance and helpfulness, prioritize creating high-quality, user-centric content rather than aiming for sheer volume.
- Embrace complexity: Develop content that addresses more nuanced and specific user needs.
- Think long-term: Remember that short-term metrics can be misleading. Focus on sustained performance and user satisfaction rather than quick wins.
- Holistic approach: Like Google, adopt a multifaceted approach to measuring your content’s success, combining quantitative metrics with qualitative assessments.
- Stay adaptable: Given the constant changes in technology and user behavior, remain flexible and ready to adjust your strategies as needed.
- Human-centric: While leveraging AI and data analytics, don’t underestimate the importance of human insight in understanding and meeting user needs.
As Tucker’s insights show, this user-first approach is at the heart of Google’s efforts to improve search quality – and it should be at the center of every marketer’s strategy as well.
Listen to the discussion on measuring search quality in the video below, starting at the 17:39 mark:
Featured Image: Screenshot from YouTube.com/GoogleSearchCentral, June 2024
