Beyond the Traditional: What If the Data We Need Doesn’t Exist Yet?

Organizations are increasingly moving from experimenting with AI to integrating it into everyday business operations. Predictive models support decisions, generative AI assists employees, and AI agents are beginning to perform more complex tasks across business processes.

Yet as AI becomes more capable, an important question deserves greater attention:

Are we testing AI solutions against enough of the situations they might encounter in practice?

Real-world data remains essential, but it has limitations. Some situations occur too rarely to provide sufficient examples. Other data is sensitive, difficult to access or expensive to prepare. And certain scenarios may never have happened, even though we need to understand how an AI solution might respond to them.

This is where synthetic data offers an interesting opportunity.

What is synthetic data?

Synthetic data is artificially generated information designed to reproduce selected characteristics, relationships or scenarios of the real world. It can include structured records, images, documents, conversations or simulated interactions between people and systems.

Synthetic data can be generated using statistical methods, rule-based simulations or generative AI models. Unlike simply copying existing records, these approaches create new artificial examples for a defined purpose. This reveals an interesting two-way relationship:

AI can help generate synthetic data, and synthetic data can help us test AI.

The value is not necessarily in creating larger datasets. It is in creating relevant examples and scenarios that help us understand how a solution behaves.

From training AI to testing AI

Synthetic data is often associated with training machine-learning models, particularly when real examples are limited. But its potential extends further.

For predictive AI, synthetic data can help supplement rare events or create controlled test cases to evaluate model performance.

For generative AI, it can support evaluation through artificial questions, documents and interactions designed to reveal inaccuracies, inconsistencies or weaknesses.

For agentic AI, synthetic data can become particularly valuable. An AI agent may interpret instructions, retrieve information, interact with systems and take actions. Testing such a solution requires more than checking whether it produces a convincing answer. What happens when a customer provides incomplete information? When instructions conflict? When a connected system becomes unavailable? Or when an agent encounters a situation outside the normal workflow?

Synthetic conversations, records and simulated environments can help teams explore these scenarios before deploying agents into live processes.

The opportunity is to move beyond testing only what we expect AI to do and begin exploring what could happen when reality becomes more complicated.

When does synthetic data create the most value?

Not every AI project requires synthetic data. Its value is greatest when conventional approaches leave important gaps.

When suitable real-world data is difficult to obtain. Sensitive information, limited access and rare events can delay development and testing. Synthetic data may allow teams to experiment earlier, provided privacy and quality requirements are met.

When existing test data doesn't cover enough scenarios. Historical records only describe what has already happened. Artificially generated scenarios can help explore unusual combinations, exceptions and potential failures.

When the consequences of failure are significant. For business-critical processes, identifying weaknesses before deployment can be more valuable than simply accelerating development.

When repeated, controlled testing is needed. Synthetic datasets and simulations can help teams compare different model versions or agent configurations against consistent scenarios.

These applications can help organizations test more systematically and make better-informed decisions about deployment.

How do we know synthetic testing is effective?

Generating data is relatively easy compared with demonstrating that the data is useful. A synthetic dataset may look realistic but fail to capture important relationships, behaviours or unexpected events. It can also reflect the assumptions and biases of the methods used to generate it. Synthetic data generated from sensitive real-world information is not automatically anonymous, either. Privacy and disclosure risks may still need assessment.

For these reasons, synthetic testing should complement, not replace, real-world validation.

Organizations should examine whether generated scenarios are relevant, diverse and sufficiently representative for the intended purpose. In agentic AI, this also means looking beyond the agent's final answer to evaluate the decisions it makes, the tools it uses and the consequences of its actions.

The objective is not to prove that AI performs well on artificial tests. It is to discover where the solution may fail and understand what needs to improve.

Can we measure the business value?

I believe we should. The success of a synthetic-data initiative should not be measured by how many artificial records or conversations it generates. More useful questions are:

• Does it increase coverage of important or previously untested scenarios?

• Does it reveal weaknesses that conventional testing missed?

• Does it reduce the time required to prepare and execute meaningful tests?

• Does it help teams make better deployment decisions?

• Do these benefits justify the costs of generating, maintaining and validating the synthetic data?

For AI agents, organizations can also assess task-completion rates, error handling, tool-use accuracy and recovery from unexpected situations. These measures help connect synthetic testing with operational value. The financial case should include the costs of tools, expertise, integration, evaluation and ongoing maintenance. Faster test generation is valuable, but only when the resulting tests are relevant and reliable.

From an interesting technique to a strategic capability

For organizations considering synthetic data, I would recommend starting with a specific testing challenge rather than selecting a technology first.

• Identify an AI solution where real-world test data is insufficient or difficult to access.

• Determine which scenarios are missing, why they matter and how improved testing would create value.

• Then generate a limited set of synthetic examples, validate their relevance with business and technical experts, and compare the results with existing testing approaches.

• If the approach reveals additional weaknesses or meaningfully improves testing efficiency, there may be a case for scaling it.

Over time, organizations could develop reusable scenario libraries, evaluation methods and testing environments that support multiple AI solutions. That is where synthetic data starts becoming more than a technical experiment. It becomes part of how an organization develops and evaluates AI.

A broader perspective on AI readiness

Much of today's AI transformation conversation focuses on selecting technology, preparing data and developing organizational capabilities. These foundations remain important. But as AI becomes increasingly embedded in business processes, the ability to challenge and evaluate AI solutions before deployment deserves equal attention.

Synthetic data offers another way to strengthen that capability.

It cannot eliminate uncertainty, guarantee reliability or replace real-world experience. But it can help organizations explore situations they would otherwise struggle to test, identify weaknesses earlier and make more informed decisions.

For me, this is the real strategic opportunity. The question is no longer only whether we have enough data to build AI. It is whether we are creating enough opportunities to understand how AI will behave before we scale it.

Make Data & AI work for you.
Simple. Strategic. Future-ready.