Testing Emerging Technology Before Investing in It

A request to explore a new technology is too open-ended to produce a useful business decision. It can lead to an impressive demonstration while leaving the original problem largely unexamined.

Write down what the business does not yet know. Can a new approach improve a task that is currently too slow, unreliable or difficult to perform? Is the obstacle technical capability, the quality of available information or the way the work is organised?

The question should connect the technology to a consequence the business cares about. A successful experiment then produces evidence relevant to a decision, rather than a collection of features that happen to work.

The UK Government's guidance to solve a whole problem for users is useful here: investigate the actual need before fixing on the solution. A commercial business can apply the same discipline to its own investment choices.

Some questions can be resolved before building anything. A provider may not support the required market, the necessary data may be unavailable or a simpler process change may address the problem. Finding that early is a useful result.

For questions that require a trial, identify the person who will decide what happens next. They should agree that the experiment can lead to rejection or deferral, as well as adoption. Otherwise the trial risks becoming a demonstration staged to support a decision already made.

Document how the task works today. Include its delays and errors, but also the useful judgement and informal coordination that keep it functioning.

Then identify a simpler alternative worth comparing. That may be an improvement to the current process, a conventional software feature or an existing product. The new technology should be assessed against a credible option, not a deliberately weak version of the status quo.

Consider a hypothetical distributor evaluating connected sensors for storage monitoring. A demonstration may show that a sensor can send readings to a dashboard. The business question is broader: can the proposed arrangement detect relevant exceptions and help staff respond appropriately within the operating conditions of the site?

The comparison should include the current checks, a realistic improvement to those checks and the proposed sensor workflow. Each option needs a fair account of the work involved.

Choose a baseline that reflects normal variation. A quiet demonstration day may not represent the periods when the problem is most consequential. Record the limits of the test so they remain visible during the decision.

UK Government guidance on measuring service benefits emphasises baselines and the distinction between different kinds of benefit. For a business, that means treating an estimated saving, a measured improvement and cash actually released as different outcomes.

A useful comparison explains which option solves enough of the problem at an acceptable operating cost.

Choose the evidence that would justify proceeding before seeing the results. This reduces the temptation to redefine success around whatever the demonstration happens to do well.

The conditions should cover the intended benefit and the constraints that cannot be traded away. A faster process may still be unsuitable if it produces errors the organisation cannot reliably detect or requires information it is not entitled to use.

Set thresholds from the business's needs and the consequence of failure. Avoid borrowing a convenient benchmark that has little connection to the actual task. For a small business, the relevant standard may be straightforward: the new arrangement must improve the selected process without creating an unmanageable review queue or support burden.

Define when the experiment should stop. A stop condition might be a material data exposure, an unresolved failure in a critical case or evidence that the proposed benefit depends on an assumption that is false.

For AI-specific proposals, NIST's AI Risk Management Framework core connects understanding the context and risks with an initial decision on whether to proceed. That principle should be applied before extending an AI trial into consequential operations.

Also state what an inconclusive result means. It may justify a narrower follow-up with a revised question. It should not automatically justify a larger deployment.

A pilot should be able to end with a well-supported decision not to proceed. That option gives the evidence a real role in the investment.

A prototype often benefits from favourable conditions: clean input, close attention from its creators and a limited set of cases. The operating service will need to survive conditions that are less convenient.

Include representative exceptions in the evaluation. What happens when information is missing, a connection fails or the same event arrives twice? Who recognises the problem? What can staff do while the system is unavailable?

Return to the hypothetical sensor example. A useful trial would examine the route from an unusual reading to an acknowledged response. It would also ask how staff distinguish a real exception from a faulty or disconnected device. The business needs a workable response process, not simply a stream of notifications.

AWS's Operational Readiness Reviews guidance treats readiness as a continuing review informed by operating experience. A smaller organisation can use that idea to ask a focused set of questions about the service it is considering.

Identify ownership of monitoring, access, updates and incident response. Ask what documentation and skills the business needs to retain. Include the provider dependencies that could prevent the team from diagnosing or recovering the service.

Some conditions will remain untested. State them plainly and decide whether they can be investigated before adoption or must become explicit limits on the initial use. A narrow, understood deployment is easier to evaluate than an expanding trial with unclear boundaries.

The cost of an experiment may bear little resemblance to the cost of operating the result. Trial credits, temporary licences and close supplier support can hide the resources a normal month will require.

Estimate the ongoing arrangement using actual terms where available. Include integration, administration, support, training and the effort needed to maintain the underlying information. If usage affects cost, examine a realistic range and identify who will monitor it.

Separate costs that arise only during setup from those that recur. Include the work of replacing or retiring the existing process. Running both approaches indefinitely may undermine the benefit the proposal was meant to deliver.

Check what the business can retain if the provider changes or the experiment ends. Can it obtain its data in a usable form? Are the configuration and integration documented? What licences or service dependencies would prevent another team from continuing the work?

These questions are especially useful when the technology itself is evolving quickly. The organisation should understand which parts of the proposed solution are stable business assets and which depend on a particular supplier's continued offering.

A promising result may justify adoption within a defined scope, while still making a broader commitment premature. Match the size and reversibility of the investment to the strength of the evidence. More uncertainty should change the terms of the decision, rather than disappear from the presentation.

At the end of the trial, bring the evidence back to the original question. Describe what was tested, what changed against the baseline and which conditions remain unresolved.

Separate observation from interpretation. The team may have measured a shorter processing step while still estimating its effect on the complete service. It may have demonstrated a feature without proving that staff can operate it independently. Those distinctions matter when another person reviews the recommendation.

Record one of the available next steps: adopt within an agreed scope, conduct a specific further investigation, defer until a named condition changes or stop. Give the next step an owner and explain why it follows from the evidence.

If the decision is to proceed, carry the remaining assumptions into delivery planning. Assign review points to them. A trial finding should not become an unquestioned fact merely because the project now has a budget.

If the decision is to stop, preserve what was learned and remove temporary access, data and dependencies through the agreed process. The organisation should be able to use the evidence later without keeping an abandoned experiment running.

A completed experiment leaves a decision the business can explain and act on. Preserve the result even when the answer is to wait: it gives the next proposal a better starting point than another demonstration of the same feature.