What to check before adopting an AI product
A convincing demonstration is a starting point. To decide whether AI software fits your work, evaluate the task, the evidence supporting the output, and the consequences of a mistake. The right checks depend on the product and on how your team intends to use it.
Define a representative task
Choose examples that resemble your real workflow: the formats people use, the information they expect, and the decisions they need to make. Define success before testing. A document answer might require the correct source and a useful citation; an action-oriented tool might require correct permissions and an approval step.
Check the source and the failure case
For retrieved answers, check whether the cited passage supports the claim. Try missing information, ambiguous requests, and documents that disagree. A helpful system should make uncertainty visible. A generated response should not be treated as verified simply because it is fluent.
Review access and human control
Ask what the product stores, which systems it can access, and whether it uses your organisation's identity and permissions. For actions, find the review or approval boundary. Claims about an underlying cloud platform do not replace a product-specific explanation of data processing and controls.
Separate an evaluation plan from published results
Microsoft Foundry provides evaluation and tracing capabilities, with availability varying by feature. For future Smart365.AI releases, our direction is to document representative tasks, criteria, and limitations. We have not published model-quality scores for the Dashboard prototype because it performs no live model inference.
Review the Dashboard's current AI boundary