When AI gives advice - who do we really trust?

This week I am participating in NOVA Day. Many of the conversations there are naturally about what AI can do for businesses. It is important, but one issue that I believe receives too little attention is how people react when the systems start giving advice.

AI does not eliminate human bias, but rather can amplify it.

Part of the explanation lies in how generative AI actually works. The models do not respond because they "know". They statistically calculate what is the most likely next word based on large amounts of training data and the context they are given. This means that the models reflect patterns in the data and are influenced by how we frame the question.

But bias does not only occur in the model. It also occurs in the meeting between model and human.

Research from, among others, MIT Sloan, Harvard Business School and Stanford points to several mechanisms that can weaken decision-making quality when AI is used in practice:

Automation bias
When people attach too much importance to the recommendation from the system, because it comes from a model and is perceived as objective.

Algorithm aversion
When people lose confidence in the model after one visible error, and then reject it entirely, even though over time it may be better than gut feeling alone.

Data bias
If the data base is skewed, so will the analyses.

Confirmation bias
We like to interpret AI responses in a way that confirms what we already believe.

In practice, I see this clearly in commercial situations. An AI model can analyze sales data and suggest which customers should be prioritized, which offers should be followed up or which accounts have the greatest growth potential. The salesperson or manager can make two mistakes by either following the recommendation blindly, or ignoring it completely.

If the recommendation is followed uncritically, biases in the data or model can be reinforced. If it is rejected because it "feels wrong", you simultaneously lose the value of the pattern recognition the model can actually add.

This is where the management challenge lies.

The mature organization does not ask people to choose between technology and discretion. It builds processes where recommendations are tested, model limitations are understood, and analysis is combined with judgment. In other words, a culture is being built where people still ask questions, even when the system seems secure. It is only then that AI becomes decision support in the true sense of the word.

To me, this is one of the most interesting things about AI right now. Not just what the model can produce, but what happens to our judgment ability when the answer comes quickly, looks convincing, and is wrapped with high confidence.

In the work of building and testing my own AI models and tools, I notice this clearly. Small adjustments in data, context or instructions can produce different recommendations. It constantly reminds me that the model gives suggestions and not definitive answers. Testing is a necessity on an ongoing basis.