An Uptime Mindset Gives You A Better Way to Think About AI

The uptime mindset and AI

It has been shown that AI models in real-world contexts do not always perform as expected [based on pre-deployment] testing environments. Post deployment issues include … hallucination, sycophantic behavior, security exploits, and false claims … models have been found to detect when they are being evaluated … the variability introduced by AI models, coupled with the many system components … and user interactions, forms a large attack surface …

NIST Trustworthy and Responsible AI¹

This month the National Institute of Standards and Technology (NIST) issued a report about the challenges of post-deployment testing of AI systems in organizations. Real-world testing has been a standard practice of any tech installation. But AI tools present new challenges.

The report confirms areas of concern NIST raised in its 2023 publications, but with M365 Copilot and other newly mainstream business-use AI, the attack surface has grown.

Here are the four broad issues pointed out by by NIST:

Hallucination, sometimes called confabulation, is when the LLM predicts statistically likely next words that are not true. So documentation it writes may be wrong (this came up on The Pitt HBO show this season).

Sycophantic behavior is the result of humans training the AI and preferring AI answers that are polite or otherwise agreeable to what strikes the trainers as disagreeable; this human preference defeats what’s actually true. And you will get no sign that anything’s wrong (unlike the tells with a human fawning liar).

Gen AI has some unique security exploits. Some can be handled with proper restrictive permissions. Others include indirect prompt injection (the top security issue according to security nonprofit OWASP): hidden instructions on a document (webpage, email, attachment, image) that instruct an AI agent to execute a malicious action. These malicious actions – let’s say getting your M365 Copilot-enabled account to email a spreadsheet to a criminal – may evade detection because it is done without malware and can happen during normal working hours and executing tasks within account permissions.

False claims refer to AI misrepresenting itself – asserting understanding, emotions or intent it does not have. This is again the result of how gen AI is trained: human-written text is the dataset for AI, which humans then reward when returned in agents’ responses. Simulated understanding leads to misplaced trust which leads to lax oversight. I don’t mean to pick on autonomous-car drivers that fall asleep, but it’s such a vivid picture of any employee that thinks ‘the AI’s got this.’

How to get the wins and minimize the risks

An uptime mindset – where the aim is that your organization maintains operation, not severely disrupted by a disaster or other incident – poses questions like these:

• What are the areas of our work that would most benefit from generative AI automation?
• What do we gain or lose by deploying this AI to do x job?
• Are we trading doing for overseeing? What kind of a return does that give on invested time and money?
• Is this AI tool reliable enough to build x process around?
• If this AI tool becomes inaccessible (like because it’s a cloud service or there’s an internet outage), what would be the effect?
• If this AI tool fails, will a staff member be able to perform the function?
• What data will this tool be able to access? What will it retain? Worth the risk?
• Who in the process will be held accountable for an AI mistake?
• What would it look like to change your minds about an AI tool’s integration?

Understand Shared Responsibility of Cloud-Based AI Software

From Microsoft’s Shared Responsibility terms: For Software as a Service (SaaS) solutions like Copilot, Microsoft manages the infrastructure, model operations, and embedded safety systems. Customers are responsible for appropriate use, access controls, and user education². You’ll find this principle holds for all cloud-based platforms.

Microsoft puts the burden on its licensees to monitor and control how Copilot is used within their organization. This means that if a prompt injection attack succeeds — getting Copilot to leak data, for example — Microsoft is unlikely to be any help. Losing data may not only be an issue with clients but with compliance bodies, too. Prompt injection threats are still emerging, and effective defenses are emerging, too, but limiting who has Copilot access and what data it can reach is among the concrete steps available to lessen your risk.

From a legal point-of-view, Attorney Charles Nerko explained to the Heller Report: Companies are expected to supervise their AI systems just as they would their human workforce … your business bears primary legal responsibility for any issues that arise. 

Bottom line is that you’re responsible for all AI use and results. This includes the trend toward employees bringing their own AIs to projects – a practice which gives your organization no controls just liability.

So with regard to AI software, do you have established governance processes of direction and control (aka strategy and oversight), so that your organization’s objectives are achieved and risks are managed while being attentive to your financial and training investments? Bryley has the program that can help you get these set up right.

Just the best and brightest

Having an uptime mindset is about advancing intelligently – with the right planning, preparation, testing and with the right expectations.

There’s no doubt that gen AIs are an intriguing, exciting and emerging tool. But also, as NIST shows, there’s no doubt that gen AIs have weaknesses.

These models have also been updated without subscribers being aware – will an AI-assigned process yield a different result Tuesday than it did Monday? A lot is hidden, not to mention happening in immense distant datacenters, and therefore beyond organization’s oversight.

Bryley’s staff can help guide you in maintaining uptime as you deploy or explore integrating AI in your workflow. To speak to Bryley’s Roy Pacitto please complete the form, below. Or you can email Roy at RPacitto@Bryley.com or reach him by phone at 978.562.6077 x217.

^{1 https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-4.pdf
2 https://learn.microsoft.com/en-us/compliance/assurance/assurance-artificial-intelligence}