Review of Microsoft’s year in Responsible AI

Microsoft recently published their responsible AI transparency report. As they are the major shareholder in Open AI, their view on ethical use of AI is, by dint of market share, industry-leading. So what do they say?

Actually a lot. I’m impressed with how many tools they’ve made available to us all to use - apparently 30 tools and 100 features, many open access (some open source), including:

• Best practice documents

• Transparency Note on how to deploy Azura OpenAI models responsibly, with examples

• Case study on how to red team a LLM application

• Prompt shield to block prompt injection attacks

• Safety evaluation tool

• Risks & safety monitoring in Azure OpenAI service

Let’s look into a few of these.

Red Teaming caught my eye as it’s a form of destructive testing that’s been around for a while but not well adapted for AI. Normally, a Red Team exercises techniques that a bad actor might to gain unauthorised access or advantage from software. Microsoft extends this concept to AI by simulating actions by both adversarial users to misuse an application, and also non-adversarial users that could still generate stereotyping content. Their Red Team are experts independent from their product team. Because of the additional resource and the very high skill level of the resource, this approach seems out of reach for most companies that aren’t big AI.

New measurement capabilities have been introduced for risks & safety monitoring. Efficacy metrics can now include groundedness (how well generated answers align with the input “ground truth”), relevance (how directly generated answers are pertinent to input prompts) and similarity (how equivalent a generated answer is to input prompts). I’m excited that there might be computed metrics that speak to how trustworthy generated individual answers are.

There are also metrics for the application as a whole, metrics that measure the application’s likelihood to produce excluded content, and that measure the applications resiliency against jailbreaking. The ML development platform could already enable fine-tuning model; now models can be fine-tuned to increase safety by optimising these metrics.

There are now multimodal content filters and detect not only harmful and prejudicial content, but also jailbreaking content. Their filters are not only improved using GPT-4 but also uses expert human annotators.

Microsoft recognises that creating disinformation is the biggest harm that GenAI has given to the world in this global election year, so now automatically assigns content credentials (using tech developed by our very own BBC) to AI-generated images from its services. Which presumably can be removed by the next AI bot in line.

Azure OpenAI is starting to look like not just a capable but also a responsible AI platform. However to take advantage of the tools and best practices I’d have to go all in on Azure. This means I have a new trade-off to consider: responsibility at a cost.