Taming the Titans: A New Risk Framework for LLMs in the Enterprise

It is impressive how profoundly LLM’s have penetrated every corner of the enterprise world. From customer support to security assessments, these new technologies, arguably new for their gargantuan outreach, have made their way onto every machine on which business is done. They are innovative, they are smart, and they are powerful. But as with the titans of myth, immense power comes with immense risk.

For any organisation harnessing GenAI power, simply plugging in an LLM and hoping for the best is not a strategy, but a dangerous, foolish gamble. The surprisingly popular absence of highly comprehensive internal risk frameworks, designed for the unique challenges around LLM-based systems within individual environments, systems can fail, reputations can be damaged, and whole companies can go bankrupt. But when most publicly accepted frameworks are verbose and at times seemingly designed to hinder innovation, what can tech departments with tight deadlines and a need for rapid innovation do to stay compliant while remaining competitive?

As the AI product manager for Jaja Finance (a UK-based fintech scale-up), I’ve faced this exact conundrum. The answer, for us (and for many homologues from various industries), was creating our own framework based on what already exists. Something that works, something that addresses our own issues (both the ones we have and the ones we estimate or even hope to have as we scale), and something that enables us to innovate safely.

In this post, I hope to offer you some considerations for building your own LLM-based risk frameworks. I could argue for the need of an emerging global standard, but if you follow my logic you will understand why that is simply not feasible right now. But before we get there, let’s start with the basics.

Why a Risk Framework is Non-Negotiable

The case for a specialised LLM risk framework is built on a simple truth: these are not traditional software systems. Their complexity and novelty introduce challenges that standard IT risk models were never designed to handle.

For example, LLMs can be inherently unstable and unpredictable. Unlike deterministic software that produces a consistent output for a given input, LLMs exhibit variant behaviours and produce different, sometimes even nonsensical or factually incorrect outputs. These infamous "hallucinations" aren't just quirks: in a business context, they can lead to reputational damage, poor decision-making, and legal liabilities.

Beyond that, LLMs bring about novel risks previously unseen in the industry. The very technology – the notoriously secret computations that happen within a neural network - that makes them powerful also makes them vulnerable. We're seeing new attack vectors like "prompt injection," where malicious actors can craft inputs to hijack the model's output, or "data poisoning," where the model's training data is subtly corrupted to create hidden biases or backdoors. There are also critical concerns around intellectual property leakage, as sensitive enterprise data used in prompts could inadvertently be absorbed into the model, and the inherent ethical biases that can be amplified by these systems.

Beyond the technical dangers, there further happens to be a beneficial reason to create and adopt a risk framework. Standard frameworks, I’d argue of any size, are the best justification for correct budget assignments and sustainable adoption of seemingly tool ecosystems, which can highly elevate the quality of your outputs and the safety of your inputs.

The good news is that there are many frameworks already on the market, so you’ve got plenty to choose from. The bad news? They may not all fit.

Rudimentary Landscape Study

It is worth restating the obvious: that the world of risk management is not starting from scratch, especially in tech. We have robust, time-tested frameworks that have served enterprises well for decades. I’m currently working on a more in-depth analysis of the frameworks I’m going to mention (which are by all means not all that’s available, but the most famous options), but if you are not familiar with them, I’m including hyperlinks:

The NIST Risk Management Framework (RMF) provides a fantastic, structured approach for managing security and privacy risk. ISO 31000 offers excellent principles for enterprise-wide risk management, and COSO's ERM framework is the gold standard for connecting risk to strategy and performance.

More recently, specific guidance like the NIST AI Risk Management Framework (AI RMF) and the OWASP Top 10 for LLM Applications have emerged, providing a much-needed focus on the specific vulnerabilities of AI. These are crucial steps in the right direction, offering a vocabulary and a starting point for thinking about AI risk.

My Issue with Current Risk Frameworks

While these frameworks are foundational, there are a couple of challenges they might pose, from the way they’re elaborated to their potential implementation:

1. They are an adaptation of traditional frameworks – To put it simply, NIST has been around for ever, ISO standards have been governing tech products for decades, so there will always be a risk of legacy bias persistence.

2. They cover a general, high-level usage of LLM’s – This is not something bad, it’s a necessity. A standard framework needs to be accurate for most if not all use cases. But with a kaleidoscopic use-case encyclopedia for LLM’s, should risks not be tabulated with a bit more granularity and specificity?

3. They are hard to adopt, especially by smaller companies - My prediction is that, if you’re a start-up, you won’t be building extensive ISO31000 documentation before adopting LLM’s into your systems. You’re looking to innovate. The verbiage in the big frameworks makes them unattractive for small but ambitious agents looking to make it in an insanely competitive market. So, risk gets pushed to later, and growth is prioritized.

But if we argued that a risk framework is necessary – at any level of growth – and the major standards are not always suitable, what’s there to do?

Recommendation: A Tailored, Multi-Dimensional LLM Risk Framework

To navigate this new landscape, organisations need a more holistic, adaptive framework. Because LLM adoption differs vastly depending on use case, instead of creating a risk management standard, the standard should be in how risks frameworks are created, internally.

At Jaja Finance, we’ve incorporated the features of all the aforementioned frameworks and made it our own. This helped us create a scalable strategy that allows us to adopt emergent technologies safely and understand our appetite for risk vs innovation. We’ve been quite successful in this, with no major security issues around our LLM systems and an arguably large ecosystem (RAG systems, 5 agents in production etc) yielding significant successes. I wanted to share how we’ve approached it.

It's rather simple (as it should be). We followed four pillars:

1. Multi-Dimensional Assessment: Risks should not be viewed as a monolithic category. They need to be assessed across several key dimensions, such as:

• Direct LLM Risks: The model itself. (e.g., hallucinations, performance degradation, model collapse).

• Security Risks: Malicious actors and vulnerabilities. (e.g., prompt injection, data poisoning, model theft).

• People Risks: Human interaction and oversight. (e.g., misuse by employees, over-reliance on model outputs, social engineering).

• Infrastructure Risks: The underlying systems supporting the model. (e.g., API failures, cloud misconfigurations, data pipeline integrity).

Naturally, this can differ based on your own usage and context, but what matters is exhaustivity. Ensure that everything that can affect your LLM’s is documented, from a person pressing the wrong button to the publishers taking the whole model down overnight.

2. Dual-Plane Risk Quantification: For each identified risk, quantification is key. This should be done on two planes to demonstrate the value of controls:

• Inherent Risk: The level of risk before any controls are applied. This is calculated based on likelihood and impact (e.g., on a 1-5 scale that observes occurrences over a sensible period).

• Residual Risk: The level of risk after controls have been implemented. This distinction is vital for justifying investment in security measures and for understanding which risks remain a genuine concern.

We are often unaware how much of the world relies on a risk control rather than inherent safety. In other words, how bad would it be if the gates were open instead of shut? It is imperative to present both scenarios, to enforce the control’s value and emphasize the risk’s actual impact.

3. Action-Oriented Control Documentation: Controls shouldn't just be a checklist; they should be a documented, active strategy. For every risk, controls should be documented in three categories:

• Preventative: What is done to stop the risk from materialising? (e.g., input guardrails to prevent prompt injection).

• Monitoring: How will you know if the risk is materialising? (e.g., observability for anomalous outputs or high-toxicity scores).

• Mitigative: If the risk occurs, what is the plan to reduce its impact? (e.g., a human-in-the-loop review process for sensitive tasks; an incident response playbook).

Prevent, observe, fix is an easy mindset that I like to translate as ‘reasonable effort.’ When everything that can be done to diminish a risk’s impact has been done (in other words, when non-repudiation has been reached), responsibility is distributed and consequences are a natural outcome rather than a mere mistake. In the case of reputational danger, this can be lifesaving.

4. Clear Governance and Escalation: Finally, the framework must define how risks are handled and communicated. This ensures accountability and timely intervention.

• Team Level: Handles low-level technical risks and operational alerts, and is usually mitigated with good coding and ops practices.

• Department Level: Reviews systemic risks and the effectiveness of controls.

• Executive Level: Informed of high-impact residual risks that could affect business strategy or performance, as well as what controls are needed to address them. Since they are the ones approving budgets, any costly control will need to be presented and justified.

• Board Level: Receives reports on critical risks that pose a threat to the organisation's reputation, financial stability, or legal standing.

Making It Your Own: Adapting the Framework to Your Environment

A framework is only useful if it's used. Its implementation must match the culture and agility of the organisation. Regardless of your organisation’s size, a risk framework for LLM usage is essential. Here’s a quick guide on how you could tailor these ideas to fit your size and speed (with a promise of a more in-depth blogpost to come on this very topic):

• For Start-ups (prioritising speed): Focus on a "minimum viable" implementation. Concentrate on the most severe technical and security risks. Use a lightweight system for documenting controls (a shared document or wiki is fine). The goal is to move fast but equip yourself with a risk-adverse mindset for when you scale.
  
  

• For Scale-ups (prioritising growth & compliance): Begin to formalise the process. Broaden the scope to include people and infrastructure risks more thoroughly as the team grows. Start documenting controls with an eye toward future audits (e.g., SOC 2, ISO 27001). This is about building a scalable foundation for trust, building risk resilience for your corporate future.

• For Corporations (prioritising stringent compliance): Implement the full framework with meticulous detail. Integrate it with existing GRC (Governance, Risk, and Compliance) platforms. Demand detailed documentation, clear audit trails, and formal, scheduled reporting to executive and board levels. Here, the framework is a core component of enterprise-wide risk management and regulatory adherence, and can be the decisive factor between innovation and disaster.

Conclusion

LLMs hold the key to the next wave of innovation, but as we all agree at the Centre for GenAI Ops, they must be deployed and operated with wisdom and foresight. By moving beyond traditional models and embracing a framework that is tailored, multi-dimensional, quantifiable, and adaptive, organisations can unlock the immense potential of these titans—safely and responsibly.