The Evolution of AI Operations

Introducing GenAIOps

Rodrigo Masini

5/15/20244 min read

Introduction: Pioneering a New Era with ChatGPT

When ChatGPT was unveiled in November 2022, it marked a transformative shift in AI capabilities, promising a level of human-like interaction that could make informed decisions without the extensive need to filter through extensive, lower-quality content. As a Technical Lead in Machine Learning Operations (MLOps), I recognized that the complexities of deploying such technologies would challenge traditional ML systems, suggesting a more profound revolution in the AI landscape.

I remember an emergent effort to leverage MLOps concepts and practices to the realm of LLMs. While I was convinced that we need something similar to MLOps, it was not clear what specific challenges and complexities a company willing to adopt Generative AI (GenAI) could face.

The Emergence of Generative AI Operations

Moving forward, after I designed and deliver the first two GenAI platform projects in the second quarter of 2023, it became apparent that the operational paradigms for LLMs required a unique set of practices—akin to, yet distinct from MLOps.

Not just my experience but also that of other AI SMEs indicated less than satisfactory results in their initial attempts to implement GenAI capabilities in an enterprise context. Challenges such as balancing context size and few-shot prompts, the GPU costs for running models with more than 70 billion parameters at 16-bit precision, and hallucinations were beyond those faced by traditional ML systems.

LLMOps surge to specifically targets the operationalization of LLMs for NLP-based applications. This specialization was trying to emphasizes the need for precise deployment strategies for GenAI-driven applications, spotlighting the unique challenges and tailored solutions required for effective management of LLMs.

However, the Generative AI technology is disruptive. What that means? Well for an organization this is equals a review of processes, roles, procedures, policies, and even its culture. For instance, in one project, we experimented with Kanban, Waterfall, and even Lean Six Sigma methodologies to manage development and delivery tasks. None of these methodologies could adequately respond to the absurdly fast-paced changes in this field, nor could they match the needed flexibility while maintaining governance standards. Experimentation became essential, as was exhaustive testing against new metrics discovered daily.

This need for a new approach was underscored in the third quarter of 2023 when the GenAIOps CIC introduced the term Generative AI Operations (GenAIOps). The main takeaway was that GenAI represents a significant paradigm shift and thus requires a corresponding transformation in how it is operationalized within an enterprise environment

This innovative approach not only expands traditional MLOps frameworks but also introduces advanced strategies for deploying generative AI across various modalities. Unlike LLMOps, GenAIOps addresses the dynamic and customized operational and governance processes needed due to its profound impact on the operational models of businesses.

Driving Innovation with Specialized Tools and Techniques

GenAIOps extends beyond the basic elements of Generative AI deployment such as data readiness and model serving on specialized GPUs. It also involves the design, decision-making, and implementation of advanced capabilities like synthetic data management, model embedding management, and the management of prompt stores and vector databases. These tools not only significantly enhance the flexibility and efficiency of AI applications but also introduce an additional layer of governance.

In the enterprise GenAI tech stack, GenAI Hubs serve as an organizational-level repository for GenAI artifacts, and GenAI Blueprints provide a minimal, functional modularization code for rapid experimentation. These components are crucial for streamlining processes and accelerating deployment cycles, as noted in a 2023 MIT Sloan Management Review article highlighting the importance of modular design in AI operations.

ARENA, a platform for testing, validating, and ensuring visibility, becomes integral pre- and post-code merge. This provides tech leads with the necessary documentation and lineage to stay in compliance with organizational policies while justifying the need for further investment. This strategic integration aligns with findings from a Gartner report that emphasizes the importance of robust testing environments in AI deployments.

Combining Responsible AI frameworks with business scorecards has proven to be a highly effective strategy. This integration fosters dramatic improvements in organizational AI deployment by ensuring that AI operations are not only technically sound but also align with broader business and ethical standards. According to a 2023 study by Deloitte, organizations that integrate ethical considerations into their AI frameworks see improved stakeholder trust and better long-term outcomes.

What comes next for GenAIOps?

I considered myself as one of the pioneers MLOps engineers and MLOps strategiest since I start to work in such role in 2019. What I see for GenAIOps is a fast and more complex journey when compared to MLOps.

There are several efforts towards democratizing the GenAI technology, however in the costs of unstructured and agentic solutions. While the high abstraction of released solutions allow more people to experiment with the new capabilities, it also reduces the importance of fundamental quality level requirement in an operational environment like prompt engineering, exhaustive pre LLM query security guardrails, etc. For example, RAG is the most important and must-have capability that any company should have. It provides low cost, updated and business context aware capability to organizations adoption GenAI. Unfortunately, we saw several examples of Naive RAG (simple and not production ready) in media. Even in research like the poor paper RAFT released where it is not described the RAG system used to compare the results with RAG plus CoT.

Its crucial that the community have a reference and place of best practices for GenAI projects. The GenAIOps organization fosters dramatic improvements in organizational GenAI operations by ensuring that developers are not only technically sound but also align with broader business and ethical standards.