Dear CIO,

Some architectures stand the test of time… until they don’t. As artificial intelligence reshapes enterprises, familiar stacks like LAMP are beginning to show their age. Managing today’s AI at scale demands a re-imagination of the entire foundation. Enter NORMAL: a new AI stack purpose-built for the complexities of modern organizations. In this article, we’ll unpack each layer of this new stack, exploring how NORMAL helps teams embrace the future of AI with clarity, flexibility, and purpose.

Best Regards,
John, Your Enterprise AI Advisor

NORMAL is the New Normal

How Do We Normalize AI in Enterprise IT Operations? 

Some things never change, while others undergo radical transformations. Regarding contemporary AI, we must learn from the past while embracing evolving mindsets and the future. Traditional models for managing infrastructure and application stacks, such as the long-standing LAMP stack, must give way to innovative approaches. 

LAMP gives way to new approaches explicitly designed for managing AI at scale. Consider NORMAL—an innovative, multi-layered stack that addresses the complex challenges of deploying and managing AI systems in large, modern organizations. This article will explore each component of the NORMAL stack, tailored for a diverse IT audience, including developers, operations teams, product managers, tech leads, and CIOs.

N – A New AI Stack for Modern Organizations

The "N" in NORMAL signifies a novel approach to managing AI. Unlike traditional stacks that once formed the backbone of web development, the NORMAL stack is crafted to tackle the complexities of modern AI applications. It recognizes that there isn’t a single path to success - countless ways to implement solutions at every stack level. This flexibility allows organizations to choose the best tools and methods that align with their specific operational needs while ensuring scalability, agility, and performance in management AI.

For organizations scaling AI initiatives, this is not just a technological shift—it’s a fundamental change in approach. Developers can leverage multiple programming models, operations can integrate varied systems seamlessly, and decision-makers can see tangible improvements in how AI systems are deployed and managed.

O – Embracing Modern and Post-Modern Observability

Observability is the "O" in NORMAL and is critical to maintaining reliable AI systems. Modern observability tools such as Honeycomb and Dynatrace have revolutionized how we monitor and troubleshoot traditional applications. In the new stack, they are pivotal in providing real-time insights and helping detect anomalies before they escalate into critical issues.

However, the NORMAL stack extends beyond traditional monitoring. It integrates post-modern observability through advanced ML-based evaluation tools such as Arize and Langsmith. These tools monitor system performance and deliver deep analytics on AI model behavior and decision-making processes. They offer metrics like bias, toxicity, correctness, and hallucinations. This layered approach enables teams to comprehend system performance from technical and business perspectives, ensuring that every aspect of the AI ecosystem is observable, measurable, and improvable. 

Soon, modern and postmodern observability will converge, referred to as Autonomous Observability. This implementation will employ advanced agentic processing for monitoring and remediation. Ultimately, these solutions will merge the modern and postmodern elements of observability. 

Postmodern Observability

https://www.linkedin.com/pulse/postmodern-observability-john-willis-fut9e/

It is essential to use observability when it comes to contemporary AI governance. 

Addressing Contemporary AI Governance Challenges

• Modern Observability Tools for LLMs

  • Dynatrace

  • Honeycomb

• Post-Modern Observability Tools

  • Arize

  • Galileo

  • Langsmith

While Dynatrace and Honeycomb provide robust and comprehensive observability for a wide range of systems—focusing on infrastructure, application performance, and real-time diagnostics—Arize, Galileo, and Langsmith specialize in machine learning observability, delivering targeted insights into model performance, drift, and behavior in production. Essentially, the first group emphasizes understanding and maintaining overall system health, while the latter focuses on ensuring that machine learning models perform reliably and transparently once deployed.

Please note that these are just examples. 

R – RAG and Augmentation Generation

The "R" in NORMAL stands for Retrieval-Augmented Generation (RAG), an approach that extends AI’s ability to process and generate information by integrating external data sources. RAG represents a step from traditional AI generation techniques by combining retrieval methods with generative capabilities.

Moreover, the NORMAL stack supports various augmentation generation variants—such as Cache Augmented Retrieval (CAG), GraphRAG, and Agentic RAG—each providing distinct advantages based on the application. Whether enhancing search results, improving content generation, or delivering real-time insights, these techniques enable organizations to make more effective, data-driven decisions. They ensure that AI systems create content with a contextual understanding that enhances accuracy and relevance.

• Enterprise RAG Solutions

  • MongoDB Atlas Vector Search

  • Elastic

  • Postgres

  • Pinecone

  • Milvus

• Framework RAG Solutions

  • LamaIndex (SimpleVectorStore)

  • Hayststack (InMemoryDocumentStore

• Commodity RAG Solutions

  • ChromaDB

  • FAISS

  • SQLite-VSS
    

• GraphRAG Solutions

  • Neo4j

  • Microsoft GraphRAG

Please note that these are just examples. 

M – Robust Model Management

Model Management represents the "M" in NORMAL and is crucial in maintaining the lifecycle of AI models. This component includes everything from embeddings and fine-tuned embeddings to the storage and versioning of models as artifacts. Tools like Artifactory can handle these models like traditional development pipelines manage software artifacts.

Effective model management ensures reliable deployment, systematic updates, and governance through well-defined policies for AI models. Organizations can ensure consistency, compliance, and rapid iteration by treating models as first-class citizens within the software delivery lifecycle. This approach is especially crucial in environments where AI models continuously evolve and must adapt to new data and changing business conditions.

• AI/ML models as Artifacts

  • Artifactory

  • Nexus

• Version Control

  • Github

  • Gitlab

• Model Versioning

  • DVC

  • MLFlow

Please note that these are just examples. 

A – Agents and Agentic Processing

The "A" in NORMAL stands for Agents and Agentic Processing, a concept that adds a layer of automation and dynamic decision-making to the AI stack. Agents are autonomous code modules that perform tasks, interact with other system components, and learn from their environment interactions.

Managing agents like other software artifacts—using robust version control, testing, and deployment pipelines—ensures their behavior is predictable and reliable. Agents can automate routine processes, manage workflows, and provide real-time responses to changing system conditions. This reduces the burden on human operators and enhances the overall efficiency of AI-driven systems.

• Agents

  • LangGraph

  • BeeAI

  • CrewAI

Please note that these are just examples. 

L – LLM Management for Next-Generation AI

Finally, the "L" in NORMAL represents LLM Management, which is becoming increasingly important as large language models (LLMs) become integral to modern applications. Managing LLMs involves more than simply deploying models—it necessitates a comprehensive infrastructure-as-code approach for deployment. Large enterprises require an architected solution for managing developer and service resources at scale. 

A contemporary AI architecture should include an LLM gateway implementation (for example, a common entry point for all chat requests that can authenticate with OKTA). Beneath the gateway, there would be an LLM router and a pool of fit-for-purpose models. This could involve a MOE (Mixture of Experts) structure and orchestrating various LLM architectures to operate effectively and cohesively.

LLM management ensures these effective models are scalable, secure, and integrated smoothly into the broader AI stack. Organizations can maximize LLMs' full potential by automating infrastructure provisioning and upholding robust deployment practices while minimizing operational complexities. This layer of the stack also includes small language model orchestration. 

Below is a table summarizing some of the challenges in productizing LLMs:

• LLM Orchestration

  • LangChain

  • LlamaIndex

  • HeyStack

  • DSPy
    

• LLM Frontier Model Providers (Closed Source)

  • OpenAI

  • Anthropic

  • Microsoft

  • Google (Gemini)
    

• LLM Open-Weight Models

  • Granite 3.2 8b and 2b - IBM 

  • Gemma-2 27B – Google

  • Command R+ – Cohere

  • Grok-1 – xAI

  • Mistral Large 2 – Mistral AI

  • LLaMA 3.1 405B – Meta AI

  • DeepSeek Coder V2 – DeepSeek AI

  • Nemotron-4 340B – NVIDIA

  • Phi-3 Medium – Microsoft

  • OpenLM 7B – Apple

Please note that these are just examples. 

Integrating NORMAL into the Software Delivery Lifecycle

The success of the NORMAL stack relies on its integration with established software delivery practices like DevOps, DevSecOps, GitOps, and SRE for reliability and manageability. Each NORMAL component—from LLM management to observability—is part of a holistic lifecycle emphasizing continuous integration, delivery, and ongoing improvement.

This integrated approach ensures that every aspect of AI management is secure, compliant, and aligned with overarching strategic goals. It also promotes an environment where feedback loops facilitate rapid iteration, resulting in more resilient and adaptable AI systems.

Conclusion

The NORMAL stack significantly changes how modern organizations approach AI management. By comprehensively understanding each layer—from the foundational stack to the intricacies of observability, retrieval, model and agent management, and LLM oversight—NORMAL offers a complete architecture that addresses the ever-evolving demands of contemporary AI implementations.

For developers, operations teams, product managers, tech leads, and CIOs alike, embracing something like NORMAL means more than just adopting new tools—it requires rethinking the entire AI lifecycle for a future where AI is seamlessly integrated into every facet of business operations. As AI continues to transform industries, NORMAL is set to become the new standard in large-scale AI management.

“This newsletter is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.