RAG vs MCP: What Each Does, When to Use Both

RAG (Retrieval Augmented Generation) retrieves relevant documents from your data to ground LLM responses. MCP (Model Context Protocol) gives LLMs a standardized way to interact with tools and take actions on external systems. RAG fills the model's knowledge gap and MCP fills its action gap. Most production AI systems need both.

The question "should we use RAG or MCP?" keeps surfacing in enterprise architecture discussions. It's the wrong question.

Retrieval Augmented Generation (RAG) and Model Context Protocol (MCP) operate at different layers. RAG retrieves data from your documents. MCP performs actions on external systems. Asking which one to choose is like asking whether you need a library or a telephone.

One of RAG's original co-authors and CEO of Contextual AI, Douwe Kiela explains; "I think people have rebranded it now as context engineering, which includes MCP and RAG." If one of the people who created RAG doesn't see them as competitors, you probably shouldn't either.

The real question: does your data infrastructure support both? Modern AI systems often use RAG to find relevant documents and MCP to take actions based on that information. The organizations deploying agentic AI at scale are already doing both.

RAG solves a specific problem: grounding LLM responses in your own data instead of the model's training data.

Consider a customer support chatbot. A user asks "How do I prepare for my knee surgery?"

Without RAG, the model draws on its internal knowledge and returns a generic answer. With RAG, the system retrieves relevant documents from your knowledge base and generates an accurate response grounded in that material.

Your content gets broken into chunks, either paragraphs or short sections. An embedding model converts each chunk into a vector: a numerical representation of its semantic meaning. Similar content produces similar vectors. These go into a vector database.

When a user's query arrives, the system vectorizes it, retrieves the most semantically similar snippets, and packages them into the prompt alongside the query. Don calls this the "prompt before the prompt." The LLM generates a response based on only the context it was given and not its entire training set.

Long-context models from Anthropic (200K tokens) and Google (1M+ tokens) have prompted speculation that RAG is dying. Four reasons it persists:

1. Token usage at scale. Sending 200K tokens with every query is expensive for high-volume applications. Retrieving five targeted chunks is cheaper by orders of magnitude.
2. Retrieval quality. Five relevant paragraphs often produce better LLM responses than flooding the model with everything. Precision beats volume.
3. Data freshness. Vector databases update incrementally as underlying data changes. Context window approaches require full re-ingestion.
4. Privacy and compliance. RAG indexes can be encrypted and access-controlled independently. Sending all content to the LLM layer simultaneously is a problem in regulated industries where some content is restricted.

In healthcare, RAG adoption is most advanced and the results here are concrete. A radiology clinical decision support study found that RAG eliminated hallucinations, dropping from 8% to 0%. In clinical nutrition, a RAG-augmented system achieved 99.25% accuracy summarizing clinical nutrition data from EHR nursing notes. A RAG system leveraging curated textbooks reached 91.5% accuracy on radiation oncology board examinations.

A variant called GraphRAG models entities and relationships as a knowledge graph rather than flat document chunks, enabling multi-hop retrieval logic for complex chronic disease management.

MCP solves a different problem: giving AI models a standardized way to perform actions on external systems.

Before MCP, every tool required custom integration code, which meant different parameters, different protocols, different response formats. MCP has standardizes tool access. In his video on MCP, Don defines it:

An MCP server wraps one or more tools; internal APIs, external databases, or any callable service, and implements two methods:

1. List tools - the LLM asks "what tools do you have?"
2. Call tool - the LLM says "call this tool with these arguments"

That's it. An MCP client (built into the host application or agent framework) knows how to talk to any MCP server using this standard protocol.

Don draws a parallel:

"In our industry, you might think of this a little bit like a FHIR server. A FHIR server is just a REST server, but there's a particular way that you can ask for patient information, update patient information. The responses back have a certain format that you know how to digest. MCP is similar, it's a standardized way where tools are referred to and you can get results."

FHIR standardized healthcare data exchange. MCP standardizes AI tool access. Researchers are developing frameworks for integrating MCP with FHIR.

Don outlines five reasons MCP matters for enterprise organizations:

1. Tool portfolios compound. Three tools on your first project won't justify the overhead. Scale to 100 tools, and the standardization pays off.

2. Technology agnostic. The tool can be written in any language. The client can be something else entirely. This arm's-length relationship means your tools outlast your current technology stack.

3. Standards simplify change. Switch LLM providers. Change agent frameworks. Your MCP tools don't need to be rewritten.

4. Vendors are shipping MCP servers. Salesforce, Jira, and enterprise platforms across categories now bundle MCP servers. In healthcare, GenomOncology launched BioMCP in April 2025 with 12 biomedical data entities spanning clinical trials, genomic variants, drugs, and published literature.

5. Internet-native. MCP servers don't have to run locally. Weather APIs, documentation services, external databases, any internet resource can be wrapped as an MCP server.

MCP servers are particularly valuable for internal tools where trusted users need to interact with structured data in unpredictable ways such as updating CRM records, fetching live data, triggering workflows.

Anthropic launched MCP in November 2024. OpenAI adopted it in March 2025. Google DeepMind followed in April 2025. In December 2025, Anthropic donated MCP to the Linux Foundation's Agentic AI Foundation (AAIF), co-founded with Block and OpenAI, reporting more than 10,000 active public MCP servers across platforms including ChatGPT, Cursor, Gemini, and Microsoft Copilot.

Function calling is the mechanism inside an LLM where the model signals "I want to invoke a tool." Every platform implements it differently. MCP sits on top and it standardizes how tools are described and invoked regardless of which LLM you use. Function calling is the intent. MCP is the infrastructure that makes the intent portable.

Dimension	RAG	MCP
Primary function	Retrieve relevant context from data sources	Standardized tool access for AI agents
Data type	Unstructured data (documents, articles, manuals)	Structured data (APIs, databases, live systems)
Data flow	One-way: retrieve → inject → generate	Two-way: query → action → result → next step
State changes	No - read data only	Yes - can write, update, trigger actions
Scaling	Increase vector storage and re-indexing power	Add new server endpoints and connections
Core problem	LLM doesn't know your private data	LLM can't act on external systems consistently
Best for	Public-facing apps with predictable questions	Internal tools with unpredictable interactions

Dr. Peter Lee, President of Microsoft Research, described agentic AI as four interlocking capabilities: memory, entitlements, actions, and reasoning. RAG addresses memory which is the external context the agent needs. MCP addresses entitlements and actions such as what the agent can access and do. Reasoning is the LLM itself.

Three questions clarify the decision.

If the knowledge lives in static, unstructured data like clinical guidelines, policy manuals, product documentation, then RAG retrieves it at query time and injects it into the prompt.

If the knowledge is live, structured data such as current lab results, real-time inventory, pending order status, then MCP queries the source directly.

If the use case is question-answering, summarization, or document search, RAG alone is likely sufficient.

If the AI needs to update a record, schedule an appointment, or trigger a workflow, it needs MCP.

A single use case can start with RAG and add function calling if needed. But if you're planning multiple AI workflows across departments, invest in MCP infrastructure now. The tooling compounds.

Scenario	Architecture	Why
Customer FAQ chatbot	RAG only	Read-only; retrieves from internal knowledge bases
Technical documentation search	RAG only	Semantic search over unstructured data
Drug interaction lookup	GraphRAG	Complex entity relationships require multi-hop reasoning
Prior authorization assistant	RAG + MCP	Needs policy knowledge (RAG) and can submit requests (MCP)
Clinical decision support	Agentic RAG + MCP	Multi-step retrieval + action across external data sources
CRM/EHR intake automation	MCP primary	Writes to databases, schedules, sends notifications
Real-time data dashboard	MCP primary	Fetches live structured data from multiple sources

A patient referral workflow illustrates how both work together:

1. MCP checks the patient's live record in the EHR (real-time structured data)
2. RAG retrieves relevant clinical guidelines from the knowledge base (document retrieval)
3. MCP submits the referral to the scheduling system (action)
4. MCP sends a notification to the referring physician (action)
5. The LLM synthesizes everything into a response and audit trail

RAG provides safety through constraint. The system embeds and indexes content in advance; at query time, it retrieves only the relevant snippets. There's no direct connection to live systems, which limits the attack surface.

Organizations in regulated industries typically deploy RAG on-premises or in private cloud. Vector databases run inside the organization's infrastructure, embeddings are generated locally, and sensitive data never leaves the enterprise boundary.

MCP's ability to perform actions introduces risks that read-only RAG architectures don't face. MCP enables AI to act on behalf of a human, which creates exposure if the server is compromised or poorly configured. OWASP published an MCP-specific Top 10 in 2025:

Token mismanagement. Hard-coded credentials and long-lived tokens in MCP configurations can expose connected systems if compromised. The protocol specifies OAuth2 for authentication, but enforcement depends on implementation.

Context and prompt injection. In MCP ecosystems, prompt injection can trigger automated actions and a user-submitted message containing embedded instructions that an over-permissioned agent executes against production systems. Bloated context windows compound the problem: the model may act on irrelevant or conflicting information.

Data exfiltration and schema discovery. A malicious actor could craft prompts to extract sensitive data from connected databases, or map out the entire database schema to plan further attacks. Over-permissioned agents amplify both risks.

Recommended mitigations: MCP server allowlisting with cryptographic verification, least-privilege access controls, human approval gates for high-risk operations, sandbox environments, and audit logging.

For healthcare organizations, FHIR's existing authentication framework such as SMART on FHIR with OAuth2 provides a natural security layer for MCP-connected agents.

An AI agent accessing patient data through a FHIR server wrapped as an MCP tool inherits the FHIR server's authentication, authorization, and audit infrastructure.

Organizations running InterSystems HealthShare or InterSystems IRIS for Health already have FHIR-native infrastructure with HITRUST r2 certification, giving their AI agents an existing compliance foundation.

The most advanced AI systems combine RAG and MCP inside multi-agent architectures. Each LLM call becomes an agent with a distinct role. Not all agents are LLMs; some are tools. An orchestrator directs the workflow dynamically.

The agentic pattern gives AI the revision loop that single LLM calls lack.

In Don’s interview with Dr. Peter Lee of Microsoft Research, Lee described a healthcare orchestrator agent developed with Stanford Medicine for tumor board meetings.

The agent retrieves patient records (MCP), invokes radiology and pathology AI models (MCP), accesses published research from external data sources (RAG), and records meeting decisions (MCP). RAG and MCP, each doing what it's designed for.

Microsoft's MAI-DXO system extends this: multiple agents with distinct roles; a primary diagnostic agent, a contrarian agent that challenges every conclusion, a cost-conscious agent that questions every test order.

Through a "chain of debate," the system achieved 85.5% diagnostic accuracy on published NEJM case proceedings which is more than four times the rate of experienced physicians tested under the same conditions.

1. Audit your underlying data. RAG retrieves what's in your data sources. If your documentation is inconsistent or your policy documents contradict each other, RAG will surface those problems at scale. A unified data platform like InterSystems IRIS combines transactional, analytical, and vector search in a single engine which reduces the fragmentation that degrades retrieval quality.

2. Assess your API readiness. Your existing REST endpoints; including FHIR APIs for healthcare organizations are natural candidates for MCP server wrappers. If you already have well-documented internal APIs with OAuth2 authentication, you're closer to MCP readiness than you think.

3. Start with a RAG pilot. Customer chatbots, documentation search, and policy Q&A are proven entry points with bounded, well-maintained data sources.

4. Establish MCP governance before opening tool access. Allowlisting, access controls, human approval gates, and audit logging should be in place before agents interact with production systems. Joel Venco, CIO of Hartford Health, borrowed from Mario Andretti: "Really good brakes allow you to drive faster." Governance isn't a speed limiter — it's what lets you deploy with confidence.