Skip to content

Best RAG Frameworks Compared: Licenses, Web Loading and a 310-Page Benchmark

Retrieval-augmented generation (RAG) frameworks do the unglamorous work between your documents and a language model: load, split, embed, index, retrieve. This guide compares eleven options, explains which build each suits, and reports a dated benchmark run on 310 documentation pages, including a separate graph test for LightRAG and Microsoft GraphRAG. The headline finding is that framework choice barely moved answer accuracy; what you feed the index moved it a lot.

What a RAG framework actually does

A RAG framework is a library or server that runs the retrieval pipeline. It pulls documents from files, databases or web pages, breaks them into chunks, converts chunks to embeddings and writes them to a vector index. At question time it finds the closest chunks and hands them to a model alongside the question. Many frameworks add agents, evaluation hooks and tracing on top.

It is not the same thing as a vector database (Qdrant, Milvus and Weaviate store vectors; frameworks call them), an evaluation library (Ragas and DeepEval score pipelines rather than build them), or a rented hosted RAG service.

Do you need a framework at all?

For one data source and a fixed flow, plain Python with an embeddings model and a vector client is often the easiest thing to read, debug and keep alive. The calculus changes when you add a second source, a reranker, hybrid search, an agent that decides when to retrieve, production tracing or an evaluation loop. Each of those is a library call in a framework and a small project in plain Python. A framework is worth its overhead once you need a broad set of integrations, agent orchestration, evaluation or tracing and would otherwise have to assemble them yourself.

Eleven frameworks at a glance

The comparison below covers language, license, how each runs and whether it took part in the benchmark. Most are open-source libraries you install; a few run as servers. The benchmark column distinguishes the full five-pipeline test from the separate graph subset used for LightRAG and Microsoft GraphRAG.

How the benchmark was run

The same retrieve-and-answer pipeline was built in LangChain, LlamaIndex, Haystack and DSPy, plus a plain Python baseline. Pages, questions, model and vector store were held constant so the framework was the only variable. The corpus was 310 pages from four public documentation sites, crawled once on October 3, 2026 and then frozen. Each page was stored three ways: raw HTML from a plain GET, markdown as returned by the scraping API, and that markdown with navigation and footers stripped by a short cleaner script.

The question set had 100 items, each with a reference answer and the URL of the page holding it. Fifty-seven were written by hand after reading the frozen pages; Ragas 0.4.3 generated 60 candidates, of which 43 were kept and 17 rejected as duplicates, glued-together multi-hop items or unparseable. Tags: 86 single-hop, 14 multi-hop, 28 JavaScript-only. Every pipeline used Claude Haiku 4.5 at temperature 0 with 512 max output tokens, BAAI/bge-small-en-v1.5 embeddings run locally, Qdrant 1.19.1 with cosine distance, 800-character chunks with 100 characters of overlap, and top-k 5. Each question ran three times, scheduled round-robin so API latency drift hit every framework equally.

LightRAG and Microsoft GraphRAG each construct their own graph index, which is why they were evaluated apart from the five-pipeline test: a 100-page subset, 40 questions, and each tool's own retrieval defaults (LightRAG mix mode, GraphRAG local search). Measured dimensions were code size (pygount, shared runner excluded), package footprint, cold start, time to first answer, p50 and p95 latency over 300 queries, tokens and dollars per query from a logging proxy, and Ragas faithfulness, context precision, context recall and answer correctness judged by Haiku 4.5 on run 1, plus retrieval hit rate and mean reciprocal rank against the gold URLs. The whole benchmark, including test-set generation, judging and both graph pipelines, cost about $55 in Anthropic API calls plus 554 scraping credits for the crawl.

What the corpus contained

Four documentation sites made up the corpus. Two are pre-rendered at build time, so a plain HTTP request returns the text; two are Docsify sites that render in the browser, and those 61 pages are the JavaScript-only slice, roughly a fifth of the total. The table records how much text a default loader extracted from each site, which is the number that predicts whether you need rendering.

Benchmark results: the framework was not the variable

On the cleaned corpus, the five pipelines produced statistically indistinguishable answers, ran within a fraction of a second of each other, and, DSPy aside, cost the same per query. A paired bootstrap test against the plain Python baseline put every framework's answer correctness inside the same band (0.631 to 0.641), so no framework separated itself from the baseline.

Latency was dominated by the model. Work outside the model call (embedding the question, searching Qdrant, the framework's own code) took 63 to 67 ms at the median in every pipeline; about 97% of each query was the model. Haystack's context precision was 0.09 higher than plain Python's, possibly because its splitter falls back to sentence boundaries, though LlamaIndex's splitter is sentence-aware too and scored lowest, so that is a partial explanation at best; across 16 comparisons, one fluke is what chance alone would produce. DSPy cost about 20% more per query because its prompt format adds about 300 input tokens to every call.

This is a strong signal for retrieve-and-answer apps on English docs with one model, not a verdict on agents or tracing. On a retrieve-and-answer job, the framework determines the amount of code you maintain and the blast radius when a dependency changes; the model determines the answer.

Ingestion mattered more than framework choice

Rendering was the step that changed outcomes. Cleaned markdown beat default-loader text by 0.14 in answer correctness (95% confidence interval +0.09 to +0.19). On the 28 JavaScript-only questions, every default-corpus answer was some version of "I don't know". Pages whose text is already in the HTML scored nearly the same under either ingestion route (0.608 against 0.648).

Cleaning bought a smaller index, not better answers: stripping navigation and footers cut chunk count by 22 to 24% while correctness held (0.633 against 0.637). If your sources include client-rendered pages, the rendering step is worth more than any framework switch, and cleanup pays for itself in a smaller, cheaper index.

Where plain Python matched the frameworks

The 47-line plain Python pipeline (sentence-transformers, qdrant-client and the Anthropic SDK) matched every framework on answer correctness and latency, with the smallest install: 60 packages against 85 to 110. It took 13 more lines than LangChain, which is the price of writing your own splitter. It also had the least surface area to break. Anthropic's Python SDK 1.0 dropped the temperature argument, and the plain pipeline absorbed that with a single line (extra_body); LlamaIndex, Haystack and LightRAG instead had to hold the SDK below 1.0 until their integrations caught up.

The eleven frameworks in detail

Each entry below follows the same order: who it is for, language and license, GitHub stars as checked on October 3, 2026, what it does well for RAG, how it loads web pages, what to watch out for and, where benchmarked, how it did. Star counts are a dated snapshot and will have moved.

LangChain and LangGraph (Python, JS/TS; MIT; 147K+ stars). The integration hub of the Python and JavaScript AI world, with LangGraph as the agent runtime, which makes it the default agentic RAG choice. Version 1.0 moved old chains, agents, memory and retrievers into langchain-classic; the new core is create_agent plus middleware (human approval, summarization, PII redaction) on LangGraph. LangSmith, the paid hosted product, adds tracing and evaluation. Built-in loaders (WebBaseLoader, RecursiveUrlLoader, SitemapLoader) fetch raw HTML without JavaScript; Playwright and Selenium loaders drive a browser you install. Watch for old tutorials: 2023 and 2024 code breaks on 1.x imports. CVE-2025-68664 hit serialization in langchain-core, so stay on 0.3.81, 1.2.5 or later. Benchmark: 34 lines, the fewest of the five. Correctness 0.641 and p50 1.78 s both fall within the tie band, and langchain-anthropic 1.7.5 ran against the current Anthropic SDK with no changes.

LlamaIndex (Python, TypeScript; MIT core; 52K+ stars). The framework that cares most about what happens before retrieval: loading, parsing, chunking and indexing. It offers the shortest path from documents to a query engine, with node parsers, many index types and Workflows for multi-step logic. The company now concentrates on LlamaParse, its hosted parser for hard documents, priced at 10,000 free credits a month then $1.25 per 1,000 at the time of writing. SimpleWebPageReader (requests plus html2text) and TrafilaturaWebReader do not run JavaScript; WholeSiteReader drives Selenium; llama-index-readers-web has readers for several scraping services. Two catches: llama-index-readers-web is GPL-3.0-or-later, which matters commercially, and llama-index-llms-anthropic 0.12.2 broke against Anthropic SDK 1.x, with the query engine catching the error and returning the string "Empty Response" instead of raising. Pin anthropic<1 until the integration catches up and assert against that string in tests. Benchmark: best p95 latency (2.52 s) and MRR (0.77), correctness 0.631 (a tie), and the heaviest install at 110 packages with a 2.2-second framework import.

Haystack (Python; Apache 2.0; maintained by Deepset, which also sells a managed platform; 26K+ stars). RAG from explicit, typed components wired into pipelines, making it the easiest of the big three to test and reason about. Pipelines are graphs of components with declared inputs and outputs, so a wrong connection fails at build time rather than in production; hybrid retrieval, evaluators and OpenTelemetry tracing are built in. Web loading uses LinkContentFetcher plus HTMLToDocument over plain HTTP without JavaScript; on the Docsify page the pair produced a document with no content. Haystack 3.x is not backward compatible with 2.x: AsyncPipeline has been removed (async now lives in Pipeline), auto-generated document IDs changed so documents must be re-ingested, and tracing stays off until you enable it. Sentence Transformers embedders moved out of core into sentence-transformers-haystack, and RecursiveDocumentSplitter needs nltk installed separately. anthropic-haystack 6.2.0 and 6.3.0 hit the same Anthropic SDK 1.x error as LlamaIndex but raised it loudly, which is the behaviour you want. Benchmark: 48 lines; the only quality difference surviving the significance test was context precision (0.605 vs 0.515 for plain Python), possibly from its sentence-aware splitter, though LlamaIndex's is sentence-aware too and scored lowest; correctness 0.641, p95 2.96 s. A rerun on 3.3.0 matched (0.634 correctness, 0.596 precision).

DSPy (Python; MIT; from Stanford NLP; 38K+ stars). Treats a RAG pipeline as a program whose prompts are optimized against a metric rather than edited by hand. You write modules with typed signatures (context, question -> answer) and optimizers such as GEPA, MIPROv2 and SIMBA rewrite the prompts against a metric you define. The short comparison: LangChain gives you building blocks and integrations; DSPy tunes the program you build from them. It has no loaders, splitters or vector store clients, shipping only a small in-memory Embeddings retriever, so you bring the rest. In DSPy 3.4.0, import dspy installs a lazy-import proxy for numpy, and importing qdrant_client afterwards crashed inside numpy with "data type 'bool' not understood"; importing numpy before dspy fixed it, and a similar onnxruntime report is open upstream. Alphabetical import sorting puts dspy first, so expect to meet this. Benchmark: 53 lines because the splitter and Qdrant calls were written by hand; about 300 more input tokens per query (20% higher cost) and a 4.4-second first answer in a fresh process; correctness 0.641, a tie. No optimizer was run, so this is DSPy without the feature it exists for.

RAGFlow (Python and Go; Apache 2.0; from InfiniFlow; runs with Docker; 91K+ stars). A self-hosted RAG engine you run as a server with a web UI, not a library you import. Its standout is DeepDoc, a layout-aware parser that reads tables and scanned pages text-only loaders miss; around it you get agents, data connectors, an HTTP API and an MCP server behind one UI that non-developers can use. It accepts web pages as a document type and recently added sitemap-based website ingestion; the README does not say it runs JavaScript, so render client-rendered docs first and upload the markdown. It ships breaking changes often: agents built before v0.20.0 must be rebuilt, and from v0.22.0 Docker images no longer bundle embedding models. v0.27.2 (September 10, 2026) is the stable line; v1.0.0-rc1 (September 29, 2026) rewrites the server in Go, swaps Redis for NATS, and migrates data in a way you cannot roll back. Wait for 1.0 to leave release candidate before betting production on it.

LightRAG (Python; MIT; from the HKU Data Intelligence Lab; 39K+ stars). Builds a knowledge graph and a vector index side by side at a fraction of Microsoft GraphRAG's cost; the paper appeared in Findings of EMNLP 2025. It extracts entities and relations per chunk, merges them into a graph, and mixes graph context with vector chunks at query time. Documents can be added without rebuilding the whole index, and lightrag-server ships a web UI. It has no web loader, so rendered markdown goes straight in. Known open issue with duplicate entities under several names (#1323), and it is maintained by a university lab rather than a company. Its Anthropic adapter passes temperature straight to the SDK, so with Anthropic SDK 1.x every extraction failed until anthropic<1 was pinned, and failed queries came back as empty answers rather than errors. It de-duplicates documents by the last path segment, so pnpm.io/python and uv's .../policies/python/ collided and one was silently dropped; give each document a unique name. Benchmark (graph subset): indexing cost $5.98, each query read about 28,000 input tokens, context recall 0.973 against 0.716 for plain Python, correctness 0.672 against 0.709 (a tie), p50 6.48 s. Against GraphRAG on the same 100 pages: GraphRAG cost $10.53 to index, LightRAG 57% as much, with slightly higher context recall (0.973 vs 0.939) and correctness close to GraphRAG's (0.672 vs 0.657).

Microsoft GraphRAG (Python; MIT; 36K+ stars). Builds a knowledge graph, summarizes the communities it finds, then answers corpus-wide questions from those summaries. Global search answers from the community reports, which lets it handle "what are the main themes" questions that top-k vector retrieval cannot; local search targets specific questions and was the mode used in the benchmark. No web loader: it reads files or a DataFrame, so feed it rendered markdown. The README says the project "is largely in maintenance mode, and won't be accepting new PRs or implementing new features", so treat it as a reference implementation. Release 3.3.0 (October 8, 2026) still needs Python below 3.14. Defaults assume OpenAI: vector size had to change from 3,072 to 384 for a local embedding model, and default entity types (organization, person, geo, event) suit news more than software docs. Indexing is the expensive part; Microsoft's own LazyGraphRAG research reported indexing costs at 0.1% of full GraphRAG. Benchmark (graph subset): indexing cost $10.53 across 1,715 LLM calls, p50 5.80 s, context recall 0.939, correctness 0.657, slightly below plain Python's 0.709.

Cognee (Python; Apache 2.0 core; 31K+ stars). Turns documents and conversations into a knowledge graph that agents query as long-term memory. It extracts entities and relationships as it stores content and lets agents recall them across sessions; its API now reads like memory rather than retrieval (remember, recall, improve, forget), which tells you who it is built for. The core ships without a web loader, and source connectors sit in a separate integrations repo, so the dependable route is to hand it rendered markdown. Its documentation flags Postgres as a graph store as a demo-grade feature, with a production-ready build sold under a commercial licence, so a plan to keep everything in one Postgres should include that cost.

txtai (Python; Apache 2.0; from NeuML; 12K+ stars). A single package bundling an embeddings database with RAG, agent and workflow pipelines. One install gives dense and sparse vector indexes, a graph, SQL over embeddings and a RAG pipeline, with web and MCP APIs on top; for a small team that wants one dependency instead of five, the pitch holds. Its Textractor pipeline converts HTML to markdown with static parsing (Apache Tika handles other file types), with no browser in the loop, so JavaScript-rendered pages need rendering first. The ecosystem is much smaller than the top three's: when you hit an edge case, you are more likely to read the source than find a Stack Overflow answer.

Mastra and the Vercel AI SDK (TypeScript; Mastra Apache 2.0 except ee/ directories under the Mastra Enterprise License, free for development and licensed for production; AI SDK Apache 2.0; Mastra 28K+, Vercel AI 27K+ stars). The two TypeScript options worth your time, at different heights. Mastra ships agents, workflows and RAG ready to use: MDocument with recursive and sliding-window chunking, embedMany, and vector store adapters for pgvector, Pinecone, Qdrant, MongoDB and others. The AI SDK is a model and provider layer with streaming UI helpers; it is not a retrieval framework, so RAG is something you build on top. Neither renders JavaScript: fetch rendered markdown and pass it in as text. Mastra places its enterprise features (auth, the agent builder and the editor) in ee/ directories, so audit which of them you rely on before shipping. Pick Mastra for ready-made RAG and agents, the AI SDK for thin building blocks and full control.

Microsoft Agent Framework (.NET and Python; MIT; 13K+ stars; a Go SDK is in public preview and its docs say RAG is not available there yet). Microsoft's agent SDK for .NET and Python, and the direct successor to Semantic Kernel and AutoGen. It reached GA with version 1.0 in April 2026 (PyPI dates agent-framework 1.0.0 to April 2, 2026) and has already moved to 1.21.0. Retrieval arrives through context providers and connectors rather than a dedicated retrieval stack, alongside graph-based workflows, middleware, telemetry and MCP clients; if your company already runs on Azure, the integrations are the point. It does not load web pages itself: a tool or MCP server fetches them and the framework orchestrates the agent. If you are still on Semantic Kernel, plan the move: Semantic Kernel receives critical fixes for at least a year past Agent Framework's GA, yet new features land only in Agent Framework, so start new projects there.

Choosing by what you are building

Because the five pipelines landed within 0.01 of each other on correctness, the deciding factors are everything around the retrieve-and-answer job: integrations, agents, document parsing and the language your team already writes. That is also the honest answer to LlamaIndex vs LangChain or Haystack vs LangChain: all three tied on answer quality here, so choose on design.

  • An agent that calls tools and needs many integrations: LangChain with LangGraph.
  • Lots of PDFs and documents to parse and index: LlamaIndex, with LlamaParse for the hard files.
  • Explicit pipelines you want to test and trace: Haystack.
  • Prompts you want to tune against a metric: DSPy on top of your retriever.
  • A self-hosted app with a UI: RAGFlow.
  • Questions across a whole corpus: LightRAG first, Microsoft GraphRAG if you need its global search.
  • Memory for agents: Cognee.
  • TypeScript: Mastra or the Vercel AI SDK.
  • .NET or Azure: Microsoft Agent Framework.
  • A small app with stable requirements: plain Python.

When weighing LangChain alternatives or LlamaIndex alternatives, let the reason for switching pick the successor. A simple flow that feels buried under LangChain's abstractions will read more clearly as Haystack's explicit pipelines or as plain Python. If prompt tuning by hand is the pain, DSPy changes how you work rather than the plumbing underneath. And if LlamaIndex's ingestion is serving you well while agents have become the bottleneck, LangGraph is the logical place to go next.

Getting web pages into the index

The default web loaders in most frameworks pull raw HTML and never execute JavaScript, so any page assembled in the browser arrives blank or partly filled. Open-source crawlers such as Crawl4AI close that gap by running a browser you host yourself. The dependable sequence is to render the page, turn it into markdown, remove navigation and footers, and attach the source URL to every chunk before embedding.

Every loader in the table below was run against two pages: a Docsify page that renders in the browser and a server-rendered pnpm page. Not one of the loaders bundled with these frameworks retrieved the Docsify page, and the browser-driven options require you to run Chromium. Plain HTTP fails in a second way as well: minutes after the test crawl (roughly 170 plain requests from a single IP within an hour, so a rate limit that was deserved), pnpm.io began replying to plain requests with a "Vercel Security Checkpoint" page returned as HTTP 200, and WebBaseLoader and SimpleWebPageReader ingested it as though it were documentation without raising an error.

Step 1: check whether your loader runs JavaScript

Before indexing a site, load one page with your framework's default loader and look at the length. The check that produced the first row of the loader table needs pip install langchain-community beautifulsoup4, since LangChain 1.x no longer bundles the community loaders. The page in a browser is a full quick-start guide; the loader saw 7 characters, the word "docsify" from the title. The loader ignores everything after #, so every page on a hash-routed site maps to the same URL, and that URL holds an empty shell that JavaScript fills in later. If the number you get is tiny, or the same for every page, you need rendering.

Step 2: render pages to markdown through the HTML API

The HTML API loads the page in a real browser and can hand back markdown rather than HTML. Send credentials in the Authorization: Bearer header (the api_key query parameter is deprecated) and take the credit cost from the Spb-cost response header; that header matched the true cost on every call made, it is listed under Response Headers in the API docs, and the dashboard's usage page remains the official running total. The same page that gave the loader 7 characters returned 4,633 characters for 5 credits.

Enable render_js only where it is actually required. Against the server-rendered pnpm docs, render_js=false delivered the clean article for 1 credit; switching rendering on made the markdown extractor grab the sidebar rather than the article, at a cost of 5. For the benchmark corpus, the 61 Docsify pages were rendered and the other 249 fetched without a browser, for 554 credits across 310 pages.

Step 3: wrap the markdown in LlamaIndex and Haystack documents

LlamaIndex and Haystack each lack a built-in scraping component, and neither requires one. Both document types accept text plus a metadata dictionary, so the markdown can be passed in directly. Keep the URL and fetch time in the metadata so answers can cite the page and you know when to refresh it. Both objects then carry the same markdown and metadata, and VectorStoreIndex.from_documents([llama_doc]) or a Haystack indexing pipeline takes over exactly as in the benchmark code.

Step 4: give LangChain agents live pages

When an agent needs to decide mid-run which page to consult, the vendor's LangChain integration package turns the scraping APIs into LangChain tools. Version compatibility deserves a check first: release 1.0.0 still pins langchain<0.4, so under LangChain 1.x the practical route is pip install --no-deps alongside langchain, langchain-anthropic and requests, with ANTHROPIC_API_KEY and the vendor's API key environment variable exported. The tool is imported from langchain.tools, which 1.x continues to expose, and it behaved correctly in a test on 1.4.3: the agent rendered the Docsify page, read it, and reported that docsify serve listens on port 3000. Keep in mind that ScrapeUrlTool writes each result into a local scraping_results/ folder; unless return_content=true is passed, the agent receives a confirmation that the page was saved rather than the page text itself, which is the reason the system prompt requests that flag. Treat live fetching as a complement to an index, not a substitute, since every question answered this way incurs a fresh fetch.

Step 5: crawl a whole docs site to markdown chunks

When you want a complete index, the CLI can crawl from a sitemap and emit one markdown file per page together with a manifest.json that records, for each URL, its file, fetch time, status and credits consumed. The --exclude-pattern option came out of an early run: without it, the crawler also followed six images linked from the pages, wrote each as an empty .md file and billed a credit for every one, so 84 pages cost 90 credits; adding the pattern brought the run to 84 credits for 84 pages. If you would rather produce embedding-sized pieces directly, scrape accepts a URL list and performs the chunking itself, emitting one JSON object per chunk with the source URL included. Expect the opening chunk to be the page's navigation menu: the CLI reproduces the page's markdown faithfully, menus and all, which is exactly why a cleanup stage belongs in the pipeline. In the ingestion test, cleanup shrank the index by 22 to 24% with no loss in retrieval, while rendering raised answer correctness from 0.50 to 0.64.

Why clean web data is the real prerequisite

Every framework discussed here presumes that clean text is what arrives, yet the open web seldom cooperates: default loaders produced empty or title-only output for roughly a fifth of the corpus, and one site began returning a bot check partway through testing. The work that has to happen before a framework ever touches a page includes JavaScript rendering (with wait_for for slow pages), markdown output suitable for chunking and embedding, whole-site crawls that produce a URL manifest or split content into chunks, agent tools for fetching at question time, and an MCP server for MCP clients. The benchmark results make the case: rendering shifted correctness further than any framework swap did, and cleanup repaid its cost through a smaller index.

The eleven RAG frameworks compared
FrameworkBest forLanguageLicenseRuns asBenchmarked
LangChain and LangGraphAgentic RAG and apps that need the widest set of integrationsPython, JS/TSMITLibraryYes
LlamaIndexDocument-heavy apps where most of the work is loading, parsing and indexingPython, TSMIT (core)LibraryYes
HaystackExplicit, typed production pipelines that are easy to test and tracePythonApache 2.0LibraryYes
DSPyTeams that want to tune a RAG program against a metric instead of hand-editing promptsPythonMITLibraryYes
RAGFlowA self-hosted RAG engine with a web UI, especially for messy documents such as scanned PDFs and tablesPython, GoApache 2.0ServerNo
LightRAGGraph-based RAG at a much lower cost than full GraphRAG, with incremental updatesPythonMITBothYes (graph subset)
Microsoft GraphRAGBroad questions across a whole corpus, such as themes or trends, where plain vector search strugglesPythonMITLibraryYes (graph subset)
CogneeGraph-based long-term memory for agentsPythonApache 2.0 (core)LibraryNo
txtaiA lightweight, all-in-one Python packagePythonApache 2.0BothNo
Mastra and the Vercel AI SDKTypeScript teams building RAG or agentsTypeScriptApache 2.0 (Mastra ee/ excepted)LibraryNo
Microsoft Agent Framework.NET and Azure teams building agents with retrieval.NET, PythonMITLibraryNo

Licenses and star counts were checked on October 3, 2026; verify current licence terms and versions before adopting. The "Benchmarked" column distinguishes the full five-pipeline test from the separate graph subset used for LightRAG and Microsoft GraphRAG.

Pipeline versions used in the benchmark
PipelineKey packages
Plain Pythonanthropic 1.11.0, qdrant-client 1.19.1, sentence-transformers 6.1.0
LangChainlangchain 1.4.3, langchain-core 1.6.6, langchain-anthropic 1.7.5, langchain-qdrant 1.1.0, langchain-huggingface 1.2.2, langchain-text-splitters 1.1.2
LlamaIndexllama-index-core 0.14.25, llama-index-llms-anthropic 0.12.2, llama-index-embeddings-huggingface 0.8.0, llama-index-vector-stores-qdrant 0.10.3, anthropic 0.125.0
Haystackhaystack-ai 3.2.0 (rerun on 3.3.0), anthropic-haystack 6.2.0 (6.3.0), qdrant-haystack 11.0.0, sentence-transformers-haystack 0.1.2 (0.2.0), anthropic 0.125.0
DSPydspy 3.4.0, litellm 1.103.2, qdrant-client 1.19.1
LightRAGlightrag-hku 1.5.7, anthropic 0.125.0
Microsoft GraphRAGgraphrag 3.2.0, litellm 1.100.1
Evaluationragas 0.4.3 (with langchain-community 0.3.31)

Each pipeline has its own uv.lock. Run on October 3, 2026, on Python 3.12.15 and an Apple Silicon Mac.

Built-in web loaders against a JavaScript page
FrameworkBuilt-in web loaderRuns JavaScriptWhat came back from the JavaScript page
LangChainWebBaseLoaderNo7 characters: the word "docsify" from the <title>
LangChainSitemapLoaderNoNot run separately: it subclasses WebBaseLoader and fetches the same way
LangChainRecursiveUrlLoaderNoThe empty app shell as raw HTML, plus a 404 page it reached by following ${url} out of a JavaScript template string
LangChainAsyncChromiumLoader, PlaywrightURLLoader, SeleniumURLLoaderYes (drives a browser you install)Not run: needs a local browser
LlamaIndexSimpleWebPageReader, TrafilaturaWebReaderNo0 characters
LlamaIndexWholeSiteReaderYes (Selenium)Not run: needs a local browser
HaystackLinkContentFetcher + HTMLToDocumentNoA document with content=None
DSPyNonen/an/a

Each loader was run against two pages: a Docsify page that renders in the browser and a server-rendered pnpm page. Raw output is in the benchmark repo.

Frequently asked questions

Is ChatGPT a RAG LLM?

ChatGPT is a chat app built on OpenAI's models, not a RAG framework. When it searches the web or reads files you upload, it retrieves outside content before answering, which is RAG in practice. With those tools off, it answers only from what the model learned during training.

What is the difference between RAG and an LLM?

A large language model produces text by drawing on patterns absorbed during training. RAG is an arrangement built around such a model: relevant documents are fetched first, inserted into the prompt, and only then is the answer generated. This arrangement allows the model to respond using up-to-date or proprietary information and to point to its sources, all without retraining.

Can you use LangChain and LlamaIndex together?

Yes, and it is a common pairing. LlamaIndex loads, parses and indexes the documents, and a LangGraph agent calls that index as a retriever or tool. Keep ownership clear: one framework handles ingestion and retrieval, the other runs the agent loop, so you debug one set of abstractions at a time.

Which RAG frameworks are free for commercial use?

Nearly all of them. LangChain, LlamaIndex core, DSPy, LightRAG, Microsoft GraphRAG and Microsoft Agent Framework are MIT, and Haystack, RAGFlow, txtai and Cognee's core are Apache 2.0. Check three exceptions: Mastra's ee/ directories, Cognee's production Postgres graph store, and the GPL-licensed llama-index-readers-web.

Do long-context models make RAG obsolete?

No. Models can read more per request, but the evidence does not crown a winner: the LaRA benchmark (ICML 2025, 2,326 cases, 11 LLMs) found the better choice depends on the model, context length and task. Retrieval also keeps token bills down and makes citing sources straightforward.

How do you evaluate a RAG pipeline?

Establish a fixed set of questions, each paired with a reference answer and the page that supports it, and then assess retrieval and answer generation as two separate concerns. Ragas and DeepEval cover faithfulness, context precision and context recall, while verifying that the correct page was retrieved requires no language model at all. Repeat the same evaluation whenever chunking, embeddings or prompts change.

The 310-page corpus: four documentation sites
SiteBuilt withRenderedPagesText a default loader gets (median, whitespace collapsed)
uv docsMkDocs MaterialAt build time845,392 characters
pnpm docs (English, current version)DocusaurusAt build time1652,521 characters
go-echarts docsDocsifyIn the browser3922 characters
docsify.js.orgDocsifyIn the browser227 characters

Crawled once on October 3, 2026 and then frozen. The two Docsify sites form the JavaScript-only slice: 61 pages, about a fifth of the corpus.