AIT-RAG-01
Knowledge and retrieval design
Defines the governed corpus, ingestion process, chunking, metadata, and retrieval strategy for a professional use case.
Loading your workspace…
Loading your workspace…
AI & TECHNOLOGY · COMPLETE DRAFT · READY FOR REVIEW
A technical course for building source-grounded retrieval workflows and evaluating retrieval, answer support, citations, and abstention separately.
GOVERNANCE GATE
This version exposes the proposed sources, competencies, instruction, evidence requirements, and rubric for accountable review. It cannot be enrolled in, completed, or used to issue a credential until an authorized reviewer approves a final immutable version and the program is separately published.
Status: Complete draft awaiting accountable instructional review
Draft version: 1.0.0-draft.1
Estimated learning time: 210 minutes
Program level: Course
Evidence activities: 3
Credential pathway: Proposed Certificate of Completion after review, publication, learner evidence approval, and separate administrator issuance.
LEARN → APPLY → DEMONSTRATE → REVIEW → VERIFY
AIT-RAG-01
Defines the governed corpus, ingestion process, chunking, metadata, and retrieval strategy for a professional use case.
AIT-RAG-02
Evaluates retrieval relevance, answer correctness, evidential support, citation quality, and abstention.
AIT-RAG-03
Controls access, freshness, observability, failure handling, and correction across the RAG lifecycle.
CONTINUING PROFESSIONAL CASE
An organization wants an assistant to answer staff questions from policy manuals. The prototype cites outdated sections, misses exceptions split across pages, and sometimes answers from general model knowledge when retrieval is weak.
Constraint: Policies vary by role and location, update monthly, and include confidential appendices with restricted access.
Learner task: Design the corpus, ingestion, retrieval, grounded-answer contract, access controls, and evaluation set for a defensible RAG system.
FIVE SOURCE-GROUNDED LESSONS
LESSON 1
Purpose: Define what the system may know and claim.
RAG introduces an external knowledge layer that can be updated and inspected, but it does not guarantee that the right passage will be retrieved or faithfully used.
Professional example: The assistant answers policy questions only from approved, current documents and abstains outside that corpus.
Evidence connection: The original RAG work separates parametric memory from retrieved non-parametric memory and highlights provenance and updateability.
Define the policy assistant's authorized corpus, excluded knowledge, and claim boundary.
What does adding retrieval guarantee?
Expected: External evidence can be supplied, but retrieval and generation still require evaluation. — RAG creates an evidence path, not an automatic truth guarantee.
LESSON 2
Purpose: Prepare documents so relevant evidence remains retrievable and governed.
Parsing, chunk boundaries, overlap, metadata, versions, and access attributes shape what the retriever can find. Document meaning and policy structure should guide these choices.
Professional example: A policy section and its exceptions remain linked through headings and version metadata.
Evidence connection: Retrieval documentation exposes file processing, chunking, attributes, and filtering as configurable system layers.
Specify chunking, metadata, version, and access attributes for the policy manuals.
Why does chunking matter?
Expected: It affects whether complete, relevant evidence can be retrieved in context. — Chunking changes the retrievable unit and its meaning.
LESSON 3
Purpose: Find and order evidence relevant to the user's need.
Query formulation, semantic search, keyword signals, filters, top-k selection, and reranking affect recall and precision. Retrieval must be tested with real question variation.
Professional example: The system filters by location and employment role before ranking policy passages.
Evidence connection: Current retrieval systems combine vector similarity, attributes, ranking options, and query transformation.
Design retrieval for a question whose answer depends on both a main rule and an exception.
What should retrieval evaluation measure?
Expected: Whether relevant and sufficient passages appear at useful ranks. — Retrieval quality should be measured before generation can obscure it.
LESSON 4
Purpose: Constrain generation to supported claims.
A grounded-answer contract should require each consequential claim to be supported by retrieved evidence, preserve citations, expose conflicts, and abstain when evidence is insufficient.
Professional example: The assistant quotes the controlling section, cites its version, and flags a conflicting exception for human interpretation.
Evidence connection: Information-integrity guidance emphasizes provenance and support; RAG research treats evidentiality as a central challenge.
Write the policy assistant's support, citation, conflict, and abstention rules.
When is abstention appropriate?
Expected: When retrieved evidence is missing, conflicting, inaccessible, or insufficient for the claim. — A bounded system should decline unsupported claims.
LESSON 5
Purpose: Diagnose retrieval, generation, and corpus failures separately.
A useful evaluation set records expected sources and supported answers, then measures retrieval relevance, context sufficiency, answer correctness, citation support, abstention, access, and freshness.
Professional example: A test fails if the answer is correct but cites a superseded policy or restricted appendix.
Evidence connection: Evaluation guidance recommends task-specific datasets and continuous testing; NIST adds lifecycle monitoring and incident response.
Build a policy-question test matrix with expected passages, claims, access, and abstention outcomes.
How should a RAG failure be diagnosed?
Expected: Separate corpus, ingestion, retrieval, context, generation, and citation evidence. — Layered evaluation supports targeted correction.
ASSESSED APPLICATION
PROPOSED REVIEW RUBRIC
Problem framing and operational boundary is explicit, workable, and supported by relevant evidence.
Competency: AIT-RAG-01
Technical design and implementation rationale is explicit, workable, and supported by relevant evidence.
Competency: AIT-RAG-02
Evaluation design and preserved evidence is explicit, workable, and supported by relevant evidence.
Competency: AIT-RAG-03
Risk controls and accountable governance is explicit, workable, and supported by relevant evidence.
Competency: AIT-RAG-03
Failure analysis, revision, and professional judgment is explicit, workable, and supported by relevant evidence.
Competency: AIT-RAG-03
PRIMARY SOURCE BASE
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela · Advances in Neural Information Processing Systems 33 · 2020
Introduces the influential formulation combining parametric generation with retrieved non-parametric memory for knowledge-intensive tasks.
OpenAI · OpenAI API Documentation · Current documentation accessed August 2026
Provides current primary documentation for semantic search, vector stores, files, attributes, ranking, and retrieval configuration.
National Institute of Standards and Technology · NIST AI 600-1 · 2024
Provides a cross-sector framework for identifying, measuring, and managing generative-AI risks across the lifecycle.