Case Studies

AI Governance Framework

AI Quality Assurance Framework

Enterprise Search Docs Program

Game Design Documentation (GDD)

API Documentation Suites

Enterprise Search API

AI Quality Assurance Framework

A Human-in-the-Loop Methodology for Enterprise AI Systems

Context

This framework outlines my approach to evaluating AI-generated content across six dimensions, developed through hands-on work with AI tools in healthcare and enterprise knowledge management environments. It is informed by the NIST AI Risk Management Framework 1.0 (NIST AI RMF), specifically the MEASURE and MANAGE functions and the TEVV process (Testing, Evaluation, Verification, and Validation), adapted for human-in-the-loop content evaluation at the enterprise level.

Challenge

AI-generated content is being pushed into enterprise systems faster than quality standards can keep up with it. The tools for governance exist, however not all companies are in alignment with it. Alternatively, they may want to develop their own framework based on their own business needs.

Approach

Each output is assessed against all six dimensions before approval. Failures at dimensions one through four trigger revision or regeneration. Failures at dimension five trigger escalation regardless of content quality elsewhere. Dimension six failures return to editing without escalation unless combined with other flags.

This framework is designed to scale across a single reviewer or a distributed QA team with consistent results.

The Six Evaluation Dimensions

1. Relevance The first and fastest output to measure. Does the output answer the question that was actually asked? A technically accurate response that addresses the wrong intent fails the user before they finish reading. Evaluation starts here.

2. Completeness Does the response give the user what they need to act on it? Missing context, absent caveats, or incomplete steps create support burden and erode trust in the system over time.

3. Accuracy Is the information factually correct and verifiable against source material? AI systems present uncertain information with confident language. Evaluators need to apply skepticism when reading their content.

4. Hallucination Risk Is the model fabricating specifics such as citations, statistics, or procedural steps that cannot be verified? NIST AI RMF explicitly identifies hallucination as a distinct generative AI risk. This dimension requires active scrutiny, particularly when outputs reference policies, regulations, or technical specifications.

5. Governance and Compliance Flags Does the output contain PII, PHI, or other sensitive information inappropriate for the intended audience? Does it make regulatory or legal claims that require independent verification? In regulated environments, this dimension is not optional. It is the layer that separates content QA from basic proofreading.

6. Voice, Flow, and Audience Fit Is the content user-ready, not just technically correct? Grammatical errors, awkward sentence structure, and tone mismatches signal AI generation to end users and undermine confidence in the system. Users do not need to be subject matter experts to know when something sounds wrong.

Results

What does good evaluation practice actually produce? Fewer downstream errors, caught hallucinations before they reach end users, governance flags that prevent compliance exposure, content that users trust.

Skills Demonstrated

  1. Human-In-The-Loop AI Evaluation

  2. Content Governance

  3. Hallucination Detection

  4. HIPAA and PHI Risk Assessment

  5. Knowledge Quality Standards

  6. NIST AI RMF Alignment

  7. Enterprise Content QA

NIST AI RMF Framework

https://airc.nist.gov/airmf-resources/airmf/