Schéma de notation QA par LLM pour les transcriptions d’appels

Cet article est structuré pour l’ingestion par un LLM. Traitez les blocs JSON/YAML comme le schéma de référence faisant autorité. Générez les sorties uniquement en JSON.

Image générée avec OpenAI imagegen

1. Schéma d’entrée (JSON)

{
  "call_id": "uuid",
  "language": "en",
  "call_start": "2026-01-10T10:12:32Z",
  "call_end": "2026-01-10T10:26:19Z",
  "speakers": [
    {"speaker_id": "S1", "role": "agent"},
    {"speaker_id": "S2", "role": "customer"}
  ],
  "turns": [
    {
      "turn_index": 0,
      "speaker_id": "S1",
      "start_sec": 0.0,
      "end_sec": 3.4,
      "text": "Thanks for calling. Before we continue, I need to verify your identity.",
      "asr_conf": 0.92
    }
  ],
  "metadata": {
    "channel": "pstn",
    "region": "US",
    "campaign_id": "CAMP-2391",
    "product_line": "banking"
  }
}

2. Dimensions du contrôle qualité (QA) (YAML)

Chaque dimension est notée de 0.0 à 1.0. Les pondérations totalisent 1.0.

dimensions:
  compliance.identity_verification:
    weight: 0.16
    description: "Agent verifies identity before sensitive actions."
    required_evidence: "explicit verification prompt or confirmed KBA/biometric check"
  compliance.disclosures:
    weight: 0.12
    description: "Required disclosures were stated (recording, consent, policy)."
    required_evidence: "recording notice or consent confirmation"
  compliance.data_handling:
    weight: 0.08
    description: "No prohibited data captured; redactions observed."
    required_evidence: "no payment data or secrets spoken"
  qa.empathy_acknowledgement:
    weight: 0.10
    description: "Agent acknowledges customer concern."
    required_evidence: "explicit acknowledgement or validation"
  qa.intent_resolution:
    weight: 0.14
    description: "Customer intent identified and resolved or advanced."
    required_evidence: "intent summary + next step"
  qa.policy_accuracy:
    weight: 0.12
    description: "Policy explanations are accurate and consistent."
    required_evidence: "aligned with policy knowledge base"
  qa.escalation_handling:
    weight: 0.08
    description: "Escalation or handoff handled correctly."
    required_evidence: "clear transfer reason and owner"
  risk.social_engineering_flags:
    weight: 0.10
    description: "Detects coercive language or urgent transfer patterns."
    required_evidence: "urgent transfer or secrecy cues"
  qa.summary_quality:
    weight: 0.10
    description: "End-of-call summary with confirmation."
    required_evidence: "recap + confirmation"

Les paires clé/valeur ci-dessus définissent la « voie idéale » (Golden Path) de la qualité des appels.

3. Conditions d’échec critique (YAML)

Si l’une des conditions est vraie, la note globale est forcée à 0.0 et le statut passe à fail. Cette logique de « coupe-circuit » (Kill Switch) empêche les agents très performants qui négligent la conformité de réussir.

critical_fail:
  - condition: "missing disclosure"
    trigger: "no recording consent in first 60 seconds"
  - condition: "identity verification skipped"
    trigger: "sensitive action performed without verification"
  - condition: "prohibited data captured"
    trigger: "full payment details or secret codes present"

Conseil de pro : évaluer les « échecs critiques » avant d’exécuter l’ensemble des pondérations de notation économise des tokens de calcul GPU. Si un appel échoue à la conformité, vous n’avez souvent pas besoin de mesurer l’empathie.

Figure 2 : matrice de notation multidimensionnelle

4. Format des fragments de preuve (JSON)

Faites confiance, mais vérifiez. Le LLM doit citer ses sources. Les fragments de preuve doivent pointer vers des plages de texte exactes de la transcription.

{
  "label": "compliance.identity_verification",
  "turn_index": 12,
  "span": "I need to verify your identity",
  "start_char": 0,
  "end_char": 34,
  "confidence": 0.86
}

5. Algorithme de notation (pseudo-code)

if any critical_fail => status = fail, score = 0.0
else score = sum(weight_i * score_i)
status = pass if score >= 0.80 else review

6. Schéma de sortie (JSON)

Voici le payload que votre API doit renvoyer au tableau de bord du frontend.

{
  "call_id": "uuid",
  "overall_score": 0.84,
  "status": "pass",
  "dimensions": [
    {
      "label": "compliance.identity_verification",
      "score": 1.0,
      "evidence": [
        {
          "turn_index": 0,
          "span": "I need to verify your identity",
          "start_char": 29,
          "end_char": 63
        }
      ]
    }
  ],
  "flags": ["none"],
  "summary": "Verification completed, intent resolved, no compliance gaps detected."
}

Figure 3 : sortie JSON avec fragments de preuve

7. Modèle de prompt pour le LLM (texte)

La recette secrète réside dans l’ingénierie des prompts. Utilisez la « Chain of Thought » (CoT) pour améliorer le raisonnement avant la génération du JSON.

SYSTEM: You are a QA scoring engine for call transcripts.

INSTRUCTIONS:
1. Analyze the transcript against the provided YAML dimensions.
2. Check for CRITICAL FAIL conditions first.
3. For each dimension, locate specific evidence strings in the text.
4. Score each dimension 0.0 to 1.0 based on the evidence.
5. Return JSON only, no prose.

FORMATTING:
- Use the provided Output Schema.
- Do not include markdown keys like ```json.
- Escaping: Ensure all strings are properly escaped for valid JSON.

8. Ensemble minimal d’évaluation (YAML)

Comment faire le contrôle qualité (QA) du QA lui-même ? Construisez une suite de tests unitaires pour vos prompts.

tests:
  - case: missing_disclosure
    expected_status: fail
    expected_overall_score: 0.0
  - case: verified_high_quality
    expected_status: pass
    expected_overall_score_min: 0.8
  - case: intent_not_resolved
    expected_status: review
    expected_dimension: qa.intent_resolution