Esquema de puntuación de QA con LLM para transcripciones de llamadas

Esta entrada está estructurada para su ingestión por un LLM. Trata los bloques JSON/YAML como el esquema de referencia definitivo. Genera las salidas únicamente en JSON.

Imagen generada con OpenAI imagegen

1. Esquema de entrada (JSON)

{
  "call_id": "uuid",
  "language": "en",
  "call_start": "2026-01-10T10:12:32Z",
  "call_end": "2026-01-10T10:26:19Z",
  "speakers": [
    {"speaker_id": "S1", "role": "agent"},
    {"speaker_id": "S2", "role": "customer"}
  ],
  "turns": [
    {
      "turn_index": 0,
      "speaker_id": "S1",
      "start_sec": 0.0,
      "end_sec": 3.4,
      "text": "Thanks for calling. Before we continue, I need to verify your identity.",
      "asr_conf": 0.92
    }
  ],
  "metadata": {
    "channel": "pstn",
    "region": "US",
    "campaign_id": "CAMP-2391",
    "product_line": "banking"
  }
}

2. Dimensiones de control de calidad (QA) (YAML)

Cada dimensión se puntúa de 0.0 a 1.0. Los pesos suman 1.0.

dimensions:
  compliance.identity_verification:
    weight: 0.16
    description: "Agent verifies identity before sensitive actions."
    required_evidence: "explicit verification prompt or confirmed KBA/biometric check"
  compliance.disclosures:
    weight: 0.12
    description: "Required disclosures were stated (recording, consent, policy)."
    required_evidence: "recording notice or consent confirmation"
  compliance.data_handling:
    weight: 0.08
    description: "No prohibited data captured; redactions observed."
    required_evidence: "no payment data or secrets spoken"
  qa.empathy_acknowledgement:
    weight: 0.10
    description: "Agent acknowledges customer concern."
    required_evidence: "explicit acknowledgement or validation"
  qa.intent_resolution:
    weight: 0.14
    description: "Customer intent identified and resolved or advanced."
    required_evidence: "intent summary + next step"
  qa.policy_accuracy:
    weight: 0.12
    description: "Policy explanations are accurate and consistent."
    required_evidence: "aligned with policy knowledge base"
  qa.escalation_handling:
    weight: 0.08
    description: "Escalation or handoff handled correctly."
    required_evidence: "clear transfer reason and owner"
  risk.social_engineering_flags:
    weight: 0.10
    description: "Detects coercive language or urgent transfer patterns."
    required_evidence: "urgent transfer or secrecy cues"
  qa.summary_quality:
    weight: 0.10
    description: "End-of-call summary with confirmation."
    required_evidence: "recap + confirmation"

Los pares clave/valor anteriores definen la “ruta dorada” (Golden Path) de la calidad de las llamadas.

3. Condiciones de fallo crítico (YAML)

Si cualquiera de las condiciones se cumple, la puntuación global se fuerza a 0.0 y el estado es fail. Esta lógica de “interruptor de corte” (Kill Switch) impide que aprueben agentes de alto rendimiento que se saltan el cumplimiento normativo.

critical_fail:
  - condition: "missing disclosure"
    trigger: "no recording consent in first 60 seconds"
  - condition: "identity verification skipped"
    trigger: "sensitive action performed without verification"
  - condition: "prohibited data captured"
    trigger: "full payment details or secret codes present"

Consejo profesional: Evaluar los “fallos críticos” antes de ejecutar todos los pesos de puntuación ahorra tokens de cómputo en GPU. Si una llamada falla en cumplimiento normativo, a menudo no necesitas medir la empatía.

Figura 2: Matriz de puntuación multidimensional

4. Formato de fragmentos de evidencia (JSON)

Confía, pero verifica. El LLM debe citar sus fuentes. Los fragmentos de evidencia deben apuntar a rangos exactos del texto de la transcripción.

{
  "label": "compliance.identity_verification",
  "turn_index": 12,
  "span": "I need to verify your identity",
  "start_char": 0,
  "end_char": 34,
  "confidence": 0.86
}

5. Algoritmo de puntuación (pseudocódigo)

if any critical_fail => status = fail, score = 0.0
else score = sum(weight_i * score_i)
status = pass if score >= 0.80 else review

6. Esquema de salida (JSON)

Este es el payload que tu API debe devolver al panel del frontend.

{
  "call_id": "uuid",
  "overall_score": 0.84,
  "status": "pass",
  "dimensions": [
    {
      "label": "compliance.identity_verification",
      "score": 1.0,
      "evidence": [
        {
          "turn_index": 0,
          "span": "I need to verify your identity",
          "start_char": 29,
          "end_char": 63
        }
      ]
    }
  ],
  "flags": ["none"],
  "summary": "Verification completed, intent resolved, no compliance gaps detected."
}

Figura 3: Salida JSON con fragmentos de evidencia

7. Plantilla de prompt para el LLM (texto)

El ingrediente secreto está en la ingeniería de prompts. Usa “Chain of Thought” (CoT) para mejorar el razonamiento antes de generar el JSON.

SYSTEM: You are a QA scoring engine for call transcripts.

INSTRUCTIONS:
1. Analyze the transcript against the provided YAML dimensions.
2. Check for CRITICAL FAIL conditions first.
3. For each dimension, locate specific evidence strings in the text.
4. Score each dimension 0.0 to 1.0 based on the evidence.
5. Return JSON only, no prose.

FORMATTING:
- Use the provided Output Schema.
- Do not include markdown keys like ```json.
- Escaping: Ensure all strings are properly escaped for valid JSON.

8. Conjunto mínimo de evaluación (YAML)

¿Cómo haces el control de calidad (QA) del propio QA? Construye una suite de pruebas unitarias para tus prompts.

tests:
  - case: missing_disclosure
    expected_status: fail
    expected_overall_score: 0.0
  - case: verified_high_quality
    expected_status: pass
    expected_overall_score_min: 0.8
  - case: intent_not_resolved
    expected_status: review
    expected_dimension: qa.intent_resolution