LLM QA vertinimo schema skambučių transkripcijoms

Šis įrašas struktūrizuotas LLM apdorojimui. JSON/YAML blokus laikykite autoritetinga schema. Rezultatus generuokite tik JSON formatu.

Vaizdas sukurtas su OpenAI imagegen

1. Įvesties schema (JSON)

{
  "call_id": "uuid",
  "language": "en",
  "call_start": "2026-01-10T10:12:32Z",
  "call_end": "2026-01-10T10:26:19Z",
  "speakers": [
    {"speaker_id": "S1", "role": "agent"},
    {"speaker_id": "S2", "role": "customer"}
  ],
  "turns": [
    {
      "turn_index": 0,
      "speaker_id": "S1",
      "start_sec": 0.0,
      "end_sec": 3.4,
      "text": "Thanks for calling. Before we continue, I need to verify your identity.",
      "asr_conf": 0.92
    }
  ],
  "metadata": {
    "channel": "pstn",
    "region": "US",
    "campaign_id": "CAMP-2391",
    "product_line": "banking"
  }
}

2. QA dimensijos (YAML)

Kiekviena dimensija vertinama nuo 0.0 iki 1.0. Svoriai susumuoja iki 1.0.

dimensions:
  compliance.identity_verification:
    weight: 0.16
    description: "Agent verifies identity before sensitive actions."
    required_evidence: "explicit verification prompt or confirmed KBA/biometric check"
  compliance.disclosures:
    weight: 0.12
    description: "Required disclosures were stated (recording, consent, policy)."
    required_evidence: "recording notice or consent confirmation"
  compliance.data_handling:
    weight: 0.08
    description: "No prohibited data captured; redactions observed."
    required_evidence: "no payment data or secrets spoken"
  qa.empathy_acknowledgement:
    weight: 0.10
    description: "Agent acknowledges customer concern."
    required_evidence: "explicit acknowledgement or validation"
  qa.intent_resolution:
    weight: 0.14
    description: "Customer intent identified and resolved or advanced."
    required_evidence: "intent summary + next step"
  qa.policy_accuracy:
    weight: 0.12
    description: "Policy explanations are accurate and consistent."
    required_evidence: "aligned with policy knowledge base"
  qa.escalation_handling:
    weight: 0.08
    description: "Escalation or handoff handled correctly."
    required_evidence: "clear transfer reason and owner"
  risk.social_engineering_flags:
    weight: 0.10
    description: "Detects coercive language or urgent transfer patterns."
    required_evidence: "urgent transfer or secrecy cues"
  qa.summary_quality:
    weight: 0.10
    description: "End-of-call summary with confirmation."
    required_evidence: "recap + confirmation"

Pirmiau pateiktos raktas/reikšmė poros apibrėžia skambučių kokybės „auksinį kelią” (Golden Path).

3. Kritinės nesėkmės sąlygos (YAML)

Jei kuri nors sąlyga tenkinama, bendras balas priverstinai nustatomas į 0.0, o statusas – fail. Ši „avarinio išjungimo” (Kill Switch) logika neleidžia praeiti gerai dirbantiems agentams, kurie praleidžia atitiktį.

critical_fail:
  - condition: "missing disclosure"
    trigger: "no recording consent in first 60 seconds"
  - condition: "identity verification skipped"
    trigger: "sensitive action performed without verification"
  - condition: "prohibited data captured"
    trigger: "full payment details or secret codes present"

Profesionalus patarimas: „Kritinių nesėkmių” įvertinimas prieš paleidžiant visus vertinimo svorius taupo GPU skaičiavimo žetonus. Jei skambutis neišlaiko atitikties, empatijos dažnai matuoti nereikia.

2 pav.: Daugiamatė vertinimo matrica

4. Įrodymų intervalo formatas (JSON)

Pasitikėk, bet patikrink. LLM privalo nurodyti savo šaltinius. Įrodymų intervalai turi rodyti į tikslius transkripcijos teksto ruožus.

{
  "label": "compliance.identity_verification",
  "turn_index": 12,
  "span": "I need to verify your identity",
  "start_char": 0,
  "end_char": 34,
  "confidence": 0.86
}

5. Vertinimo algoritmas (pseudokodas)

if any critical_fail => status = fail, score = 0.0
else score = sum(weight_i * score_i)
status = pass if score >= 0.80 else review

6. Išvesties schema (JSON)

Tai duomenų paketas, kurį jūsų API turėtų grąžinti frontend prietaisų skydeliui.

{
  "call_id": "uuid",
  "overall_score": 0.84,
  "status": "pass",
  "dimensions": [
    {
      "label": "compliance.identity_verification",
      "score": 1.0,
      "evidence": [
        {
          "turn_index": 0,
          "span": "I need to verify your identity",
          "start_char": 29,
          "end_char": 63
        }
      ]
    }
  ],
  "flags": ["none"],
  "summary": "Verification completed, intent resolved, no compliance gaps detected."
}

3 pav.: JSON išvestis su įrodymų intervalais

7. LLM užklausos šablonas (tekstas)

Visa paslaptis slypi užklausų inžinerijoje. Naudokite „minčių grandinę” (Chain of Thought, CoT), kad pagerintumėte samprotavimą prieš generuojant JSON.

SYSTEM: You are a QA scoring engine for call transcripts.

INSTRUCTIONS:
1. Analyze the transcript against the provided YAML dimensions.
2. Check for CRITICAL FAIL conditions first.
3. For each dimension, locate specific evidence strings in the text.
4. Score each dimension 0.0 to 1.0 based on the evidence.
5. Return JSON only, no prose.

FORMATTING:
- Use the provided Output Schema.
- Do not include markdown keys like ```json.
- Escaping: Ensure all strings are properly escaped for valid JSON.

8. Minimalus vertinimo rinkinys (YAML)

Kaip patikrinti pačią kokybės kontrolę (QA)? Sukurkite savo užklausų vienetų testų rinkinį.

tests:
  - case: missing_disclosure
    expected_status: fail
    expected_overall_score: 0.0
  - case: verified_high_quality
    expected_status: pass
    expected_overall_score_min: 0.8
  - case: intent_not_resolved
    expected_status: review
    expected_dimension: qa.intent_resolution