{ "cells": [ { "cell_type": "markdown", "id": "0e084c2f", "metadata": {}, "source": [ "# Analýza sentimentu historických dopisů pomocí LLM\n", "\n", "Tento notebook ukazuje, jak využít velké jazykové modely (LLM) k analýze historických textů.\n", "\n", "**Zdroj dopisů:** https://www.iwm.org.uk/history/letters-to-loved-ones" ] }, { "cell_type": "markdown", "id": "setup-header", "metadata": {}, "source": [ "## 1. Instalace závislostí\n", "\n", "Spusťte následující buňku pro instalaci potřebných knihoven." ] }, { "cell_type": "code", "execution_count": 19, "id": "a5996bdb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install -Uqq openai pandas matplotlib ipywidgets\n", "%pip install -Uqq wordcloud" ] }, { "cell_type": "markdown", "id": "imports-header", "metadata": {}, "source": [ "## 2. Import knihoven a nastavení" ] }, { "cell_type": "code", "execution_count": 32, "id": "5d8e3784-imports", "metadata": {}, "outputs": [], "source": [ "import json\n", "import re\n", "from pathlib import Path\n", "from typing import List, Optional\n", "\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "from openai import OpenAI\n", "from pydantic import BaseModel, ValidationError\n", "\n", "plt.rcParams[\"figure.figsize\"] = (10, 6)\n", "plt.rcParams[\"font.size\"] = 12" ] }, { "cell_type": "markdown", "id": "model-config", "metadata": {}, "source": [ "## 3. Konfigurace modelu\n", "\n", "**Důležité:** Tento notebook předpokládá běžící lokální LLM server (např. LM Studio, Ollama).\n", "\n", "- Stáhněte LM Studio: https://lmstudio.ai/\n", "- Načtěte model (např. Gemma 4 E4B)\n", "- Spusťte lokální server (na portu 1234)" ] }, { "cell_type": "code", "execution_count": 33, "id": "model-config-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Připojeno k: http://127.0.0.1:1234/v1\n", "Model: google/gemma-3-4b\n" ] } ], "source": [ "BASE_URL = \"http://127.0.0.1:1234/v1\"\n", "API_KEY = \"sk-no-key-required\"\n", "MODEL_NAME = \"google/gemma-3-4b\"\n", "\n", "client = OpenAI(base_url=BASE_URL, api_key=API_KEY)\n", "print(f\"Připojeno k: {BASE_URL}\")\n", "print(f\"Model: {MODEL_NAME}\")" ] }, { "cell_type": "markdown", "id": "schema-header", "metadata": {}, "source": [ "## 4. Definice struktury výstupu\n", "\n", "Používáme **Pydantic** pro validaci dat. To zajistí, že model vrátí strukturovaná data, se kterými se dá dále pracovat." ] }, { "cell_type": "code", "execution_count": 34, "id": "schema-code", "metadata": {}, "outputs": [], "source": [ "class LetterInfo(BaseModel):\n", " sender: str\n", " recipient: str\n", " date: str\n", " location: str\n", " key_topics: str\n", " historical_context: str\n", " tone: str\n", " emotion: str" ] }, { "cell_type": "markdown", "id": "prompt-header", "metadata": {}, "source": [ "## 5. Prompt pro analýzu\n", "\n", "Prompt je klíčovou částí práce s LLM. Všimněte si:\n", "- Role (\"social historian\")\n", "- Jasné instrukce o formátu\n", "- Příklad výstupu" ] }, { "cell_type": "code", "execution_count": 35, "id": "prompt-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Prompt připraven.\n" ] } ], "source": [ "SYSTEM_PROMPT = \"\"\"Do NOT use chain of thought or step-by-step reasoning. Output ONLY the final JSON.\n", "Act as a social historian. Analyse the following letter and answer ONLY in JSON \n", "with the keys: sender, recipient, date, location, key_topics, historical_context, tone, emotion. \n", "All values must be **plain strings** (no objects, no arrays, no markdown). \n", "Do NOT add any extra text. \n", "Example: {\"sender\":\"John\",\"recipient\":\"Mary\",\"date\":\"probably 1914-1918\",\n", "\"location\":\"London\",\"key_topics\":\"war, family, fear\",\n", "\"historical_context\":\"German raid over England\",\n", "\"tone\":\"intimate\",\"emotion\":\"fear mixed with relief\"}\"\"\"\n", "\n", "print(\"Prompt připraven.\")" ] }, { "cell_type": "markdown", "id": "functions-header", "metadata": {}, "source": [ "## 6. Pomocné funkce" ] }, { "cell_type": "code", "execution_count": 36, "id": "27c677a6", "metadata": {}, "outputs": [], "source": [ "def read_letter(path: Path) -> Optional[str]:\n", " if not path.is_file():\n", " print(f\"❌ Soubor {path} nebyl nalezen.\")\n", " return None\n", " return path.read_text(encoding=\"utf-8\")\n", "\n", "\n", "def analyse_letter(text: str, model: str = MODEL_NAME) -> Optional[LetterInfo]:\n", " try:\n", " response = client.chat.completions.create(\n", " model=model,\n", " messages=[\n", " {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n", " {\"role\": \"user\", \"content\": text},\n", " ],\n", " max_tokens=4096,\n", " temperature=0.0,\n", " )\n", "\n", " # Kontrola, zda máme vůbec nějaký textový obsah (může být prázdný, pokud model jen \"přemýšlel\" a došly tokeny)\n", " if hasattr(response, \"choices\") and len(response.choices) > 0:\n", " raw = (response.choices[0].message.content or \"\").strip()\n", " else:\n", " print(f\"⚠️ Neočekávaný formát odpovědi: {response}\")\n", " return None\n", "\n", " if not raw:\n", " print(\n", " \"⚠️ Model vrátil prázdný obsah (pravděpodobně vyčerpal tokeny během reasoning fáze).\"\n", " )\n", " return None\n", "\n", " # ROBUSTNÍ PARSOVÁNÍ: Hledáme první složenou závorku '{' a poslední '}' pomocí regulárního výrazu.\n", " # Tím bezpečně ignorujeme veškerý balast kolem JSONu (např. ```json ... ``` nebo úvodní věty).\n", " match = re.search(r\"\\{.*\\}\", raw, re.DOTALL)\n", " if not match:\n", " print(\n", " f\"⚠️ V odpovědi nebyl nalezen validní JSON objekt. Raw output: {raw[:300]}...\"\n", " )\n", " return None\n", "\n", " json_str = match.group(0)\n", " data = json.loads(json_str)\n", "\n", " return LetterInfo(**data)\n", "\n", " except json.JSONDecodeError as e:\n", " print(f\"⚠️ Chyba při parsování JSON: {e}\")\n", " print(\n", " f\" Extrahovaný řetězec pro debug: {json_str if 'json_str' in locals() else 'N/A'}\"\n", " )\n", " except ValidationError as e:\n", " print(f\"⚠️ Validace Pydantinem selhala: {e}\")\n", " except Exception as e:\n", " print(f\"⚠️ Nepředvídaná chyba ({type(e).__name__}): {e}\")\n", " return None\n" ] }, { "cell_type": "markdown", "id": "sample-header", "metadata": {}, "source": [ "## 7. Ukázka: Analýza jednoho dopisu\n", "\n", "Nejprve vyzkoušíme analýzu na jednom dopisu, abychom ověřili, že vše funguje." ] }, { "cell_type": "code", "execution_count": 37, "id": "sample-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "📝 Ukázka dopisu (prvních 500 znaků):\n", "\n", "'Dear Daddy, I hope you are not alarmed, you should not be, unless you know where one of the Zepps went. I have heard that it raided London (up the Strand) and caused heavy causalities. But this I know because I saw, and so did everyone else in the house.\n", "\n", "Here is my story: I heard the clock strike 11 o'clock. I was in bed and just going to sleep. Between 2 'clock and 2.30 o'clock, Lily (the servant) woke Miss Willy and told her she could hear the guns. Miss Willy woke Poolman and told him to wa...\n", "\n", "============================================================\n", "\n", "🤖 Spouštím analýzu...\n", "\n", "✅ Výsledek analýzy:\n", " sender: Patrick\n", " recipient: Daddy\n", " date: 1914-1918\n", " location: Cheshunt, Hertfordshire\n", " key_topics: Zeppelin raid, airship crash, war casualties, family safety\n", " historical_context: World War I German Zeppelin bombing campaigns over Britain\n", " tone: Reassuring and descriptive\n", " emotion: Relief mixed with shock\n" ] } ], "source": [ "sample_letter = Path(\"historical_letter_01.txt\")\n", "text = read_letter(sample_letter)\n", "\n", "if text:\n", " print(\"📝 Ukázka dopisu (prvních 500 znaků):\\n\")\n", " print(text[:500] + \"...\")\n", " print(\"\\n\" + \"=\" * 60 + \"\\n\")\n", "\n", " print(\"🤖 Spouštím analýzu...\")\n", " result = analyse_letter(text)\n", "\n", " if result:\n", " print(\"\\n✅ Výsledek analýzy:\")\n", " for field, value in result.model_dump().items():\n", " print(f\" {field}: {value}\")\n", " else:\n", " print(\"❌ Analýza selhala.\")" ] }, { "cell_type": "markdown", "id": "batch-header", "metadata": {}, "source": [ "## 8. Dávková analýza\n", "\n", "Nyní analyzujeme všechny dostupné dopisy." ] }, { "cell_type": "code", "execution_count": 38, "id": "batch-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "📄 Zpracovávám: historical_letter_01.txt\n", " ✅ Úspěch\n", "📄 Zpracovávám: historical_letter_02.txt\n", " ✅ Úspěch\n", "📄 Zpracovávám: historical_letter_03.txt\n", " ✅ Úspěch\n", "\n", "Analyzováno 3 dopisů.\n" ] } ], "source": [ "filenames = [f\"historical_letter_{i:02}.txt\" for i in range(1, 4)]\n", "results: List[dict] = []\n", "\n", "for fn in filenames:\n", " print(f\"📄 Zpracovávám: {fn}\")\n", " text = read_letter(Path(fn))\n", "\n", " if not text:\n", " continue\n", "\n", " info = analyse_letter(text)\n", "\n", " if info:\n", " info_dict = info.model_dump()\n", " info_dict[\"filename\"] = fn\n", " results.append(info_dict)\n", " print(\" ✅ Úspěch\")\n", " else:\n", " print(\" ❌ Selhalo\")\n", "\n", "df = pd.DataFrame(results)\n", "print(f\"\\nAnalyzováno {len(df)} dopisů.\")" ] }, { "cell_type": "markdown", "id": "table-header", "metadata": {}, "source": [ "## 9. Tabulka výsledků" ] }, { "cell_type": "code", "execution_count": 39, "id": "table-code", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
senderrecipientdatelocationkey_topicshistorical_contexttoneemotionfilename
0PatrickDaddy1914-1918Cheshunt, HertfordshireZeppelin raid, airship crash, war casualties, ...World War I German Zeppelin bombing campaigns ...Reassuring and descriptiveRelief mixed with shockhistorical_letter_01.txt
1EmilyWillc. 1914-1918 (World War I)United Kingdomwar, family separation, military enlistment, h...World War I home front life, concerns over sol...intimate, anxious, lovingworry, hope, affectionhistorical_letter_02.txt
2TedGertieMayWhite Sea, Arctic regionshipping, food supplies, weather, finances, wo...World War II Arctic naval operations and Briti...informative and affectionatepride mixed with concern for familyhistorical_letter_03.txt
\n", "
" ], "text/plain": [ " sender recipient date location \\\n", "0 Patrick Daddy 1914-1918 Cheshunt, Hertfordshire \n", "1 Emily Will c. 1914-1918 (World War I) United Kingdom \n", "2 Ted Gertie May White Sea, Arctic region \n", "\n", " key_topics \\\n", "0 Zeppelin raid, airship crash, war casualties, ... \n", "1 war, family separation, military enlistment, h... \n", "2 shipping, food supplies, weather, finances, wo... \n", "\n", " historical_context \\\n", "0 World War I German Zeppelin bombing campaigns ... \n", "1 World War I home front life, concerns over sol... \n", "2 World War II Arctic naval operations and Briti... \n", "\n", " tone emotion \\\n", "0 Reassuring and descriptive Relief mixed with shock \n", "1 intimate, anxious, loving worry, hope, affection \n", "2 informative and affectionate pride mixed with concern for family \n", "\n", " filename \n", "0 historical_letter_01.txt \n", "1 historical_letter_02.txt \n", "2 historical_letter_03.txt " ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "markdown", "id": "export-header", "metadata": {}, "source": [ "## 10. Export výsledků" ] }, { "cell_type": "code", "execution_count": 40, "id": "export-code", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "📁 Uloženo do: letter_analysis.csv\n" ] } ], "source": [ "csv_path = Path(\"letter_analysis.csv\")\n", "df.to_csv(csv_path, index=False, encoding=\"utf-8\")\n", "print(f\"📁 Uloženo do: {csv_path}\")" ] }, { "cell_type": "markdown", "id": "reflection-header", "metadata": {}, "source": [ "## 11. Reflexe a diskuze\n", "\n", "### Otázky k zamyšlení:\n", "\n", "1. **Přesnost analýzy:** Jak dobře model zachytil kontext dopisů? Jsou nějaké chyby nebo nepřesnosti?\n", "\n", "2. **Subjektivita sentimentu:** Je sentiment jednoznačný? Může se interpretace lišit?\n", "\n", "3. **Historický kontext:** Co model nemohl vědět? Jaké informace chybí?\n", "\n", "4. **Limity metody:** Co analýza nezachycuje (ironie, sarkasmus, kulturní nuance)?\n", "\n", "### Kritické myšlení v digital humanities:\n", "\n", "AI nástroje mohou pomoci s analýzou velkých korpusů, ale:\n", "- Výsledky vyžadují lidskou interpretaci\n", "- Modely mají předsudky z tréninkových dat\n", "- Historický kontext vyžaduje expertní znalost\n", "\n", "**Doporučení:** Vždy ověřujte výsledky AI proti primárním zdrojům a vlastní expertíze." ] } ], "metadata": { "kernelspec": { "display_name": "jupyter", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }