索引构建脚本： scripts/build_bkm_repair_index_from_pdfs.py ，用仓库现有 pypdf 按页抽取 PDF 文本并生成 bkm_repair.json （字段： id/title/question/action/root_cause/pdf/page/snippet ），并支持可选 --enrich ollama 生成卡片所需的 action/root_cause 。

1.Architecture design

2.Technology Description

• Frontend: SvelteKit + TypeScript（复用 Open WebUI UI 体系；优先支持流式渲染以提升体感速度）
• Backend: FastAPI（参考 KPI Bot 的 Bottun 应用挂载方式；负责检索、拼装三段式输出与流式返回）
• Data: 本地文件（bkm.json + bkm.pdf）；反馈复用 Open WebUI 内置 DB
• LLM: 复用现有本地模型运行方式（例如通过 Ollama；与 KPI Bot 的 AIService 类似）；Prompt 强制约束“少总结/少重复”

2.1 Configuration（参数/模型必须配置化）

针对该 BKM Bot，重要参数与模型选择必须由配置文件驱动（并允许用环境变量覆盖），避免写死在代码里。

配置优先级建议：环境变量 > BKM bot_config.yaml > 代码默认值。

建议配置文件位置：

• 全局启用/访问控制：backend/open_webui/apps/bots/config/bots_config.yaml
• 新 BKM 细粒度参数：backend/open_webui/apps/bkm_repair/config/bot_config.yaml

关键配置项（示例分类）：

• Ollama / 推理端：
  • ollama_base_url（或 host）
  • ollama_generate_model（思考过程/生成）
  • rerank_engine=ollama 时的 rerank_ollama_model
• Embeddings：
  • embedding_engine、embedding_model
  • embedding_timeout_s、embedding_batch_size
• 检索与重排：
  • top_k、embedding_top_k
  • rerank_timeout_s、require_rerank
• 阈值与过滤：
  • 文件检索相似度阈值：min_embedding_similarity（embedding 相似度下限）
  • 重排阈值：min_rerank_score
  • 卡片展示阈值：action_suggestion_min_score
  • out-of-scope：refuse_out_of_scope、out_of_scope_message
• 数据路径：
  • bkm_json_path（或等价变量）
  • assets_base_url（右侧 PDF 预览资源前缀）

2.2 Index Build（从 PDF 构建检索数据）

索引构建通过离线脚本生成 bkm_repair.json（JSON array），字段与后端检索一致：id/title/question/action/root_cause/pdf/page/snippet。

• 脚本：scripts/build_bkm_repair_index_from_pdfs.py
• 默认输入目录：/Users/sophia/projects/new_pages_fpd_s（可用 --input-dir 覆盖）
• 默认输出：backend/open_webui/apps/bkm_repair/data/bkm_repair.json
• 可选 enrich：--enrich ollama 可调用 Ollama 为每页/分块生成 title/question/action/root_cause（用于前端卡片框框展示）。

3.Route definitions

Route	Purpose
/bkm-repair	新 BKM 聊天页（专用 UI，含来源阅读区与反馈）
/bkm-repair/v1/chat/completions	OpenAI 兼容对话接口（便于按“模型”方式接入/复用现有调用方式）
/bkm-repair/chat/search	（内部）对 BKM JSON 做检索并返回候选条目（含 action/root cause/page）
/bkm-repair/assets/{pdf}	提供 BKM PDF 静态访问，用于“#page=”跳转
/api/feedback	复用 Open WebUI 反馈接口，记录点赞点踩（meta 里带 chat_id/message_id）

隔离约束：该机器人为全新实现，后端代码不得与 backend/open_webui/apps/bkm 产生任何重叠（目录、import、路由前缀、配置文件均需独立）。

4.API definitions (If it includes backend services)

4.1 Core API

BKM 问答（OpenAI 兼容）

POST /bkm-repair/v1/chat/completions

Request:

Param Name	Param Type	isRequired	Description
model	string	true	固定为 bkm-repair-bot（示例，可配置；需避免与旧版 bkm-bot 冲突）
messages	{role:string, content:string}[]	true	OpenAI messages，取最后一条 user 作为问题
stream	boolean	false	是否流式返回（建议默认 true，以达成“快速响应”体感）

Response（核心字段约定，允许 extra）：

Param Name	Param Type	Description
choices[0].message.content	string	Markdown 文本（可直接展示），默认按“三段式”排版
choices[0].message.meta	object	结构化元信息（推荐 UI 优先使用）：

• three_stage.fast_answer: string（1–2 句）
• three_stage.key_points: {title?:string, actions:{text:string, score?:number}[], root_causes:{text:string, score?:number}[]}[]
• three_stage.evidence: {quote:string, page:number, source_pdf:string}[]
• pdf_page_refs: number[]（去重排序，用于右侧“引用页”列表）
• render_hints: {avoid_redundant_summary:boolean, max_fast_answer_sentences:number} |

Frontend 渲染约定（保持现有 Open WebUI 的 BKM Bot 对话展示方式）：

• “原因/行动建议”以可点击框框卡片呈现；每个卡片右上角展示相似度 Tag（0%–100%）。
• 相似度定义：卡片相似度为整体相似度，取“该卡片对应的文档内容（snippet/证据块中的问题或条目问句/标题）”与“用户当前问题”的相似度。
  • 数据源：后端返回 score（0…1），按同一请求口径归一化，可在多卡片之间比较。
  • 展示：前端换算 round(score * 100)，显示为 相似度 86%。
• PDF 预览与水印：完全复用现有 BKM 右侧 PDF 侧栏/抽屉实现。
  • 前端不提供下载/打印入口，并禁用/隐藏内置工具栏（按当前版本实现）。
  • PDF 预览上叠加水印层（姓名/邮箱或 display_name + 当前时间），随会话/打开动作实时生成。

思考过程流式输出（真实 streaming，非伪实时）：

• 若需要展示“思考过程”，必须使用服务端流进行增量传输（例如 SSE text/event-stream 或 chunked response），后端在生成时边产出边 flush。
• 前端必须按流逐块解析并增量渲染（不得使用 res.json()）；需要兼容流中断/重连与最终汇总内容落盘。

BKM 检索（内部，便于调试/复用）

POST /bkm-repair/chat/search

Request:

Param Name	Param Type	isRequired	Description
query	string	true	用户问题
top_k	number	false	返回条目数量，默认 5

Response:

Param Name	Param Type	Description
items	array	候选条目列表（action/root_cause/evidence/page 等）

5.Server architecture diagram (If it includes backend services)

6.Data model(if applicable)

6.1 Data model definition

（可选：若希望支持增量更新/在线管理，可将 JSON 条目落库；否则可直接文件加载，不建表。）

6.2 Data Definition Language

BKM 条目表（bkm_items，可选）

CREATE TABLE bkm_items (
  id TEXT PRIMARY KEY,
  title TEXT,
  action TEXT,
  root_cause TEXT,
  evidence TEXT,
  pdf_page INTEGER,
  source_pdf TEXT,
  raw_json JSON,
  created_at BIGINT,
  updated_at BIGINT
);

CREATE INDEX idx_bkm_items_pdf_page ON bkm_items(pdf_page);

运行逻辑

python scripts/build_bkm_repair_index_from_pdfs.py \
  --input-dir /Users/sophia/projects/new_pages_fpd_s \
  --output backend/open_webui/apps/bkm_repair/data/bkm_repair.json \
  --source jsonl \
  --enrich ollama

import argparse
import json
import os
import re
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Iterable, Iterator, Optional

import httpx
from pypdf import PdfReader


def _load_yaml_file(path: Path) -> dict[str, Any]:
    try:
        import yaml  # type: ignore

        with path.open("r", encoding="utf-8") as f:
            data = yaml.safe_load(f)
        return data if isinstance(data, dict) else {}
    except Exception:
        out: dict[str, Any] = {}
        try:
            with path.open("r", encoding="utf-8") as f:
                for raw in f:
                    line = raw.strip()
                    if not line or line.startswith("#"):
                        continue
                    if ":" not in line:
                        continue
                    k, v = line.split(":", 1)
                    k = k.strip()
                    v = v.strip().strip('"').strip("'")
                    if not k:
                        continue
                    out[k] = v
        except Exception:
            return {}
        return out


def _normalize_text(text: str) -> str:
    t = text.replace("\u0000", " ")
    t = re.sub(r"[ \t\f\v]+", " ", t)
    t = re.sub(r"\n{3,}", "\n\n", t)
    return t.strip()


def _strip_image_blocks(text: str) -> str:
    return re.sub(r"<image[^>]*>[\s\S]*?</image>", " ", text, flags=re.IGNORECASE)


def _slug(s: str, max_len: int = 80) -> str:
    s = s.strip().lower()
    s = re.sub(r"[^a-z0-9\-_.]+", "-", s)
    s = re.sub(r"-+", "-", s).strip("-")
    if not s:
        s = "item"
    return s[:max_len]


def _pick_title_from_page_text(text: str) -> str:
    for raw in text.splitlines():
        line = raw.strip()
        if not line:
            continue
        if line.lower().startswith("<image"):
            continue
        line = re.sub(r"\s+", " ", line)
        if 4 <= len(line) <= 80:
            return line
        if len(line) > 80:
            return line[:80]
    return ""


def _chunk_text(text: str, chunk_size: int, chunk_overlap: int) -> Iterator[str]:
    if chunk_size <= 0:
        yield text
        return

    if chunk_overlap < 0:
        chunk_overlap = 0
    if chunk_overlap >= chunk_size:
        chunk_overlap = max(0, chunk_size // 5)

    i = 0
    n = len(text)
    while i < n:
        end = min(n, i + chunk_size)
        yield text[i:end]
        if end >= n:
            break
        i = max(0, end - chunk_overlap)


@dataclass(frozen=True)
class OllamaConfig:
    base_url: str
    model: str
    timeout_s: int
    max_input_chars: int


def _ollama_generate_json(cfg: OllamaConfig, prompt: str) -> dict[str, Any]:
    url = cfg.base_url.rstrip("/") + "/api/generate"
    payload = {
        "model": cfg.model,
        "prompt": prompt,
        "stream": False,
        "format": "json",
    }
    timeout = httpx.Timeout(cfg.timeout_s)
    with httpx.Client(timeout=timeout) as client:
        res = client.post(url, json=payload)
        res.raise_for_status()
        data = res.json()
        if not isinstance(data, dict):
            return {}
        raw = data.get("response")
        if not isinstance(raw, str) or not raw.strip():
            return {}
        try:
            obj = json.loads(raw)
        except Exception:
            m = re.search(r"\{[\s\S]*\}", raw)
            if not m:
                return {}
            try:
                obj = json.loads(m.group(0))
            except Exception:
                return {}
        return obj if isinstance(obj, dict) else {}


def _build_enrich_prompt(page_text: str) -> str:
    return (
        "你是半导体设备维修文档的结构化抽取器。\n"
        "给定一段来自PDF某一页的文本，请输出严格JSON对象（不要任何多余文字），字段如下：\n"
        "- title: 该页主题（<=30字）\n"
        "- question: 用户可能会问的问题（<=40字，尽量口语化，可带问号）\n"
        "- action: 可执行的行动建议（<=60字，分号分隔多个步骤）\n"
        "- root_cause: 可能根本原因（<=60字，分号分隔多个原因）\n"
        "- snippet: 证据摘录/关键句（<=200字，尽量贴近原文）\n"
        "要求：内容必须来自给定文本，不要编造不存在的信息；缺失则输出空字符串。\n\n"
        f"PDF页文本：\n{page_text}\n"
    )


def _discover_pdfs(input_dir: Path, pattern: str) -> list[Path]:
    if not input_dir.exists():
        return []
    if pattern == "*.pdf":
        pdfs = [p for p in input_dir.rglob("*") if p.is_file() and p.suffix.lower() == ".pdf"]
        return sorted(set(pdfs))
    return sorted({p for p in input_dir.rglob(pattern) if p.is_file()})


def _discover_jsonl(input_dir: Path) -> list[Path]:
    if not input_dir.exists():
        return []
    return sorted({p for p in input_dir.rglob("*.jsonl") if p.is_file()})


def _infer_source_kind(input_dir: Path, pdf_pattern: str, requested: str) -> tuple[str, list[Path]]:
    requested = (requested or "").strip().lower()
    if requested in {"pdf", "jsonl"}:
        if requested == "pdf":
            return "pdf", _discover_pdfs(input_dir, pdf_pattern)
        return "jsonl", _discover_jsonl(input_dir)

    pdfs = _discover_pdfs(input_dir, pdf_pattern)
    if pdfs:
        return "pdf", pdfs
    jsonls = _discover_jsonl(input_dir)
    return "jsonl", jsonls


def _pdf_item_pdf_field(pdf_path: Path, input_dir: Path, mode: str) -> str:
    if mode == "relative":
        try:
            rel = pdf_path.relative_to(input_dir)
            return rel.as_posix()
        except Exception:
            return pdf_path.name
    return pdf_path.name


def _jsonl_item_pdf_field(jsonl_path: Path, input_dir: Path, mode: str, transform: str) -> str:
    name = jsonl_path.name
    if name.lower().endswith(".jsonl"):
        name = name[: -len(".jsonl")]

    transform = (transform or "").strip().lower()
    if transform == "strip-pptx-add-pdf":
        if name.lower().endswith(".pptx"):
            name = name[: -len(".pptx")]
        name = name + ".pdf"
    elif transform == "add-pdf":
        name = name + ".pdf"

    if mode == "relative":
        try:
            rel = jsonl_path.relative_to(input_dir)
            rel_name = rel.as_posix()
            if rel_name.lower().endswith(".jsonl"):
                rel_name = rel_name[: -len(".jsonl")]
            if transform == "strip-pptx-add-pdf" and rel_name.lower().endswith(".pptx"):
                rel_name = rel_name[: -len(".pptx")] + ".pdf"
            elif transform == "add-pdf":
                rel_name = rel_name + ".pdf"
            return rel_name
        except Exception:
            return name
    return name


def _iter_pdf_pages(pdf_path: Path, keep_image_blocks: bool) -> Iterator[tuple[int, str]]:
    with pdf_path.open("rb") as f:
        reader = PdfReader(f)
        for idx, page in enumerate(reader.pages, start=1):
            text = page.extract_text() or ""
            if not keep_image_blocks:
                text = _strip_image_blocks(text)
            yield idx, _normalize_text(text)


def _iter_jsonl_pages(jsonl_path: Path, keep_image_blocks: bool) -> Iterator[tuple[int, str]]:
    with jsonl_path.open("r", encoding="utf-8") as f:
        for raw in f:
            line = raw.strip()
            if not line:
                continue
            try:
                obj = json.loads(line)
            except Exception:
                continue
            if not isinstance(obj, dict):
                continue
            page = obj.get("page")
            text = obj.get("text")
            if not isinstance(page, int) or page <= 0:
                continue
            if not isinstance(text, str):
                continue
            if not keep_image_blocks:
                text = _strip_image_blocks(text)
            yield page, _normalize_text(text)


def _write_json_array_stream(output_path: Path, rows: Iterable[dict[str, Any]]) -> int:
    output_path.parent.mkdir(parents=True, exist_ok=True)
    n = 0
    with output_path.open("w", encoding="utf-8") as f:
        f.write("[\n")
        first = True
        for row in rows:
            if not first:
                f.write(",\n")
            first = False
            f.write(json.dumps(row, ensure_ascii=False))
            n += 1
        f.write("\n]\n")
    return n


def main() -> None:
    parser = argparse.ArgumentParser(
        description=(
            "Build bkm_repair.json from a directory of PDFs.\n"
            "Outputs a JSON array with fields: id/title/question/action/root_cause/pdf/page/snippet.\n"
            "Use --enrich ollama to fill action/root_cause via Ollama; otherwise those fields may be empty."
        )
    )
    parser.add_argument(
        "--input-dir",
        default="/Users/sophia/projects/new_pages_fpd_s",
        help="Directory containing PDF files (recursive)",
    )
    parser.add_argument(
        "--output",
        default="backend/open_webui/apps/bkm_repair/data/bkm_repair.json",
        help="Output JSON file path",
    )
    parser.add_argument(
        "--pattern",
        default="*.pdf",
        help="Glob pattern used with recursive search (default: *.pdf)",
    )
    parser.add_argument(
        "--source",
        choices=["auto", "pdf", "jsonl"],
        default="auto",
        help="Input source type. auto prefers PDFs; falls back to *.jsonl when no PDFs found.",
    )
    parser.add_argument(
        "--jsonl-pdf-name-transform",
        choices=["keep", "strip-pptx-add-pdf", "add-pdf"],
        default="strip-pptx-add-pdf",
        help="How to derive the output 'pdf' field from a *.jsonl file name.",
    )
    parser.add_argument(
        "--pdf-name-mode",
        choices=["basename", "relative"],
        default="basename",
        help="How to write the 'pdf' field (basename or path relative to input-dir)",
    )
    parser.add_argument(
        "--max-pdfs",
        type=int,
        default=0,
        help="Process at most N PDFs (0 means all)",
    )
    parser.add_argument(
        "--max-pages-per-pdf",
        type=int,
        default=0,
        help="Process at most N pages per PDF (0 means all)",
    )
    parser.add_argument(
        "--min-text-chars",
        type=int,
        default=80,
        help="Skip pages with extracted text shorter than this",
    )
    parser.add_argument(
        "--chunk-size",
        type=int,
        default=1200,
        help="Chunk size by characters (<=0 disables chunking)",
    )
    parser.add_argument(
        "--chunk-overlap",
        type=int,
        default=150,
        help="Overlap between chunks by characters",
    )
    parser.add_argument(
        "--snippet-max-chars",
        type=int,
        default=600,
        help="Max chars for snippet field",
    )
    parser.add_argument(
        "--keep-image-blocks",
        action="store_true",
        help="Keep <image ...>...</image> blocks in extracted text (default strips them)",
    )
    parser.add_argument(
        "--enrich",
        choices=["none", "ollama"],
        default="none",
        help="Whether to generate title/question/action/root_cause via Ollama",
    )
    parser.add_argument(
        "--bot-config",
        default="backend/open_webui/apps/bkm_repair/config/bot_config.yaml",
        help="BKM repair bot_config.yaml (used to read ollama settings)",
    )
    parser.add_argument(
        "--ollama-base-url",
        default="",
        help="Override Ollama base URL (e.g. http://localhost:11434)",
    )
    parser.add_argument(
        "--ollama-model",
        default="",
        help="Override Ollama model name (e.g. qwen2.5:7b)",
    )
    parser.add_argument(
        "--ollama-timeout-s",
        type=int,
        default=120,
        help="Ollama request timeout in seconds",
    )
    parser.add_argument(
        "--ollama-max-input-chars",
        type=int,
        default=6000,
        help="Max chars sent to Ollama per page/chunk",
    )
    parser.add_argument("--dry-run", action="store_true")
    args = parser.parse_args()

    input_dir = Path(args.input_dir).expanduser().resolve()
    output_path = Path(args.output).expanduser().resolve()
    source_kind, sources = _infer_source_kind(input_dir, str(args.pattern), str(args.source))
    if args.max_pdfs and args.max_pdfs > 0:
        sources = sources[: args.max_pdfs]
    if not sources:
        pdf_count = len(_discover_pdfs(input_dir, str(args.pattern)))
        jsonl_count = len(_discover_jsonl(input_dir))
        raise SystemExit(
            "No input documents found. "
            f"pdf_count={pdf_count}, jsonl_count={jsonl_count}. "
            "Check --input-dir / --pattern, or use --source jsonl if your directory contains *.jsonl." 
        )

    raw_cfg = {}
    cfg_path = Path(args.bot_config).expanduser().resolve()
    if cfg_path.exists():
        raw_cfg = _load_yaml_file(cfg_path)

    enrich_mode = str(args.enrich)
    ollama_cfg: Optional[OllamaConfig] = None
    if enrich_mode == "ollama":
        base_url = (
            str(args.ollama_base_url).strip()
            or str(os.getenv("BKM_REPAIR_OLLAMA_BASE_URL") or "").strip()
            or str(raw_cfg.get("ollama_base_url") or "").strip()
            or "http://localhost:11434"
        )
        model = (
            str(args.ollama_model).strip()
            or str(os.getenv("BKM_REPAIR_OLLAMA_GENERATE_MODEL") or "").strip()
            or str(raw_cfg.get("ollama_generate_model") or "").strip()
        )
        if not model:
            raise SystemExit("Missing Ollama model. Set --ollama-model or configure ollama_generate_model.")
        ollama_cfg = OllamaConfig(
            base_url=base_url,
            model=model,
            timeout_s=int(args.ollama_timeout_s),
            max_input_chars=int(args.ollama_max_input_chars),
        )

    def iter_rows() -> Iterator[dict[str, Any]]:
        for doc_idx, doc_path in enumerate(sources, start=1):
            if source_kind == "pdf":
                pdf_field = _pdf_item_pdf_field(doc_path, input_dir, args.pdf_name_mode)
                page_iter = _iter_pdf_pages(doc_path, bool(args.keep_image_blocks))
            else:
                pdf_field = _jsonl_item_pdf_field(
                    doc_path,
                    input_dir,
                    args.pdf_name_mode,
                    str(args.jsonl_pdf_name_transform),
                )
                page_iter = _iter_jsonl_pages(doc_path, bool(args.keep_image_blocks))

            for page_no, page_text in page_iter:
                if args.max_pages_per_pdf and args.max_pages_per_pdf > 0 and page_no > args.max_pages_per_pdf:
                    break
                if len(page_text) < int(args.min_text_chars):
                    continue

                base_title = _pick_title_from_page_text(page_text)
                if not base_title:
                    base_title = f"{Path(pdf_field).stem} 第{page_no}页"

                for chunk_i, chunk in enumerate(
                    _chunk_text(page_text, int(args.chunk_size), int(args.chunk_overlap)), start=1
                ):
                    chunk = chunk.strip()
                    if len(chunk) < int(args.min_text_chars):
                        continue

                    item_id = f"{_slug(pdf_field)}-p{page_no}-c{chunk_i}"
                    title = base_title
                    question = ""
                    action = ""
                    root_cause = ""

                    if ollama_cfg is not None:
                        prompt_text = chunk[: ollama_cfg.max_input_chars]
                        prompt = _build_enrich_prompt(prompt_text)
                        try:
                            obj = _ollama_generate_json(ollama_cfg, prompt)
                        except Exception:
                            obj = {}
                        if obj:
                            title = str(obj.get("title") or title).strip() or title
                            question = str(obj.get("question") or "").strip()
                            action = str(obj.get("action") or "").strip()
                            root_cause = str(obj.get("root_cause") or "").strip()

                    snippet = chunk[: int(args.snippet_max_chars)].strip()
                    if not snippet:
                        continue

                    yield {
                        "id": item_id,
                        "title": title,
                        "question": question,
                        "action": action,
                        "root_cause": root_cause,
                        "pdf": pdf_field,
                        "page": int(page_no),
                        "snippet": snippet,
                    }

    if args.dry_run:
        n = 0
        for _ in iter_rows():
            n += 1
            if n >= 5:
                break
        print(f"dry-run ok, sample rows: {n}")
        return

    started = time.time()
    count = _write_json_array_stream(output_path, iter_rows())
    elapsed = time.time() - started
    print(f"wrote {count} items to {output_path} in {elapsed:.1f}s")


if __name__ == "__main__":
    main()