hermes-skills/hermes-skills-research
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Publish hermes-skills-research via gitea-publish skill

Browse Source
This commit is contained in:
figmar
2026-08-09 08:19:33 +08:00
commit bca5c4f6fa
67 changed files with 29416 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
grounded-citations/SKILL.md
+232
View File
@@ -0,0 +1,232 @@
---
name: grounded-citations
description: "Ground answers and documents in cited, verifiable sources."
version: 1.1.0
author: Hermes Agent + Teknium
license: MIT
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [Research, Citations, Grounding, Sources, Web, Reports]
category: research
related_skills: [research-paper-writing, arxiv, ocr-and-documents]
---
# Grounded Citations
Every claim taken from an outside source gets an inline numbered citation and a
`Sources:` list, Perplexity-style. A ledger script owns the `url → [n]` mapping
so the numbers and URLs come from retrieval, never from memory — the model only
ever emits small integers it was handed.
For high-stakes work the same ledger doubles as a fact-checking chain: verbatim
quotes are attached to each source (rejected unless they literally appear in
the fetched page text), claims from model knowledge are flagged `[unverified]`,
and `verify --evidence` fails any draft whose cited sources carry no evidence.
This skill covers answers in chat, written documents (markdown, PDF, docx,
slides), and research reports. It does not cover academic BibTeX pipelines —
for conference papers use the `research-paper-writing` skill, which this skill
feeds (see `references/citation-formats.md`).
## When to Use
Use whenever an answer or artifact rests on information you fetched rather than
knew:
- Research, comparisons, news summaries, "what is the current state of X"
- Any deliverable you write to disk that quotes, paraphrases, or reports
outside facts — reports, briefs, docs, decks, wiki pages
- Fact-finding where the user will want to check your work
- Multi-source synthesis where conflicting sources must be attributed
Skip inline citations when the retrieval is incidental to another task — a
quick syntax/version lookup mid-coding, casual conversation, creative writing.
Mention a URL only if the user would plausibly want the link.
## Prerequisites
None beyond the standard toolset. `scripts/sources.py` is stdlib-only Python 3.
Retrieval comes from whatever is configured: `web_search`, `web_extract`,
`browser_navigate`, or `terminal` (curl, CLIs).
Ledger location: `$HERMES_HOME/cache/citations/ledger.json` (profile-aware).
Override per task with `--ledger <path>` or `HERMES_CITATION_LEDGER`.
## How to Run
```bash
S=~/.hermes/skills/research/grounded-citations/scripts/sources.py
python3 "$S" reset # start a clean ledger
python3 "$S" add https://example.com/a --title "A" # prints: [1]
python3 "$S" add https://example.com/b --title "B" # prints: [2]
python3 "$S" list # ledger table
python3 "$S" render # Sources: block
python3 "$S" verify draft.md # catch bad citations
```
`add` is idempotent and URL-normalized: the same page always returns the same
id within a ledger, so ids stay stable across many search/extract rounds.
## Quick Reference
| Action | Command |
|---|---|
| Fresh ledger for a new task | `sources.py reset` |
| Register a source, get its id | `sources.py add <url> [--title T]` |
| Register several at once | `sources.py add <url1> <url2> ...` |
| Register from JSON tool output | `sources.py ingest results.json` |
| Attach verbatim evidence to a source | `sources.py quote <id> --text "exact wording" --from page.txt` |
| Show ledger | `sources.py list [--json]` |
| Render the Sources block | `sources.py render [--style markdown\|plain\|footnotes\|bibtex\|evidence] [--only 1,3]` |
| Render only what a draft cites | `sources.py render --cited-in draft.md` |
| Rewrite a draft's Sources block in place | `sources.py render --replace-in draft.md` |
| Check a draft's citations | `sources.py verify draft.md [--strict] [--min-coverage 0.6] [--evidence]` |
## Procedure
① **Reset the ledger** at the start of a task that will produce a grounded
answer or document. Skip the reset when continuing work whose ids are already
in a draft — reusing the ledger keeps the numbering stable.
② **Register every source at retrieval time.** After each `web_search` /
`web_extract` / `browser_navigate` / fetch, pass the URLs to `sources.py add`
(or pipe the raw JSON through `sources.py ingest`). Do this *before* writing
prose. Registering later, from memory, is the failure mode this skill exists to
prevent.
③ **Write cite-while-drafting.** Place the bracketed id(s) immediately after
each sentence the source supports:
```
Ice floats because it is less dense than liquid water.[1][2]
```
- No space before the bracket; each id in its own brackets.
- Max 3 ids per sentence. Cite per sentence, not one dump at the end.
- Only ids the ledger returned. Never invent an id or a URL.
- Claims from your own knowledge get no citation.
- Conflicting sources: present both readings, each with its own id.
- Quote exact figures, dates, and names as the source states them; flag gaps
explicitly ("no source found for X") instead of smoothing them over.
④ **Append the Sources block** with `sources.py render --cited-in <draft>` so
the id → URL mapping is generated mechanically from the ledger, not retyped.
For non-markdown targets pick the matching `--style` and follow
`references/citation-formats.md` for placement (footnotes in docx, endnotes in
PDF/LaTeX, a Sources slide in decks, per-page source lists in wiki output).
⑤ **Verify before delivering** — `sources.py verify <draft>` exits non-zero on
unknown ids, on a Sources block that disagrees with the ledger, or (with
`--min-coverage`) on prose that is too thinly cited. Fix and re-run.
⑥ **Chat answers** follow the same steps with the draft in your reply: register
sources, cite inline, end with the rendered `Sources:` list. For a short answer
you may render the block from `sources.py render --only <ids>` instead of
writing to a file.
## Fact-Checking Mode
For work where the reader must be able to check the chain — medical, legal,
financial, safety, disputed claims, or when the user asks for fact-checking —
upgrade from citations to evidence:
① **Attach a verbatim quote per source.** After extracting a page, save its
text to a file and attach the sentence(s) that carry each claim:
```bash
python3 "$S" quote 1 --text "Ice is about 9% less dense than liquid water." --from page1.txt
```
The quote is rejected unless it appears verbatim in the evidence text
(insensitive to whitespace, case, and markdown markup — inline links like
`_[ERAP1](https://…)_` in extracted text match the plain prose a reader sees),
so a paraphrase or misremembered figure cannot masquerade as evidence.
Copy-paste from the fetched text; never retype. Quote the sentence as the
reader sees it — the matcher sees through the extractor's markup for you, so
you don't have to reproduce link syntax or escaped asterisks in your quote.
② **Flag model-knowledge claims with `[unverified]`.** A load-bearing claim
you could not source gets an explicit marker instead of a citation:
```
The refactor likely predates the 2.0 release.[unverified]
```
`verify --min-coverage` counts `[unverified]` sentences as covered — the goal
is declared provenance for every claim, not a citation on every sentence.
If a key claim can be checked, check it; `[unverified]` is for what genuinely
cannot be, and a fact-check deliverable dominated by `[unverified]` markers
should say so in its summary.
③ **Cross-check disputed facts against a second independent source.** When two
sources disagree, cite both readings with their own ids and quotes, and say
which you weight and why. One source is reporting; two independent sources are
corroboration.
④ **Verify with the evidence gate and render the evidence block:**
```bash
python3 "$S" verify report.md --evidence --min-coverage 0.5
python3 "$S" render --style evidence --replace-in report.md
```
`--evidence` fails the draft if any cited source has no attached quote. The
`evidence` render style prints each source's quotes beneath its URL, so the
deliverable shows claim → source → exact supporting text with nothing taken on
faith. Use `--replace-in <draft>` to rewrite an existing Sources block in place
(idempotent — safe to re-run after attaching more quotes); `--cited-in` prints
to stdout instead. Both emit the heading `## Sources` (`--style plain` emits
`Sources:`).
**What `--min-coverage` counts.** Coverage is
`sentences with declared provenance / prose sentences`. A prose sentence is a
non-empty line fragment of 4+ words after the Sources block, headings (`#`),
table rows (`|`), and fenced code are dropped; blockquote markers are stripped.
Provenance is declared by either a `[n]` citation or an `[unverified]` marker,
so a sentence carrying both counts once. Run `verify` without a threshold first
and read the `info: stats:` line to see the counts before picking a number.
## Pitfalls
- **Registering after writing.** The ledger must be populated from tool output,
not reconstructed from the draft — that reintroduces exactly the hallucinated
-URL risk the numbering removes.
- **Renumbering mid-task.** Never hand-edit ids in a draft. Ids are ledger
identities; if a draft cites `[4]`, `[4]` must stay that source. Run `reset`
only between tasks.
- **Retyping URLs into the Sources block.** Always `render`. A hand-typed URL
is an unverified claim.
- **Citing a search snippet as if you read the page.** A `web_search`
description supports only what it literally says. Cite the extracted page
when the claim needs the body — `web_extract` it first.
- **Over-citing.** Three ids on a sentence is the ceiling; a citation on every
clause makes text unreadable and hides which source carries the load.
- **Citing the ledger in code/config artifacts.** Source comments belong in
prose deliverables and doc headers, not inside generated code.
- **Parallel subagents.** Each subagent has its own working directory; point
them all at one ledger with `--ledger` (or `HERMES_CITATION_LEDGER`) if their
outputs get merged, otherwise their ids will collide.
- **Quoting from a snippet instead of the page.** Evidence quotes must come
from the extracted page text, not a search-result description — `web_extract`
first, save the text, then `quote --from` that file.
- **Paraphrasing into `quote --text`.** The verbatim check will reject it; the
fix is to find the actual sentence, not to reword until something matches.
- **Using `[unverified]` as an escape hatch.** It marks the rare claim that
genuinely cannot be sourced; if most sentences carry it, the task needed more
retrieval, not more markers.
- **Hand-editing the Sources block.** Use `render --replace-in <draft>`; slicing
the file yourself risks a stale or duplicated block that `verify` then flags.
## Verification
```bash
python3 "$S" verify report.md --strict --min-coverage 0.5
```
Green means: every `[n]` in the draft exists in the ledger, the Sources block
lists exactly the cited ids with the ledger's URLs, and the cited share of
source-bearing sentences meets the threshold. Read the warnings even when the
exit code is 0 — uncited registered sources usually mean a claim lost its
attribution during editing.
grounded-citations/references/citation-formats.md
+65
View File
@@ -0,0 +1,65 @@
# Citation formats per output target
The ledger is format-agnostic: `sources.py render --style ...` emits the block,
this file says where it goes and what the inline marker looks like.
## Markdown / chat answers
Inline `[n]` immediately after the sentence. Block at the end:
```
## Sources
[1] https://example.com/a — Page title
[2] https://example.com/b
```
`--style plain` gives a bare `Sources:` header for chat replies where a
markdown heading would be noise.
## PDF via LaTeX (`latex-pdf-report` skill)
Use `--style footnotes` and map each id to `\footnote{}` at first use, or keep
numeric markers and emit an endnotes section. For a bibliography-shaped report,
`--style bibtex` writes `@misc` entries keyed `source<N>`; cite them with
`\cite{source3}` and let BibTeX render the list.
Do not mix: either numeric `[n]` + a Sources section, or `\cite{}` + BibTeX.
Two numbering systems in one document is worse than none.
## Word (.docx, `docx` skill)
Real footnotes are preferred over inline brackets in prose documents intended
for human editing — reviewers expect Word footnotes. Keep the ledger ids as the
footnote numbers so `verify` still works on a markdown source-of-truth, and
generate the .docx from that markdown.
## Slides (.pptx, `powerpoint` skill)
Inline `[n]` in the bullet, one "Sources" slide at the end rendered with
`--style plain`. Never put a URL in a body bullet — it wrecks the layout and
can't be clicked in a projected deck.
## Spreadsheets (.xlsx)
Add a `source` column holding the id, plus a `Sources` sheet built from
`render --style plain`. Do not paste URLs into data cells.
## Wiki / multi-page output (`llm-wiki`, Obsidian)
Per-page Sources block, ids shared across pages from one ledger. Because ids
are ledger identities, `[7]` means the same page everywhere in the wiki — that
consistency is the reason not to reset the ledger between pages of one build.
## Research papers
Hand off to the `research-paper-writing` skill. Export with
`--style bibtex` into `references.bib`, then follow that skill's citation
verification (it greps `\cite{...}` against the .bib). The ledger's job ends at
producing verified URL entries; venue formatting is that skill's domain.
## Code and config artifacts
No citations inside generated code. If provenance matters, put it in the
commit message, the PR body, or a doc header — not in comments scattered
through source.
grounded-citations/references/grounding-rationale.md
+64
View File
@@ -0,0 +1,64 @@
# Why numbered ledger ids (grounding research basis)
Design notes for anyone changing the citation instructions or the ledger
mechanics. The wording in SKILL.md is not arbitrary.
## The structural trick
Hallucinated citations happen when a model reconstructs a URL from memory. If
the only thing the model has to emit is a small integer it was handed at
retrieval time, there is nothing to reconstruct — a wrong id is detectable
(it's not in the ledger) and a wrong URL is impossible (the model never types
one; `render` does). This is the property Perplexity's product relies on, and
it's why the ledger, not the prose, owns the URL.
Consequence: **register at retrieval, render mechanically.** Any workflow that
lets the model type a URL into the Sources block gives the guarantee back.
## Cite while writing, not after
ALCE (arXiv:2305.14627) evaluates attribution for LLM answers and finds that
generating citations during composition, from numbered retrieved snippets,
produces materially better attribution than post-hoc citation insertion.
Post-hoc attribution invites the model to find a source that plausibly matches
a sentence it already wrote — which is exactly how a citation ends up
supporting something the page doesn't say.
Hence: cite per supported sentence, in-line, as the sentence is written. Never
a citation dump at the end of a paragraph or document.
## Verbatim quotes ground claims
WebGPT (arXiv:2112.09332) collects verbatim quotes at browse time and composes
answers from them. The practical rule for this skill: when a claim carries a
figure, date, name, or quantity, take it from the source's own words rather
than paraphrasing from a summary of a summary. Each summarization hop is a
chance for a number to drift.
## Formatting conventions
From Perplexity's leaked system prompts (jujumilk3/leaked-system-prompts) — the
conventions are worth copying because they're the ones users have been trained
to read:
- Marker directly after the terminal punctuation, no space: `water.[1]`
- Each id in its own brackets: `[1][2]`, not `[1, 2]`
- At most 3 ids per sentence — beyond that the citation stops identifying which
source carries the claim
- Never cite a source not actually consulted
- Query classes that shouldn't be cited (translation, creative writing, casual
chat) are exempted by instruction, not by a separate classifier
Perplexity forbids raw URLs in the answer because its UI renders source cards.
Hermes has no such UI layer in chat or in a written file, so this skill renders
the id → URL list explicitly instead.
## Related in-tree implementation
`tools/web_tools.py` grew an in-process version of this idea (a `url -> [n]`
registry plus citation guidance attached to tool results) in PR #44833. That
path grounds ad-hoc web answers automatically when it lands. This skill is the
portable half: it works with any retrieval source (browser, curl, CLIs, local
PDFs) and it persists the ledger to disk so multi-turn, multi-file, and
multi-subagent work keeps stable ids. The two can coexist — the ledger is the
source of truth for anything written to a file.
grounded-citations/scripts/_hermes_home.py
+23
View File
@@ -0,0 +1,23 @@
"""Resolve HERMES_HOME for standalone skill scripts.
Skill scripts may run outside the Hermes process (system Python, nix env,
CI) where ``hermes_constants`` is not importable. This module provides the
same ``get_hermes_home()`` contract without requiring it on ``sys.path``.
When ``hermes_constants`` IS available it is used directly so profile
resolution and any future enhancements are picked up automatically.
"""
from __future__ import annotations
import os
from pathlib import Path
try:
from hermes_constants import get_hermes_home as get_hermes_home
except (ModuleNotFoundError, ImportError):
def get_hermes_home() -> Path:
"""Return the Hermes home directory (default: ``~/.hermes``)."""
val = os.environ.get("HERMES_HOME", "").strip()
return Path(val) if val else Path.home() / ".hermes"
grounded-citations/scripts/sources.py
+678
View File
@@ -0,0 +1,678 @@
#!/usr/bin/env python3
"""Citation ledger for grounded answers and documents.
Owns the ``url -> [n]`` mapping used by the ``grounded-citations`` skill.
Ids are assigned at retrieval time and never change, so a draft's ``[3]``
always resolves to the same page. The model only ever emits integers the
ledger handed it, which is what makes the citations verifiable.
Subcommands
-----------
reset start a clean ledger
add URL [URL ...] register source(s), print their ids
ingest FILE|- register every url found in JSON tool output
quote ID --text T --from FILE|- attach verbatim supporting evidence to a source
list show the ledger
render render a Sources block
verify DRAFT check a draft's citations against the ledger
Fact-checking is evidence-backed citation: ``quote`` only accepts text that
literally appears in the fetched page text you point it at, ``verify
--evidence`` requires every cited source to carry at least one such quote, and
``render --style evidence`` prints the quotes under each source so the reader
can check the chain themselves. Claims from model knowledge are declared with
an ``[unverified]`` marker rather than silently blended in.
Ledger path resolution (first wins):
--ledger PATH
$HERMES_CITATION_LEDGER
$HERMES_HOME/cache/citations/ledger.json
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
import time
from pathlib import Path
from typing import Any, Iterable
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _hermes_home import get_hermes_home # noqa: E402
SCHEMA_VERSION = 1
# A citation marker in prose: [12]. Markdown links ([text](url)) and
# reference-style labels are excluded by requiring digits only and no
# following "(" or ":".
_CITE_RE = re.compile(r"\[(\d{1,4})\](?![(:])")
_SOURCES_HEADER_RE = re.compile(r"^\s*(?:#{1,6}\s*)?(?:\*\*)?sources:?(?:\*\*)?\s*$", re.IGNORECASE)
_SOURCE_LINE_RE = re.compile(r"^\s*\[(\d{1,4})\]\s*[-–:]?\s*(\S+)")
_URL_IN_TEXT_RE = re.compile(r"https?://[^\s\"'<>)\]}]+")
_FENCE_RE = re.compile(r"^\s*(?:```|~~~)")
# Explicit declaration that a claim comes from model knowledge, not a source.
_UNVERIFIED_RE = re.compile(r"\[unverified\]", re.IGNORECASE)
# ---------------------------------------------------------------------------
# Ledger I/O
# ---------------------------------------------------------------------------
def resolve_ledger_path(explicit: str | None = None) -> Path:
if explicit:
return Path(explicit).expanduser()
env = os.environ.get("HERMES_CITATION_LEDGER", "").strip()
if env:
return Path(env).expanduser()
return get_hermes_home() / "cache" / "citations" / "ledger.json"
def normalize_url(url: str) -> str:
"""Canonicalize a URL for ledger identity.
Strips the fragment and a trailing slash so ``/page``, ``/page/`` and
``/page#section`` are one source. Query strings are significant and are
kept — they usually select different content.
"""
u = (url or "").strip()
if "#" in u:
u = u.split("#", 1)[0]
stripped = u.rstrip("/")
return stripped or u
def load_ledger(path: Path) -> dict[str, Any]:
if not path.exists():
return {"version": SCHEMA_VERSION, "sources": []}
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError) as exc:
raise SystemExit(f"error: ledger at {path} is unreadable ({exc}); run `reset` to start over")
if not isinstance(data, dict) or not isinstance(data.get("sources"), list):
raise SystemExit(f"error: ledger at {path} has an unexpected shape; run `reset`")
return data
def save_ledger(path: Path, data: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(path.suffix + f".tmp{os.getpid()}")
tmp.write_text(json.dumps(data, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
os.replace(tmp, path)
class _LedgerLock:
"""Best-effort cross-process lock (O_EXCL lockfile, stdlib only).
Parallel subagents can share one ledger via --ledger; without a lock two
concurrent ``add`` calls can assign the same id. Falls through after the
timeout rather than blocking a task forever — a stale lock must never
wedge the ledger.
"""
def __init__(self, path: Path, timeout: float = 5.0) -> None:
self.lock_path = path.with_suffix(path.suffix + ".lock")
self.timeout = timeout
self.fd: int | None = None
def __enter__(self) -> "_LedgerLock":
self.lock_path.parent.mkdir(parents=True, exist_ok=True)
deadline = time.monotonic() + self.timeout
while True:
try:
self.fd = os.open(str(self.lock_path), os.O_CREAT | os.O_EXCL | os.O_WRONLY)
return self
except FileExistsError:
if time.monotonic() >= deadline:
# Assume a stale lock from a crashed run.
try:
self.lock_path.unlink()
except OSError:
return self
continue
time.sleep(0.05)
def __exit__(self, *_exc: object) -> None:
if self.fd is not None:
try:
os.close(self.fd)
except OSError:
pass
try:
self.lock_path.unlink()
except OSError:
pass
# ---------------------------------------------------------------------------
# Core operations
# ---------------------------------------------------------------------------
def _by_url(sources: list[dict[str, Any]]) -> dict[str, dict[str, Any]]:
return {s["url"]: s for s in sources}
def add_sources(
path: Path,
urls: Iterable[str],
title: str | None = None,
accessed: str | None = None,
) -> list[dict[str, Any]]:
"""Register urls, returning their ledger entries (existing or new)."""
urls = [u for u in (str(u).strip() for u in urls) if u]
if not urls:
return []
with _LedgerLock(path):
data = load_ledger(path)
sources = data["sources"]
index = _by_url(sources)
out: list[dict[str, Any]] = []
changed = False
for raw in urls:
key = normalize_url(raw)
existing = index.get(key)
if existing is not None:
if title and not existing.get("title"):
existing["title"] = title
changed = True
out.append(existing)
continue
entry = {
"id": len(sources) + 1,
"url": key,
"title": (title or "").strip(),
"accessed": accessed or time.strftime("%Y-%m-%d"),
}
sources.append(entry)
index[key] = entry
out.append(entry)
changed = True
if changed:
save_ledger(path, data)
return out
def urls_from_json(payload: Any) -> list[tuple[str, str]]:
"""Walk arbitrary JSON tool output collecting (url, title) pairs.
Handles web_search (``data.web[]``), web_extract (``results[]``) and any
other nesting, in document order, deduped.
"""
found: list[tuple[str, str]] = []
seen: set[str] = set()
def walk(node: Any) -> None:
if isinstance(node, dict):
url = node.get("url") or node.get("link") or node.get("source_url")
if isinstance(url, str) and url.startswith(("http://", "https://")):
key = normalize_url(url)
if key not in seen:
seen.add(key)
raw_title = node.get("title") or node.get("name") or ""
found.append((url, raw_title if isinstance(raw_title, str) else ""))
for value in node.values():
walk(value)
elif isinstance(node, list):
for item in node:
walk(item)
walk(payload)
return found
# ---------------------------------------------------------------------------
# Evidence quotes (fact-checking)
# ---------------------------------------------------------------------------
def _normalize_ws(text: str) -> str:
"""Collapse all whitespace runs to single spaces for verbatim matching."""
return " ".join((text or "").split())
# Markdown artifacts that retrieval tools inject into otherwise-identical prose.
# ``web_extract`` returns markdown, so the most citation-worthy sentences are
# exactly the ones carrying inline links and emphasis around terms:
# "including _[ERAP1](https://…/erap1/)_, _[IL1A](…)_, have also been…"
# reads identically to the page a human sees. Matching has to see through that
# markup, or the skill pushes the agent toward weaker evidence fragments.
_MD_LINK_RE = re.compile(r"\[([^\]]*)\]\((?:[^()\s]|\([^()]*\))*\)")
_MD_NOISE_RE = re.compile(r"[*_`~]|\\(?=[^\w\s])")
def _match_key(text: str) -> str:
"""Canonicalize text for verbatim comparison.
Whitespace-, case-, and markdown-insensitive: inline links collapse to
their label, emphasis/code markers and backslash escapes are dropped. The
stored quote keeps whatever the caller passed, so the rendered evidence
block shows clean prose rather than extractor artifacts.
"""
collapsed = _MD_LINK_RE.sub(r"\1", text or "")
return _normalize_ws(_MD_NOISE_RE.sub("", collapsed)).casefold()
def quote_in_evidence(quote: str, evidence: str) -> bool:
"""True when ``quote`` appears verbatim in the fetched ``evidence`` text,
ignoring whitespace, case, and markdown markup on either side."""
q = _match_key(quote)
return bool(q) and q in _match_key(evidence)
def attach_quote(path: Path, source_id: int, quote: str, evidence: str) -> dict[str, Any]:
"""Attach a verbatim quote to a ledger entry after checking it against
the evidence text. Raises SystemExit on unknown id or non-verbatim text —
a quote the page does not contain is exactly the fabrication this guards
against."""
quote = (quote or "").strip()
if len(_normalize_ws(quote).split()) < 3:
raise SystemExit("error: quote too short — use at least 3 words of verbatim text")
if not quote_in_evidence(quote, evidence):
raise SystemExit(
"error: quote not found verbatim in the evidence text — "
"copy the exact wording from the fetched page, do not paraphrase"
)
with _LedgerLock(path):
data = load_ledger(path)
entry = next((s for s in data["sources"] if s["id"] == source_id), None)
if entry is None:
raise SystemExit(f"error: no source [{source_id}] in the ledger")
quotes = entry.setdefault("quotes", [])
norm = _match_key(quote)
if not any(_match_key(q.get("text", "")) == norm for q in quotes):
quotes.append({"text": quote, "added": time.strftime("%Y-%m-%d")})
save_ledger(path, data)
return entry
def render_sources(
sources: list[dict[str, Any]],
style: str = "markdown",
only: set[int] | None = None,
) -> str:
picked = [s for s in sources if only is None or s["id"] in only]
picked.sort(key=lambda s: s["id"])
if not picked:
return ""
lines: list[str] = []
if style == "bibtex":
for s in picked:
key = f"source{s['id']}"
title = s.get("title") or s["url"]
lines.append(
"@misc{%s,\n title = {%s},\n howpublished = {\\url{%s}},\n note = {Accessed %s}\n}"
% (key, title, s["url"], s.get("accessed", ""))
)
return "\n".join(lines)
if style == "footnotes":
for s in picked:
title = s.get("title")
suffix = f" — {title}" if title else ""
lines.append(f"[^{s['id']}]: {s['url']}{suffix}")
return "\n".join(lines)
header = "Sources:" if style == "plain" else "## Sources"
lines.append(header)
if style != "plain":
lines.append("")
for s in picked:
title = s.get("title")
suffix = f" — {title}" if title else ""
lines.append(f"[{s['id']}] {s['url']}{suffix}")
if style == "evidence":
for q in s.get("quotes", []):
lines.append(f' > "{q.get("text", "")}"')
return "\n".join(lines)
# ---------------------------------------------------------------------------
# Verification
# ---------------------------------------------------------------------------
def _split_draft(text: str) -> tuple[str, dict[int, str]]:
"""Split a draft into (prose, sources_block_map).
The sources block is everything after the last ``Sources`` header; its
``[n] url`` lines are parsed out so they aren't counted as prose citations.
Fenced code blocks are dropped from prose.
"""
lines = text.splitlines()
header_idx = -1
for i, line in enumerate(lines):
if _SOURCES_HEADER_RE.match(line):
header_idx = i
listed: dict[int, str] = {}
if header_idx >= 0:
for line in lines[header_idx + 1:]:
m = _SOURCE_LINE_RE.match(line)
if m:
url_match = _URL_IN_TEXT_RE.search(line)
listed[int(m.group(1))] = url_match.group(0) if url_match else m.group(2)
body_lines = lines[:header_idx]
else:
body_lines = lines
prose: list[str] = []
in_fence = False
for line in body_lines:
if _FENCE_RE.match(line):
in_fence = not in_fence
continue
if not in_fence:
prose.append(line)
return "\n".join(prose), listed
def _strip_sources_block(text: str) -> str:
"""Return the draft with its trailing Sources block removed.
Everything from the last Sources header onward goes; a draft with no such
header is returned unchanged. This is what makes ``render --replace-in``
idempotent instead of stacking duplicate blocks.
"""
lines = text.splitlines()
header_idx = -1
for i, line in enumerate(lines):
if _SOURCES_HEADER_RE.match(line):
header_idx = i
if header_idx < 0:
return text
return "\n".join(lines[:header_idx])
def _sentences(prose: str) -> list[str]:
"""Rough sentence split over prose lines, skipping headings and tables."""
out: list[str] = []
for line in prose.splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#") or stripped.startswith("|"):
continue
if stripped.startswith(">"):
stripped = stripped.lstrip("> ").strip()
for part in re.split(r"(?<=[.!?])\s+", stripped):
part = part.strip()
if len(part.split()) >= 4:
out.append(part)
return out
def verify_draft(
draft_path: Path,
sources: list[dict[str, Any]],
strict: bool = False,
min_coverage: float | None = None,
require_evidence: bool = False,
) -> tuple[int, list[str], list[str]]:
"""Return (exit_code, errors, warnings)."""
text = draft_path.read_text(encoding="utf-8")
prose, listed = _split_draft(text)
by_id = {s["id"]: s for s in sources}
errors: list[str] = []
warnings: list[str] = []
cited = [int(m) for m in _CITE_RE.findall(prose)]
cited_set = set(cited)
unknown = sorted(i for i in cited_set if i not in by_id)
if unknown:
errors.append(
"citations not in the ledger (hallucinated or renumbered): "
+ ", ".join(f"[{i}]" for i in unknown)
)
if cited_set and not listed:
errors.append("draft cites sources but has no `Sources:` block — run `render --cited-in`")
missing_from_block = sorted(cited_set - set(listed)) if listed else []
if missing_from_block:
errors.append(
"cited but absent from the Sources block: "
+ ", ".join(f"[{i}]" for i in missing_from_block)
)
for sid, url in sorted(listed.items()):
entry = by_id.get(sid)
if entry is None:
errors.append(f"Sources block lists [{sid}], which is not in the ledger")
continue
if normalize_url(url) != entry["url"]:
errors.append(
f"Sources block URL for [{sid}] does not match the ledger "
f"(block: {url} / ledger: {entry['url']}) — re-run `render`"
)
extra_in_block = sorted(set(listed) - cited_set)
if extra_in_block:
warnings.append(
"listed in Sources but never cited inline: "
+ ", ".join(f"[{i}]" for i in extra_in_block)
)
registered_uncited = sorted(set(by_id) - cited_set)
if registered_uncited:
warnings.append(
"registered in the ledger but not cited in this draft: "
+ ", ".join(f"[{i}]" for i in registered_uncited)
)
sentences = _sentences(prose)
cited_sentences = [s for s in sentences if _CITE_RE.search(s)]
unverified_sentences = [s for s in sentences if _UNVERIFIED_RE.search(s)]
covered = [s for s in sentences if _CITE_RE.search(s) or _UNVERIFIED_RE.search(s)]
coverage = (len(covered) / len(sentences)) if sentences else 0.0
if min_coverage is not None and sentences and coverage < min_coverage:
errors.append(
f"citation coverage {coverage:.0%} is below the required {min_coverage:.0%} "
f"({len(covered)}/{len(sentences)} sentences cited or marked [unverified])"
)
if require_evidence:
unevidenced = sorted(
i for i in cited_set if i in by_id and not by_id[i].get("quotes")
)
if unevidenced:
errors.append(
"cited sources carry no verbatim evidence quote (run `quote` with the "
"fetched page text): " + ", ".join(f"[{i}]" for i in unevidenced)
)
over_cited = [s for s in sentences if len(_CITE_RE.findall(s)) > 3]
if over_cited:
warnings.append(f"{len(over_cited)} sentence(s) carry more than 3 citations")
code = 1 if errors else (1 if (strict and warnings) else 0)
quoted = sum(1 for s in sources if s.get("quotes"))
stats = (
f"{len(sentences)} prose sentence(s), {len(covered)} with declared provenance "
f"({coverage:.0%}) — {len(cited_sentences)} cited, "
f"{len(unverified_sentences)} marked [unverified] (a sentence may be both); "
f"{len(cited_set)} distinct source(s) cited, "
f"{len(by_id)} in ledger ({quoted} with evidence quotes)"
)
warnings.insert(0, f"stats: {stats}")
return code, errors, warnings
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def _parse_only(spec: str | None) -> set[int] | None:
if not spec:
return None
out: set[int] = set()
for chunk in spec.replace(" ", "").split(","):
if not chunk:
continue
if "-" in chunk:
lo, _, hi = chunk.partition("-")
out.update(range(int(lo), int(hi) + 1))
else:
out.add(int(chunk))
return out
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="sources.py", description="Citation ledger for grounded answers and documents."
)
parser.add_argument("--ledger", help="ledger file path (overrides env / default)")
sub = parser.add_subparsers(dest="cmd", required=True)
sub.add_parser("reset", help="start a clean ledger")
p_add = sub.add_parser("add", help="register source url(s), print their ids")
p_add.add_argument("urls", nargs="+")
p_add.add_argument("--title", help="title for the source (single-url calls)")
p_add.add_argument("--accessed", help="access date (default: today)")
p_add.add_argument("--json", action="store_true", help="emit JSON instead of ids")
p_ing = sub.add_parser("ingest", help="register every url in JSON tool output")
p_ing.add_argument("file", help="JSON file, or - for stdin")
p_q = sub.add_parser("quote", help="attach verbatim supporting evidence to a source")
p_q.add_argument("id", type=int, help="ledger id of the source the quote supports")
p_q.add_argument("--text", required=True, help="the exact quote, copied from the page")
p_q.add_argument(
"--from",
dest="evidence",
required=True,
help="file with the fetched page text (or - for stdin) the quote must appear in",
)
p_list = sub.add_parser("list", help="show the ledger")
p_list.add_argument("--json", action="store_true")
p_render = sub.add_parser("render", help="render a Sources block")
p_render.add_argument(
"--style", default="markdown", choices=["markdown", "plain", "footnotes", "bibtex", "evidence"]
)
p_render.add_argument("--only", help="ids to include, e.g. 1,3,5-7")
p_render.add_argument("--cited-in", help="include only ids cited in this draft file")
p_render.add_argument(
"--replace-in",
help="rewrite this draft's Sources block in place (implies --cited-in on it)",
)
p_ver = sub.add_parser("verify", help="check a draft's citations against the ledger")
p_ver.add_argument("draft")
p_ver.add_argument("--strict", action="store_true", help="treat warnings as failures")
p_ver.add_argument("--min-coverage", type=float, help="required cited-sentence share, e.g. 0.5")
p_ver.add_argument(
"--evidence",
action="store_true",
help="require every cited source to carry at least one verbatim quote",
)
args = parser.parse_args(argv)
path = resolve_ledger_path(args.ledger)
if args.cmd == "reset":
with _LedgerLock(path):
save_ledger(path, {"version": SCHEMA_VERSION, "sources": []})
print(f"ledger reset: {path}")
return 0
if args.cmd == "add":
title = args.title if len(args.urls) == 1 else None
entries = add_sources(path, args.urls, title=title, accessed=args.accessed)
if args.json:
print(json.dumps(entries, indent=2, ensure_ascii=False))
else:
for e in entries:
print(f"[{e['id']}] {e['url']}")
return 0
if args.cmd == "ingest":
raw = sys.stdin.read() if args.file == "-" else Path(args.file).read_text(encoding="utf-8")
try:
payload = json.loads(raw)
except json.JSONDecodeError as exc:
print(f"error: input is not valid JSON ({exc})", file=sys.stderr)
return 2
pairs = urls_from_json(payload)
if not pairs:
print("no urls found in input", file=sys.stderr)
return 1
for url, title in pairs:
entry = add_sources(path, [url], title=title or None)[0]
print(f"[{entry['id']}] {entry['url']}")
return 0
if args.cmd == "quote":
raw = (
sys.stdin.read()
if args.evidence == "-"
else Path(args.evidence).read_text(encoding="utf-8")
)
entry = attach_quote(path, args.id, args.text, raw)
print(f"[{entry['id']}] evidence attached ({len(entry.get('quotes', []))} quote(s))")
return 0
data = load_ledger(path)
sources = sorted(data["sources"], key=lambda s: s["id"])
if args.cmd == "list":
if args.json:
print(json.dumps(sources, indent=2, ensure_ascii=False))
elif not sources:
print(f"ledger is empty: {path}")
else:
for s in sources:
title = f" {s['title']}" if s.get("title") else ""
nq = len(s.get("quotes", []))
mark = f" ({nq} quote{'s' if nq != 1 else ''})" if nq else ""
print(f"[{s['id']}] {s['url']}{title}{mark}")
return 0
if args.cmd == "render":
only = _parse_only(args.only)
draft_for_ids = args.replace_in or args.cited_in
if draft_for_ids:
draft = Path(draft_for_ids).read_text(encoding="utf-8")
prose, _ = _split_draft(draft)
cited = {int(m) for m in _CITE_RE.findall(prose)}
only = cited if only is None else (only & cited)
block = render_sources(sources, style=args.style, only=only)
if not block:
print("no sources to render", file=sys.stderr)
return 1
if args.replace_in:
target = Path(args.replace_in)
body = _strip_sources_block(target.read_text(encoding="utf-8"))
target.write_text(body.rstrip("\n") + "\n\n" + block + "\n", encoding="utf-8")
print(f"Sources block rewritten in {target}")
return 0
print(block)
return 0
if args.cmd == "verify":
draft_path = Path(args.draft)
if not draft_path.is_file():
print(f"error: no such draft: {draft_path}", file=sys.stderr)
return 2
code, errors, warnings = verify_draft(
draft_path,
sources,
strict=args.strict,
min_coverage=args.min_coverage,
require_evidence=args.evidence,
)
for w in warnings:
prefix = "info" if w.startswith("stats: ") else "warn"
print(f"{prefix}: {w}")
for e in errors:
print(f"FAIL: {e}", file=sys.stderr)
print("citations OK" if code == 0 else "verification failed")
return code
return 2
if __name__ == "__main__":
sys.exit(main())