commit aad5c86543af6e90cdbe3780f826377a2328cb13 Author: figmar Date: Sun Aug 9 08:19:26 2026 +0800 Publish hermes-skills-productivity via gitea-publish skill diff --git a/airtable/SKILL.md b/airtable/SKILL.md new file mode 100644 index 0000000..3fa1b0a --- /dev/null +++ b/airtable/SKILL.md @@ -0,0 +1,229 @@ +--- +name: airtable +description: Airtable REST API via curl. Records CRUD, filters, upserts. +version: 1.1.0 +author: community +license: MIT +platforms: [linux, macos, windows] +prerequisites: + env_vars: [AIRTABLE_API_KEY] + commands: [curl] +metadata: + hermes: + tags: [Airtable, Productivity, Database, API] + homepage: https://airtable.com/developers/web/api/introduction +--- + +# Airtable — Bases, Tables & Records + +Work with Airtable's REST API directly via `curl` using the `terminal` tool. No MCP server, no OAuth flow, no Python SDK — just `curl` and a personal access token. + +## Prerequisites + +1. Create a **Personal Access Token (PAT)** at https://airtable.com/create/tokens (tokens start with `pat...`). +2. Grant these scopes (minimum): + - `data.records:read` — read rows + - `data.records:write` — create / update / delete rows + - `schema.bases:read` — list bases and tables +3. **Important:** in the same token UI, add each base you want to access to the token's **Access** list. PATs are scoped per-base — a valid token on the wrong base returns `403`. +4. Store the token in `${HERMES_HOME:-~/.hermes}/.env` (or via `hermes setup`): + ``` + AIRTABLE_API_KEY=pat_your_token_here + ``` + +> Note: legacy `key...` API keys were deprecated Feb 2024. Only PATs and OAuth tokens work now. + +## API Basics + +- **Endpoint:** `https://api.airtable.com/v0` +- **Auth header:** `Authorization: Bearer $AIRTABLE_API_KEY` +- **All requests** use JSON (`Content-Type: application/json` for any POST/PATCH/PUT body). +- **Object IDs:** bases `app...`, tables `tbl...`, records `rec...`, fields `fld...`. IDs never change; names can. Prefer IDs in automations. +- **Rate limit:** 5 requests/sec/base. `429` → back off. Burst on a single base will be throttled. + +Base curl pattern: +```bash +curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?maxRecords=5" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` + +`-s` suppresses curl's progress bar — keep it set for every call so the tool output stays clean for Hermes. Pipe through `python3 -m json.tool` (always present) or `jq` (if installed) for readable JSON. + +## Field Types (request body shapes) + +| Field type | Write shape | +|---|---| +| Single line text | `"Name": "hello"` | +| Long text | `"Notes": "multi\nline"` | +| Number | `"Score": 42` | +| Checkbox | `"Done": true` | +| Single select | `"Status": "Todo"` (name must already exist unless `typecast: true`) | +| Multi-select | `"Tags": ["urgent", "bug"]` | +| Date | `"Due": "2026-04-01"` | +| DateTime (UTC) | `"At": "2026-04-01T14:30:00.000Z"` | +| URL / Email / Phone | `"Link": "https://…"` | +| Attachment | `"Files": [{"url": "https://…"}]` (Airtable fetches + rehosts) | +| Linked record | `"Owner": ["recXXXXXXXXXXXXXX"]` (array of record IDs) | +| User | `"AssignedTo": {"id": "usrXXXXXXXXXXXXXX"}` | + +Pass `"typecast": true` at the top level of a create/update body to let Airtable auto-coerce values (e.g. create a new select option on the fly, convert `"42"` → `42`). + +## Common Queries + +### List bases the token can see +```bash +curl -s "https://api.airtable.com/v0/meta/bases" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` + +### List tables + schema for a base +```bash +curl -s "https://api.airtable.com/v0/meta/bases/$BASE_ID/tables" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` +Use this BEFORE mutating — confirms exact field names and IDs, surfaces `options.choices` for select fields, and shows primary-field names. + +### List records (first 10) +```bash +curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?maxRecords=10" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` + +### Get a single record +```bash +curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE/$RECORD_ID" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` + +### Filter records (filterByFormula) +Airtable formulas must be URL-encoded. Let Python stdlib do it — never hand-encode: +```bash +FORMULA="{Status}='Todo'" +ENC=$(python3 -c 'import sys, urllib.parse; print(urllib.parse.quote(sys.argv[1], safe=""))' "$FORMULA") +curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?filterByFormula=$ENC&maxRecords=20" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` + +Useful formula patterns: +- Exact match: `{Email}='user@example.com'` +- Contains: `FIND('bug', LOWER({Title}))` +- Multiple conditions: `AND({Status}='Todo', {Priority}='High')` +- Or: `OR({Owner}='alice', {Owner}='bob')` +- Not empty: `NOT({Assignee}='')` +- Date comparison: `IS_AFTER({Due}, TODAY())` + +### Sort + select specific fields +```bash +curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?sort%5B0%5D%5Bfield%5D=Priority&sort%5B0%5D%5Bdirection%5D=asc&fields%5B%5D=Name&fields%5B%5D=Status" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` +Square brackets in query params MUST be URL-encoded (`%5B` / `%5D`). + +### Use a named view +```bash +curl -s "https://api.airtable.com/v0/$BASE_ID/$TABLE?view=Grid%20view&maxRecords=50" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` +Views apply their saved filter + sort server-side. + +## Common Mutations + +### Create a record +```bash +curl -s -X POST "https://api.airtable.com/v0/$BASE_ID/$TABLE" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"fields":{"Name":"New task","Status":"Todo","Priority":"High"}}' | python3 -m json.tool +``` + +### Create up to 10 records in one call +```bash +curl -s -X POST "https://api.airtable.com/v0/$BASE_ID/$TABLE" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "typecast": true, + "records": [ + {"fields": {"Name": "Task A", "Status": "Todo"}}, + {"fields": {"Name": "Task B", "Status": "In progress"}} + ] + }' | python3 -m json.tool +``` +Batch endpoints are capped at **10 records per request**. For larger inserts, loop in batches of 10 with a short sleep to respect 5 req/sec/base. + +### Update a record (PATCH — merges, preserves unchanged fields) +```bash +curl -s -X PATCH "https://api.airtable.com/v0/$BASE_ID/$TABLE/$RECORD_ID" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"fields":{"Status":"Done"}}' | python3 -m json.tool +``` + +### Upsert by a merge field (no ID needed) +```bash +curl -s -X PATCH "https://api.airtable.com/v0/$BASE_ID/$TABLE" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "performUpsert": {"fieldsToMergeOn": ["Email"]}, + "records": [ + {"fields": {"Email": "user@example.com", "Status": "Active"}} + ] + }' | python3 -m json.tool +``` +`performUpsert` creates records whose merge-field values are new, patches records whose merge-field values already exist. Great for idempotent syncs. + +### Delete a record +```bash +curl -s -X DELETE "https://api.airtable.com/v0/$BASE_ID/$TABLE/$RECORD_ID" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` + +### Delete up to 10 records in one call +```bash +curl -s -X DELETE "https://api.airtable.com/v0/$BASE_ID/$TABLE?records%5B%5D=rec1&records%5B%5D=rec2" \ + -H "Authorization: Bearer $AIRTABLE_API_KEY" | python3 -m json.tool +``` + +## Pagination + +List endpoints return at most **100 records per page**. If the response includes `"offset": "..."`, pass it back on the next call. Loop until the field is absent: + +```bash +OFFSET="" +while :; do + URL="https://api.airtable.com/v0/$BASE_ID/$TABLE?pageSize=100" + [ -n "$OFFSET" ] && URL="$URL&offset=$OFFSET" + RESP=$(curl -s "$URL" -H "Authorization: Bearer $AIRTABLE_API_KEY") + echo "$RESP" | python3 -c 'import json,sys; d=json.load(sys.stdin); [print(r["id"], r["fields"].get("Name","")) for r in d["records"]]' + OFFSET=$(echo "$RESP" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("offset",""))') + [ -z "$OFFSET" ] && break +done +``` + +## Typical Hermes Workflow + +1. **Confirm auth.** `curl -s -o /dev/null -w "%{http_code}\n" https://api.airtable.com/v0/meta/bases -H "Authorization: Bearer $AIRTABLE_API_KEY"` — expect `200`. +2. **Find the base.** List bases (step above) OR ask the user for the `app...` ID directly if the token lacks `schema.bases:read`. +3. **Inspect the schema.** `GET /v0/meta/bases/$BASE_ID/tables` — cache the exact field names and primary-field name locally in the session before mutating anything. +4. **Read before you write.** For "update X where Y", `filterByFormula` first to resolve the `rec...` ID, then `PATCH /v0/$BASE_ID/$TABLE/$RECORD_ID`. Never guess record IDs. +5. **Batch writes.** Combine related creates into one 10-record POST to stay under the 5 req/sec budget. +6. **Destructive ops.** Deletions can't be undone via API. If the user says "delete all Xs", echo back the filter + record count and confirm before firing. + +## Pitfalls + +- **`filterByFormula` MUST be URL-encoded.** Field names with spaces or non-ASCII also need encoding (`{My Field}` → `%7BMy%20Field%7D`). Use Python stdlib (pattern above) — never hand-escape. +- **Empty fields are omitted from responses.** A missing `"Assignee"` key doesn't mean the field doesn't exist — it means this record's value is empty. Check the schema (step 3) before concluding a field is missing. +- **PATCH vs PUT.** `PATCH` merges supplied fields into the record. `PUT` replaces the record entirely and clears any field you didn't include. Default to `PATCH`. +- **Single-select options must exist.** Writing `"Status": "Shipping"` when `Shipping` isn't in the field's option list errors with `INVALID_MULTIPLE_CHOICE_OPTIONS` unless you pass `"typecast": true` (which auto-creates the option). +- **Per-base token scoping.** A `403` on one base while another works means the token's Access list doesn't include that base — not a scope or auth issue. Send the user to https://airtable.com/create/tokens to grant it. +- **Rate limits are per base, not per token.** 5 req/sec on `baseA` and 5 req/sec on `baseB` is fine; 6 req/sec on `baseA` alone will throttle. Monitor the `Retry-After` header on `429`. + +## Important Notes for Hermes + +- **Always use the `terminal` tool with `curl`.** Do NOT use `web_extract` (it can't send auth headers) or `browser_navigate` (needs UI auth and is slow). +- **`AIRTABLE_API_KEY` flows from `${HERMES_HOME:-~/.hermes}/.env` into the subprocess automatically** when this skill is loaded — no need to re-export it before each `curl` call. +- **Escape curly braces in formulas carefully.** In a heredoc body, `{Status}` is literal. In a shell argument, `{Status}` is safe outside `{...}` brace-expansion context — but pass dynamic strings through `python3 urllib.parse.quote` before splicing into a URL. +- **Pretty-print with `python3 -m json.tool`** (always present) rather than `jq` (optional). Only reach for `jq` when you need filtering/projection. +- **Pagination is per-page, not global.** Airtable's 100-record cap is a hard limit; there is no way to bump it. Loop with `offset` until the field is absent. +- **Read the `errors` array** on non-2xx responses — Airtable returns structured error codes like `AUTHENTICATION_REQUIRED`, `INVALID_PERMISSIONS`, `MODEL_ID_NOT_FOUND`, `INVALID_MULTIPLE_CHOICE_OPTIONS` that tell you exactly what's wrong. diff --git a/docx/LICENSE.txt b/docx/LICENSE.txt new file mode 100644 index 0000000..c55ab42 --- /dev/null +++ b/docx/LICENSE.txt @@ -0,0 +1,30 @@ +© 2025 Anthropic, PBC. All rights reserved. + +LICENSE: Use of these materials (including all code, prompts, assets, files, +and other components of this Skill) is governed by your agreement with +Anthropic regarding use of Anthropic's services. If no separate agreement +exists, use is governed by Anthropic's Consumer Terms of Service or +Commercial Terms of Service, as applicable: +https://www.anthropic.com/legal/consumer-terms +https://www.anthropic.com/legal/commercial-terms +Your applicable agreement is referred to as the "Agreement." "Services" are +as defined in the Agreement. + +ADDITIONAL RESTRICTIONS: Notwithstanding anything in the Agreement to the +contrary, users may not: + +- Extract these materials from the Services or retain copies of these + materials outside the Services +- Reproduce or copy these materials, except for temporary copies created + automatically during authorized use of the Services +- Create derivative works based on these materials +- Distribute, sublicense, or transfer these materials to any third party +- Make, offer to sell, sell, or import any inventions embodied in these + materials +- Reverse engineer, decompile, or disassemble these materials + +The receipt, viewing, or possession of these materials does not convey or +imply any license or right beyond those expressly granted above. + +Anthropic retains all right, title, and interest in these materials, +including all copyrights, patents, and other intellectual property rights. diff --git a/docx/SKILL.md b/docx/SKILL.md new file mode 100644 index 0000000..01ffede --- /dev/null +++ b/docx/SKILL.md @@ -0,0 +1,127 @@ +--- +name: docx +description: "Create, read, edit Word .docx documents and templates." +version: 1.0.0 +author: Anthropic (adapted by Nous Research) +license: Proprietary. LICENSE.txt has complete terms +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [Word, DOCX, Documents, Office, Productivity] + category: productivity + related_skills: [pdf, xlsx, powerpoint, ocr-and-documents] +--- + +# DOCX Skill + +Create, read, and edit Word documents — reports, memos, letters, letterheads, tables of contents, tracked changes (redlining), and comments. A `.docx` is a ZIP archive of XML files; this skill covers both the high-level creation path and surgical XML editing. + +## When to Use + +Use this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx) or Word templates (.dotx). Triggers include: any mention of "Word doc", ".docx", ".dotx", or requests for a "report", "memo", "letter", or similar deliverable as a Word file; extracting or reorganizing content from .docx files; find-and-replace in Word files; inserting images; tracked changes or comments. Do NOT use for PDFs (see the `pdf` skill), spreadsheets (`xlsx`), or presentations (`powerpoint`). + +## Prerequisites + +```bash +npm ls docx --depth=0 2>/dev/null | grep -q docx || npm install docx # creation (docx-js) +pip show pandoc >/dev/null 2>&1 || true; which pandoc || sudo apt install -y pandoc # reading +which soffice || sudo apt install -y libreoffice # rendering/verification +which pdftoppm || sudo apt install -y poppler-utils # PDF → images +pip install defusedxml lxml # validation scripts +``` + +macOS: `brew install pandoc libreoffice poppler`. + +## Quick Reference + +| Task | Approach | +|---|---| +| **Create** a new document | Write a `docx` (npm) script — see gotchas below | +| **Edit** an existing document | `unzip` → edit `word/document.xml` → `zip` (docx-js cannot open existing files) | +| **Read** content | `pandoc -t markdown file.docx` (or `read_file`, which auto-extracts .docx text) | + +> Script paths below are relative to this skill's directory. + +## Creating with docx-js — gotchas + +Write the script and `require('docx')`. The model knows the API; these are the footguns: + +- **Page size defaults to A4.** For US Letter set `page: { size: { width: 12240, height: 15840 } }` (DXA; 1440 = 1″). +- **Landscape:** pass portrait dimensions and `orientation: PageOrientation.LANDSCAPE` — docx-js swaps width/height internally. +- **Tables need dual widths:** set `columnWidths` on the table AND `width` on every cell, both in `WidthType.DXA` (PERCENTAGE breaks in Google Docs). Column widths must sum to the table width. +- **Table shading:** use `ShadingType.CLEAR`, never `SOLID` (renders black). +- **Lists:** never insert `•` literally; use a `numbering` config with `LevelFormat.BULLET`. +- **`ImageRun` requires `type:`** (`"png"`, `"jpg"`, …). +- **`PageBreak` must be inside a `Paragraph`.** +- **Never use `\n`** — use separate `Paragraph` elements. +- **TOC:** headings must use built-in `HeadingLevel.*`; custom heading styles need `outlineLevel` set or they won't appear. +- **Don't use a table as a horizontal rule** — use a paragraph bottom border instead. +- **Dot-leader / right-aligned-on-same-line:** use `PositionalTab` (`alignment: PositionalTabAlignment.RIGHT`, `leader: PositionalTabLeader.DOT`) inside a `TextRun`, not literal `.` or space padding. + +## Verify the output + +After writing a `.docx`, render it and look at it: + +```bash +python scripts/office/soffice.py --headless --convert-to pdf output.docx +pdftoppm -jpeg -r 100 output.pdf page +ls page-*.jpg # then inspect each with vision_analyze +``` + +`pdftoppm` zero-pads page numbers to the width of the page count (`page-01.jpg`…`page-12.jpg`). + +## Editing existing documents + +Legacy `.doc` files must be converted first: `python scripts/office/soffice.py --headless --convert-to docx file.doc`. + +```bash +unzip -q doc.docx -d unpacked/ +find unpacked -type l -delete # strip symlink entries — docx from external parties is untrusted +python scripts/merge_runs.py unpacked/ # coalesce fragmented runs so text is findable +# edit unpacked/word/document.xml in place — do NOT reformat or pretty-print +(cd unpacked && rm -f ../out.docx && zip -Xr ../out.docx .) +python scripts/office/validate.py out.docx --original doc.docx # XSD checks; --auto-repair fixes common issues +# redlining? add --author "" to check every edit is tracked +``` + +Word splits text across many `` runs (revision ids, spell-check markers), so a phrase you can see in the document often doesn't exist as a contiguous string in the XML. `merge_runs.py` merges adjacent identically-formatted runs in `word/document.xml` without changing content or rendering; it also accepts a `.docx` directly (`python scripts/merge_runs.py doc.docx -o merged.docx`). + +**Tracked changes:** when redlining, validate with `--author ""` (needs `--original`) — it reports any text you changed without a ``/`` around it, which is easy to do by accident and invisible in the accepted view. Wrap runs in ``/`` with `w:id`, `w:author`, `w:date` attributes. Inside ``, the text element is ``, not ``. A deleted paragraph mark (``) means "merge this paragraph into the next" — so deleting a paragraph outright is that plus a `` around every run. The `` must come before the rPr's other children; their order is schema-enforced. + +To produce a clean copy with all tracked changes accepted: `python scripts/accept_changes.py in.docx out.docx`. + +Accepting a deleted paragraph mark should join that paragraph to the one below it, so a paragraph whose runs are *all* deleted vanishes. Word does this; `accept_changes.py` and `pandoc --track-changes=accept` don't always. Both fail the same way — they strip the deleted text but leave the emptied paragraph behind, which reads as a stray empty bullet when it was auto-numbered: + +- `pandoc --track-changes=accept` never joins the paragraphs. +- `accept_changes.py` (LibreOffice) joins them correctly, except when the deleted paragraph is followed by an empty spacer paragraph. + +An empty bullet in either view is an artifact of that view, not a defect in the document. Check paragraph deletions in the XML. + +## Comments + +Comments require six cross-linked files. Use the helper — directory mode when you'll also be editing `document.xml` (saves an unzip/rezip cycle), `.docx`-direct mode otherwise: + +```bash +# Against an already-unpacked directory (preferred when also placing markers) +python scripts/comment.py unpacked/ "Fees & expenses cap is too low" +python scripts/comment.py unpacked/ "Agreed" --parent 0 + +# Against a .docx directly +python scripts/comment.py contract.docx "This cap is too low" -o annotated.docx +``` + +The script writes `comments.xml`, `commentsExtended.xml`, `commentsIds.xml`, `commentsExtensible.xml`, the relationships, and the content-type overrides. Comment IDs are auto-assigned. It then prints the ``/``/`` snippet to add to `word/document.xml` so the comment anchors to specific text — until you place those markers, the comment exists but is not visible. + +## Pitfalls + +- Don't round-trip OOXML through `xml.etree.ElementTree` — it rewrites namespace prefixes and corrupts the file. Use `defusedxml.minidom` for scripted transforms. +- Zip from INSIDE the unpacked directory (`cd unpacked && zip -Xr ../out.docx .`) and `rm` the target first, or deleted parts survive in the archive. + +## Verification + +1. `python scripts/office/validate.py out.docx --original in.docx` — schema, relationship, and content-type checks; every failure names its fix. +2. Render to PDF → images (see "Verify the output") and inspect each page with `vision_analyze` — look for broken tables, missing images, spacing artifacts, leftover placeholder text. + +## Related skills + +`pdf` (PDF work), `xlsx` (spreadsheets), `powerpoint` (decks), `ocr-and-documents` (scanned input extraction). diff --git a/docx/scripts/__init__.py b/docx/scripts/__init__.py new file mode 100755 index 0000000..8b13789 --- /dev/null +++ b/docx/scripts/__init__.py @@ -0,0 +1 @@ + diff --git a/docx/scripts/accept_changes.py b/docx/scripts/accept_changes.py new file mode 100755 index 0000000..146f09c --- /dev/null +++ b/docx/scripts/accept_changes.py @@ -0,0 +1,135 @@ +"""Accept all tracked changes in a DOCX file using LibreOffice. + +Requires LibreOffice (soffice) to be installed. +""" + +import argparse +import logging +import shutil +import subprocess +from pathlib import Path + +from office.soffice import get_soffice_env + +logger = logging.getLogger(__name__) + +LIBREOFFICE_PROFILE = "/tmp/libreoffice_docx_profile" +MACRO_DIR = f"{LIBREOFFICE_PROFILE}/user/basic/Standard" + +ACCEPT_CHANGES_MACRO = """ + + + Sub AcceptAllTrackedChanges() + Dim document As Object + Dim dispatcher As Object + + document = ThisComponent.CurrentController.Frame + dispatcher = createUnoService("com.sun.star.frame.DispatchHelper") + + dispatcher.executeDispatch(document, ".uno:AcceptAllTrackedChanges", "", 0, Array()) + ThisComponent.store() + ThisComponent.close(True) + End Sub +""" + + +def accept_changes( + input_file: str, + output_file: str, +) -> tuple[None, str]: + input_path = Path(input_file) + output_path = Path(output_file) + + if not input_path.exists(): + return None, f"Error: Input file not found: {input_file}" + + if not input_path.suffix.lower() == ".docx": + return None, f"Error: Input file is not a DOCX file: {input_file}" + + try: + output_path.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(input_path, output_path) + except Exception as e: + return None, f"Error: Failed to copy input file to output location: {e}" + + if not _setup_libreoffice_macro(): + return None, "Error: Failed to setup LibreOffice macro" + + cmd = [ + "soffice", + "--headless", + f"-env:UserInstallation=file://{LIBREOFFICE_PROFILE}", + "--norestore", + "vnd.sun.star.script:Standard.Module1.AcceptAllTrackedChanges?language=Basic&location=application", + str(output_path.absolute()), + ] + + try: + result = subprocess.run( + cmd, + capture_output=True, + text=True, encoding="utf-8", errors="replace", + timeout=30, + check=False, + env=get_soffice_env(), + ) + except subprocess.TimeoutExpired: + return ( + None, + f"Successfully accepted all tracked changes: {input_file} -> {output_file}", + ) + + if result.returncode != 0: + return None, f"Error: LibreOffice failed: {result.stderr}" + + return ( + None, + f"Successfully accepted all tracked changes: {input_file} -> {output_file}", + ) + + +def _setup_libreoffice_macro() -> bool: + macro_dir = Path(MACRO_DIR) + macro_file = macro_dir / "Module1.xba" + + if macro_file.exists() and "AcceptAllTrackedChanges" in macro_file.read_text(encoding="utf-8"): + return True + + if not macro_dir.exists(): + subprocess.run( + [ + "soffice", + "--headless", + f"-env:UserInstallation=file://{LIBREOFFICE_PROFILE}", + "--terminate_after_init", + ], + capture_output=True, + timeout=10, + check=False, + env=get_soffice_env(), + ) + macro_dir.mkdir(parents=True, exist_ok=True) + + try: + macro_file.write_text(ACCEPT_CHANGES_MACRO, encoding="utf-8") + return True + except Exception as e: + logger.warning(f"Failed to setup LibreOffice macro: {e}") + return False + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Accept all tracked changes in a DOCX file" + ) + parser.add_argument("input_file", help="Input DOCX file with tracked changes") + parser.add_argument( + "output_file", help="Output DOCX file (clean, no tracked changes)" + ) + args = parser.parse_args() + + _, message = accept_changes(args.input_file, args.output_file) + print(message) + + if "Error" in message: + raise SystemExit(1) diff --git a/docx/scripts/comment.py b/docx/scripts/comment.py new file mode 100755 index 0000000..46ed5f5 --- /dev/null +++ b/docx/scripts/comment.py @@ -0,0 +1,368 @@ +"""Add comments to a DOCX document. + +Accepts either an unpacked directory OR a .docx/.dotx file directly. + +Usage: + # Against an unpacked directory (writes satellite files in place) + python comment.py unpacked/ "Comment text" + python comment.py unpacked/ "Reply text" --parent 0 + + # Against a .docx directly (extracts, writes satellite files, rezips) + python comment.py contract.docx "This cap is too low" -o annotated.docx + python comment.py contract.docx "Comment" --id 5 # explicit ID + +The comment ID is auto-assigned (max existing + 1) unless --id is given. +Plain text is XML-escaped automatically; if you pass already-escaped text +(e.g. &, ’) use --raw to skip escaping. + +After running, add markers to word/document.xml so the comment is visible: + + ... commented content ... + + +""" + +import argparse +import random +import shutil +import sys +import tempfile +import zipfile +from datetime import datetime, timezone +from pathlib import Path + +import defusedxml.minidom +from xml.parsers.expat import ExpatError +from xml.sax.saxutils import escape as xml_escape + +from office.helpers import opc_target, rezip as _rezip, safe_extract as _safe_extract + +TEMPLATE_DIR = Path(__file__).parent / "templates" +NS = { + "w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main", + "w14": "http://schemas.microsoft.com/office/word/2010/wordml", + "w15": "http://schemas.microsoft.com/office/word/2012/wordml", + "w16cid": "http://schemas.microsoft.com/office/word/2016/wordml/cid", + "w16cex": "http://schemas.microsoft.com/office/word/2018/wordml/cex", +} + +COMMENT_XML = """\ + + + + + + + + + + + + + {text} + + +""" + +COMMENT_MARKER_TEMPLATE = """ +Add to word/document.xml (markers must be direct children of w:p, never inside w:r): + + ... + + """ + +REPLY_MARKER_TEMPLATE = """ +Nest markers inside parent {pid}'s markers (direct children of w:p, never inside w:r): + + ... + + + """ + +SMART_QUOTE_ENTITIES = { + "“": "“", + "”": "”", + "‘": "‘", + "’": "’", +} + + +def _generate_hex_id() -> str: + return f"{random.randint(0, 0x7FFFFFFE):08X}" + + +def _encode_smart_quotes(text: str) -> str: + for char, entity in SMART_QUOTE_ENTITIES.items(): + text = text.replace(char, entity) + return text + + +def _append_xml(xml_path: Path, root_tag: str, content: str) -> None: + dom = defusedxml.minidom.parseString(xml_path.read_text(encoding="utf-8")) + root = dom.getElementsByTagName(root_tag)[0] + ns_attrs = " ".join(f'xmlns:{k}="{v}"' for k, v in NS.items()) + wrapper_dom = defusedxml.minidom.parseString(f"{content}") + for child in wrapper_dom.documentElement.childNodes: + if child.nodeType == child.ELEMENT_NODE: + root.appendChild(dom.importNode(child, True)) + output = _encode_smart_quotes(dom.toxml(encoding="UTF-8").decode("utf-8")) + xml_path.write_text(output, encoding="utf-8") + + +def _find_para_id(comments_path: Path, comment_id: int) -> str | None: + dom = defusedxml.minidom.parseString(comments_path.read_text(encoding="utf-8")) + for c in dom.getElementsByTagName("w:comment"): + if c.getAttribute("w:id") == str(comment_id): + for p in c.getElementsByTagName("w:p"): + if pid := p.getAttribute("w14:paraId"): + return pid + return None + + +def _next_comment_id(comments_path: Path) -> int: + if not comments_path.exists(): + return 0 + dom = defusedxml.minidom.parseString(comments_path.read_text(encoding="utf-8")) + ids = [] + for c in dom.getElementsByTagName("w:comment"): + try: + ids.append(int(c.getAttribute("w:id"))) + except ValueError: + pass + return (max(ids) + 1) if ids else 0 + + +def _get_next_rid(rels_path: Path) -> int: + dom = defusedxml.minidom.parseString(rels_path.read_text(encoding="utf-8")) + max_rid = 0 + for rel in dom.getElementsByTagName("Relationship"): + rid = rel.getAttribute("Id") + if rid and rid.startswith("rId"): + try: + max_rid = max(max_rid, int(rid[3:])) + except ValueError: + pass + return max_rid + 1 + + +def _has_relationship(rels_path: Path, target: str) -> bool: + dom = defusedxml.minidom.parseString(rels_path.read_text(encoding="utf-8")) + return any( + rel.getAttribute("Target") == target + for rel in dom.getElementsByTagName("Relationship") + ) + + +def _has_content_type(ct_path: Path, part_name: str) -> bool: + dom = defusedxml.minidom.parseString(ct_path.read_text(encoding="utf-8")) + return any( + o.getAttribute("PartName") == part_name + for o in dom.getElementsByTagName("Override") + ) + + +_COMMENT_RELS = [ + ("http://schemas.openxmlformats.org/officeDocument/2006/relationships/comments", "comments.xml"), + ("http://schemas.microsoft.com/office/2011/relationships/commentsExtended", "commentsExtended.xml"), + ("http://schemas.microsoft.com/office/2016/09/relationships/commentsIds", "commentsIds.xml"), + ("http://schemas.microsoft.com/office/2018/08/relationships/commentsExtensible", "commentsExtensible.xml"), +] +_COMMENT_OVERRIDES = [ + ("/word/comments.xml", "application/vnd.openxmlformats-officedocument.wordprocessingml.comments+xml"), + ("/word/commentsExtended.xml", "application/vnd.openxmlformats-officedocument.wordprocessingml.commentsExtended+xml"), + ("/word/commentsIds.xml", "application/vnd.openxmlformats-officedocument.wordprocessingml.commentsIds+xml"), + ("/word/commentsExtensible.xml", "application/vnd.openxmlformats-officedocument.wordprocessingml.commentsExtensible+xml"), +] + + +def _ensure_comment_relationships(unpacked_dir: Path) -> None: + rels_path = unpacked_dir / "word" / "_rels" / "document.xml.rels" + if not rels_path.exists(): + return + dom = defusedxml.minidom.parseString(rels_path.read_text(encoding="utf-8")) + root = dom.documentElement + comment_types = {rel_type for rel_type, _ in _COMMENT_RELS} + existing = set() + for rel in dom.getElementsByTagName("Relationship"): + if rel.getAttribute("Type") not in comment_types: + continue + part = opc_target( + rel.getAttribute("Target"), + "word/document.xml", + rel.getAttribute("TargetMode"), + ) + if part is not None: + existing.add(part) + next_rid = _get_next_rid(rels_path) + changed = False + for rel_type, target in _COMMENT_RELS: + if opc_target(target, "word/document.xml") in existing: + continue + rel = dom.createElement("Relationship") + rel.setAttribute("Id", f"rId{next_rid}") + rel.setAttribute("Type", rel_type) + rel.setAttribute("Target", target) + root.appendChild(rel) + next_rid += 1 + changed = True + if changed: + rels_path.write_bytes(dom.toxml(encoding="UTF-8")) + + +def _ensure_comment_content_types(unpacked_dir: Path) -> None: + ct_path = unpacked_dir / "[Content_Types].xml" + if not ct_path.exists(): + return + dom = defusedxml.minidom.parseString(ct_path.read_text(encoding="utf-8")) + root = dom.documentElement + existing = { + o.getAttribute("PartName") + for o in dom.getElementsByTagName("Override") + } + changed = False + for part_name, content_type in _COMMENT_OVERRIDES: + if part_name in existing: + continue + override = dom.createElement("Override") + override.setAttribute("PartName", part_name) + override.setAttribute("ContentType", content_type) + root.appendChild(override) + changed = True + if changed: + ct_path.write_bytes(dom.toxml(encoding="UTF-8")) + + +def add_comment( + unpacked_dir: Path | str, + text: str, + comment_id: int | None = None, + author: str = "Claude", + initials: str = "C", + parent_id: int | None = None, + raw: bool = False, +) -> tuple[int, str, str]: + unpacked_dir = Path(unpacked_dir) + if not raw: + text = xml_escape(text) + author = xml_escape(author, {'"': """}) + initials = xml_escape(initials, {'"': """}) + word = unpacked_dir / "word" + if not word.exists(): + raise FileNotFoundError(f"{word} not found (not an unpacked .docx?)") + + comments = word / "comments.xml" + if comment_id is None: + comment_id = _next_comment_id(comments) + + parent_para = None + if parent_id is not None: + parent_para = _find_para_id(comments, parent_id) if comments.exists() else None + if not parent_para: + raise ValueError(f"parent comment {parent_id} not found") + + para_id, durable_id = _generate_hex_id(), _generate_hex_id() + ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + if not comments.exists(): + shutil.copy(TEMPLATE_DIR / "comments.xml", comments) + _ensure_comment_relationships(unpacked_dir) + _ensure_comment_content_types(unpacked_dir) + _append_xml( + comments, + "w:comments", + COMMENT_XML.format( + id=comment_id, author=author, date=ts, initials=initials, + para_id=para_id, text=text, + ), + ) + + ext = word / "commentsExtended.xml" + if not ext.exists(): + shutil.copy(TEMPLATE_DIR / "commentsExtended.xml", ext) + if parent_para is not None: + _append_xml( + ext, "w15:commentsEx", + f'', + ) + else: + _append_xml( + ext, "w15:commentsEx", + f'', + ) + + ids = word / "commentsIds.xml" + if not ids.exists(): + shutil.copy(TEMPLATE_DIR / "commentsIds.xml", ids) + _append_xml( + ids, "w16cid:commentsIds", + f'', + ) + + extensible = word / "commentsExtensible.xml" + if not extensible.exists(): + shutil.copy(TEMPLATE_DIR / "commentsExtensible.xml", extensible) + _append_xml( + extensible, "w16cex:commentsExtensible", + f'', + ) + + action = "reply" if parent_id is not None else "comment" + return comment_id, para_id, f"Added {action} id={comment_id} (paraId={para_id})" + + +def main() -> None: + p = argparse.ArgumentParser(description="Add a comment to a DOCX (directory or .docx file).") + p.add_argument("input", help="Unpacked DOCX directory OR a .docx/.dotx file") + p.add_argument("text", help="Comment text (plain text; XML-escaped automatically)") + p.add_argument("--raw", action="store_true", + help="Treat text as pre-escaped XML (skip automatic escaping)") + p.add_argument("--id", type=int, dest="comment_id", + help="Comment ID (default: auto-assign as max existing + 1)") + p.add_argument("--author", default="Claude", help="Author name") + p.add_argument("--initials", default="C", help="Author initials") + p.add_argument("--parent", type=int, help="Parent comment ID (makes this a reply)") + p.add_argument("-o", "--output", + help="Output .docx path (only used when input is a .docx; default: overwrite input)") + args = p.parse_args() + + src = Path(args.input) + + try: + if src.is_dir(): + if args.output: + print("Warning: --output ignored for directory input", file=sys.stderr) + cid, _, msg = add_comment( + src, args.text, comment_id=args.comment_id, + author=args.author, initials=args.initials, + parent_id=args.parent, raw=args.raw, + ) + print(msg) + elif src.is_file() and src.suffix.lower() in (".docx", ".dotx"): + out = Path(args.output) if args.output else src + with tempfile.TemporaryDirectory() as tmp: + tmp_path = Path(tmp) + with zipfile.ZipFile(src) as zf: + _safe_extract(zf, tmp_path) + cid, _, msg = add_comment( + tmp_path, args.text, comment_id=args.comment_id, + author=args.author, initials=args.initials, + parent_id=args.parent, raw=args.raw, + ) + _rezip(tmp_path, out) + print(msg) + print(f"Wrote {out} (comment defined; add markers to word/document.xml to make it visible)") + else: + print(f"Error: {src} is neither a directory nor a .docx/.dotx file", file=sys.stderr) + sys.exit(1) + except (FileNotFoundError, ValueError, zipfile.BadZipFile, ExpatError) as e: + print(f"Error: {e}", file=sys.stderr) + sys.exit(1) + + if args.parent is not None: + print(REPLY_MARKER_TEMPLATE.format(pid=args.parent, cid=cid)) + else: + print(COMMENT_MARKER_TEMPLATE.format(cid=cid)) + + +if __name__ == "__main__": + main() diff --git a/docx/scripts/merge_runs.py b/docx/scripts/merge_runs.py new file mode 100755 index 0000000..4c7c1bf --- /dev/null +++ b/docx/scripts/merge_runs.py @@ -0,0 +1,310 @@ +"""Merge adjacent identically-formatted runs in a DOCX. + +Word fragments paragraph text across many elements (revision ids, +spell-check markers, editing history), which makes find-and-replace on +word/document.xml unreliable — the string you're looking for is split +across runs. This coalesces adjacent runs whose formatting () is +identical, strips rsid attributes and proofErr markers, and consolidates the +text elements — , and for text inside a tracked deletion. + +Rendering is unchanged. The text you search is what Word draws, which is not +always the bytes in the file: an element without xml:space="preserve" has its +edge whitespace trimmed before it reaches the page, so `Hello ` +followed by `world` reads "Helloworld" and merges to exactly that. + +Runs in two different / wrappers are never merged: that would +rewrite tracked-change structure, collapsing separate revisions into one. + +Only word/document.xml is processed (not headers, footers, or footnotes). + +Usage: + python merge_runs.py unpacked/ # after unzip, before editing + python merge_runs.py document.docx # rewrite in place + python merge_runs.py document.docx -o out.docx +""" + + +import argparse +import sys +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.minidom + +from office.helpers import XML_SPACE, rendered_text, rezip, safe_extract + +WORDML_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + + +def merge_runs(input_dir: str) -> tuple[int, str]: + doc_xml = Path(input_dir) / "word" / "document.xml" + + if not doc_xml.exists(): + return 0, f"Error: {doc_xml} not found" + + try: + dom = defusedxml.minidom.parseString(doc_xml.read_text(encoding="utf-8")) + root = dom.documentElement + run_names = _run_tag_names(root) + + _remove_elements(root, "proofErr") + + runs = _find_runs(root, run_names) + _strip_rsid_attrs(runs) + + merge_count = 0 + for container in {run.parentNode for run in runs}: + merge_count += _merge_runs_in(container, run_names) + + doc_xml.write_bytes(dom.toxml(encoding="UTF-8")) + return merge_count, f"Merged {merge_count} runs" + + except Exception as e: + return 0, f"Error: {e}" + + + + +def _is_element(node, tag: str) -> bool: + name = node.localName or node.tagName + return name == tag or name.endswith(f":{tag}") + + +def _run_tag_names(root) -> set[str]: + names = set() + for attr in root.attributes.values(): + if attr.value == WORDML_NS: + if attr.name == "xmlns": + names.add("r") + elif attr.name.startswith("xmlns:"): + names.add(attr.name.split(":", 1)[1] + ":r") + return names or {"w:r", "r"} + + +def _find_elements(root, tag: str) -> list: + results = [] + + def traverse(node): + if node.nodeType == node.ELEMENT_NODE: + if _is_element(node, tag): + results.append(node) + for child in node.childNodes: + traverse(child) + + traverse(root) + return results + + +def _find_runs(root, run_names: set[str]) -> list: + return [e for e in _find_elements(root, "r") if _is_run(e, run_names)] + + +def _get_child(parent, tag: str): + return next(iter(_get_children(parent, tag)), None) + + +def _get_children(parent, tag: str) -> list: + return [ + child + for child in parent.childNodes + if child.nodeType == child.ELEMENT_NODE and _is_element(child, tag) + ] + + +def _is_adjacent(elem1, elem2) -> bool: + node = elem1.nextSibling + while node: + if node == elem2: + return True + if node.nodeType == node.ELEMENT_NODE: + return False + if node.nodeType == node.TEXT_NODE and node.data.strip(XML_SPACE): + return False + node = node.nextSibling + return False + + + + +def _remove_elements(root, tag: str): + for elem in _find_elements(root, tag): + if elem.parentNode: + elem.parentNode.removeChild(elem) + + +def _strip_rsid_attrs(runs: list): + for run in runs: + for attr in list(run.attributes.values()): + if "rsid" in attr.name.lower(): + run.removeAttribute(attr.name) + + + + +def _merge_runs_in(container, run_names: set[str]) -> int: + merge_count = 0 + run = _first_child_run(container, run_names) + + while run: + while True: + next_elem = _next_element_sibling(run) + if next_elem and _is_run(next_elem, run_names) and _can_merge(run, next_elem): + _merge_run_content(run, next_elem) + container.removeChild(next_elem) + merge_count += 1 + else: + break + + _consolidate_text(run) + run = _next_sibling_run(run, run_names) + + return merge_count + + +def _first_child_run(container, run_names: set[str]): + for child in container.childNodes: + if child.nodeType == child.ELEMENT_NODE and _is_run(child, run_names): + return child + return None + + +def _next_element_sibling(node): + sibling = node.nextSibling + while sibling: + if sibling.nodeType == sibling.ELEMENT_NODE: + return sibling + sibling = sibling.nextSibling + return None + + +def _next_sibling_run(node, run_names: set[str]): + sibling = node.nextSibling + while sibling: + if sibling.nodeType == sibling.ELEMENT_NODE: + if _is_run(sibling, run_names): + return sibling + sibling = sibling.nextSibling + return None + + +def _is_run(node, run_names: set[str]) -> bool: + return node.tagName in run_names + + +def _can_merge(run1, run2) -> bool: + rpr1 = _get_child(run1, "rPr") + rpr2 = _get_child(run2, "rPr") + + if (rpr1 is None) != (rpr2 is None): + return False + if rpr1 is None: + return True + return rpr1.toxml() == rpr2.toxml() + + +def _merge_run_content(target, source): + for child in list(source.childNodes): + if child.nodeType == child.ELEMENT_NODE: + name = child.localName or child.tagName + if name != "rPr" and not name.endswith(":rPr"): + target.appendChild(child) + + +def _element_text(elem) -> str: + return "".join( + child.data + for child in elem.childNodes + if child.nodeType in (child.TEXT_NODE, child.CDATA_SECTION_NODE) + ) + + +def _has_preserve(elem) -> bool: + return elem.getAttribute("xml:space") == "preserve" + + +def _rendered_text(elem) -> str: + return rendered_text(_element_text(elem), _has_preserve(elem)) + + +def _consolidate_text(run): + for tag in ("t", "delText"): + _consolidate_text_elements(run, tag) + + +def _consolidate_text_elements(run, tag: str): + t_elements = _get_children(run, tag) + + for i in range(len(t_elements) - 1, 0, -1): + curr, prev = t_elements[i], t_elements[i - 1] + + if _is_adjacent(prev, curr): + merged = _rendered_text(prev) + _rendered_text(curr) + had_preserve = _has_preserve(prev) or _has_preserve(curr) + + new_text = run.ownerDocument.createTextNode(merged) + for node in list(prev.childNodes): + if node.nodeType in (node.TEXT_NODE, node.CDATA_SECTION_NODE): + prev.removeChild(node) + else: + run.insertBefore(node, curr) + prev.appendChild(new_text) + for node in list(curr.childNodes): + if node.nodeType not in (node.TEXT_NODE, node.CDATA_SECTION_NODE): + run.insertBefore(node, curr) + + if merged != merged.strip(XML_SPACE) or had_preserve: + prev.setAttribute("xml:space", "preserve") + elif prev.hasAttribute("xml:space"): + prev.removeAttribute("xml:space") + + run.removeChild(curr) + + + + +def _merge_or_die(path: Path) -> str: + _, msg = merge_runs(str(path)) + if msg.startswith("Error"): + print(msg, file=sys.stderr) + sys.exit(1) + return msg + + +def main() -> None: + p = argparse.ArgumentParser( + description="Merge adjacent identically-formatted runs in a DOCX (directory or .docx file)." + ) + p.add_argument("input", help="Unpacked DOCX directory OR a .docx/.dotx file") + p.add_argument( + "-o", "--output", + help="Output .docx path (only valid when input is a .docx; default: overwrite input)", + ) + args = p.parse_args() + + src = Path(args.input) + + try: + if src.is_dir(): + if args.output: + p.error("--output is only valid for .docx input; directory input is modified in place") + print(_merge_or_die(src)) + elif src.is_file() and src.suffix.lower() in (".docx", ".dotx"): + out = Path(args.output) if args.output else src + with tempfile.TemporaryDirectory() as tmp: + tmp_path = Path(tmp) + with zipfile.ZipFile(src) as zf: + safe_extract(zf, tmp_path) + msg = _merge_or_die(tmp_path) + rezip(tmp_path, out) + print(f"{msg}; wrote {out}") + else: + print(f"Error: {src} is neither a directory nor a .docx/.dotx file", file=sys.stderr) + sys.exit(1) + except (OSError, ValueError, zipfile.BadZipFile) as e: + print(f"Error: {e}", file=sys.stderr) + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/docx/scripts/office/helpers/__init__.py b/docx/scripts/office/helpers/__init__.py new file mode 100644 index 0000000..d3c5817 --- /dev/null +++ b/docx/scripts/office/helpers/__init__.py @@ -0,0 +1,111 @@ +import os +import posixpath +import re +import stat +import tempfile +import urllib.parse +import zipfile +from pathlib import Path + +OOXML_FAMILY = { + ".docx": "docx", + ".dotx": "docx", + ".pptx": "pptx", + ".potx": "pptx", + ".xlsx": "xlsx", + ".xltx": "xlsx", +} + +_SCHEME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9+.\-]*:") + +SLIDE_REL_TYPE = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide" + + +def opc_target(target: str, source_part: str, target_mode: str = "") -> str | None: + if not target: + return None + if target_mode.lower() == "external": + return None + if _SCHEME_RE.match(target): + return None + + target = urllib.parse.unquote(target) + + if "\\" in target: + raise ValueError(f"relationship target is not a POSIX part name: {target!r}") + + if target.startswith("/"): + joined = target.lstrip("/") + else: + joined = posixpath.join(posixpath.dirname(source_part), target) + + parts: list[str] = [] + for segment in posixpath.normpath(joined).split("/"): + if segment in ("", "."): + continue + if segment == "..": + if not parts: + raise ValueError(f"relationship target escapes the package: {target!r}") + parts.pop() + else: + parts.append(segment) + + if not parts: + raise ValueError(f"relationship target resolves to nothing: {target!r}") + return "/".join(parts) + + +def rels_source_part(rels_file: Path, unpacked_dir: Path) -> str: + owner_dir = rels_file.parent.parent.relative_to(unpacked_dir) + return posixpath.join(owner_dir.as_posix(), rels_file.name[: -len(".rels")]).lstrip("./") + + +def part_text(data: bytes) -> str: + return data.decode("utf-8", "surrogateescape") + + +XML_SPACE = " \t\r\n" + + +def rendered_text(text: str, preserve: bool) -> str: + return text if preserve else text.strip(XML_SPACE) + + +def safe_extract(zf: zipfile.ZipFile, dest: Path) -> None: + dest = dest.resolve() + for m in zf.infolist(): + if stat.S_ISLNK(m.external_attr >> 16): + raise ValueError(f"symlink archive entry not allowed: {m.filename!r}") + target = (dest / m.filename).resolve() + if not target.is_relative_to(dest): + raise ValueError(f"unsafe archive entry: {m.filename!r}") + zf.extract(m, dest) + + +def rezip(src_dir: Path, out_path: Path) -> None: + files = sorted(p for p in src_dir.rglob("*") if p.is_file()) + ct = src_dir / "[Content_Types].xml" + fd, tmp_name = tempfile.mkstemp( + prefix=out_path.name + ".", suffix=".tmp", dir=out_path.parent + ) + tmp_out = Path(tmp_name) + try: + with os.fdopen(fd, "wb") as fh: + with zipfile.ZipFile(fh, "w", zipfile.ZIP_DEFLATED) as zf: + if ct.exists(): + zf.write(ct, ct.relative_to(src_dir), compress_type=zipfile.ZIP_STORED) + for f in files: + if f == ct: + continue + zf.write(f, f.relative_to(src_dir)) + if out_path.exists(): + mode = out_path.stat().st_mode & 0o777 + else: + umask = os.umask(0) + os.umask(umask) + mode = 0o666 & ~umask + os.chmod(tmp_out, mode) + os.replace(tmp_out, out_path) + finally: + if tmp_out.exists(): + tmp_out.unlink() diff --git a/docx/scripts/office/helpers/pptx_chart.py b/docx/scripts/office/helpers/pptx_chart.py new file mode 100644 index 0000000..209cb7c --- /dev/null +++ b/docx/scripts/office/helpers/pptx_chart.py @@ -0,0 +1,170 @@ +"""Find chart XML that PowerPoint refuses but the schema accepts. + +Detection only: for either fault more than one repair is valid, and only the +author knows which was meant. +""" + + +from __future__ import annotations + +import re +from typing import Mapping + +from . import part_text + + +_CHART_PART_RE = re.compile(r"ppt/charts/chart\d+\.xml") + +_GROUPING_RE = re.compile(r"""]*?\bval=["'](\w+)["']""") +_DLBL_POS_RE = re.compile(r"""]*?\bval=["'](\w+)["']""") + +def _strip_ext_lst(text: str) -> str: + out, cursor = [], 0 + for lo, hi in _ext_lst_spans(text): + out.append(text[cursor:lo]) + cursor = hi + out.append(text[cursor:]) + return "".join(out) + +_BAR_GROUP_RE = re.compile(r"]*(?.*?", re.DOTALL) + +STACKED_GROUPINGS = frozenset({"stacked", "percentStacked"}) +ILLEGAL_ON_STACKED = frozenset({"outEnd"}) +LEGAL_ON_STACKED = ("ctr", "inEnd", "inBase") + + +def _check_stacked_label_positions(part: str, xml: str) -> list[str]: + problems: list[str] = [] + for match in _BAR_GROUP_RE.finditer(xml): + block = _strip_ext_lst(match.group(0)) + group = match.group(1) + + grouping = _GROUPING_RE.search(block) + if grouping is None or grouping.group(1) not in STACKED_GROUPINGS: + continue + + bad = [p for p in _DLBL_POS_RE.findall(block) if p in ILLEGAL_ON_STACKED] + for pos in sorted(set(bad)): + problems.append( + f'{part}: {bad.count(pos)} data label(s) use dLblPos="{pos}" on a ' + f"{grouping.group(1)} {group}; PowerPoint allows only " + f"{', '.join(LEGAL_ON_STACKED)} there" + ) + return problems + + + +_ANY_CHART_GROUP_RE = re.compile(r"]*(?.*?", re.DOTALL) + +_AXID_RE = re.compile( + r"""\s*]*?\bval=["'](-?\d+)["']\s*(?:/>|>\s*)""" +) + +_AXIS_DECL_RE = re.compile( + r"""]*(?\s*]*?\bval=["'](-?\d+)["']""" +) + +AXID_LIMIT = { + "barChart": 2, "lineChart": 2, "areaChart": 2, "scatterChart": 2, + "bubbleChart": 2, "radarChart": 2, "stockChart": 2, + "bar3DChart": 3, "line3DChart": 3, "area3DChart": 3, + "surfaceChart": 3, "surface3DChart": 3, +} + +AXID_MINIMUM = { + "barChart": 2, "lineChart": 2, "areaChart": 2, "scatterChart": 2, + "bubbleChart": 2, "radarChart": 2, "stockChart": 2, + "bar3DChart": 2, "area3DChart": 2, "surfaceChart": 2, + "line3DChart": 3, "surface3DChart": 3, +} + + +def _declared_axes(xml: str) -> dict[str, list[str]]: + axes: dict[str, list[str]] = {} + for kind, axid in _AXIS_DECL_RE.findall(xml): + axes.setdefault(kind, []).append(axid) + return axes + + +def _canonical_ids(axes: dict[str, list[str]], limit: int) -> list[str] | None: + category = axes.get("catAx", []) + axes.get("dateAx", []) + value = axes.get("valAx", []) + series = axes.get("serAx", []) + if len(category) != 1 or len(value) != 1 or len(series) > 1: + return None + ids = [category[0], value[0]] + if limit >= 3 and series: + ids.append(series[0]) + return ids + + +def _undeclared_axes(kind: str, block: str, axes: dict[str, list[str]]) -> list[str] | None: + if kind not in AXID_LIMIT: + return None + ids = _AXID_RE.findall(block) + declared = {i for group in axes.values() for i in group} + if len([i for i in ids if i in declared]) >= 2: + return None + return ids + + +def _check_chart_axis_references(part: str, xml: str) -> list[str]: + axes = _declared_axes(xml) + problems: list[str] = [] + declared = {i for group in axes.values() for i in group} + for match in _ANY_CHART_GROUP_RE.finditer(xml): + kind, block = match.group(1), match.group(0) + ids = _undeclared_axes(kind, block, axes) + if ids is None: + continue + if not ids: + problems.append( + f"{part}: declares no this part can resolve; a chart " + f"group needs {AXID_MINIMUM[kind]}, and PowerPoint discards one with fewer" + ) + continue + dead = [i for i in ids if i not in declared] + canonical = _canonical_ids(axes, AXID_LIMIT[kind]) + if canonical is not None and len(canonical) >= AXID_MINIMUM[kind]: + hint = f"Fix: point them at the axes this part declares ({', '.join(canonical)})" + else: + hint = ("Fix: the part declares several axes of a kind -- declare the " + "secondary axes the series expects, or drop them") + detail = (f"of which {', '.join(dead)} name no declared axis" + if dead else f"only {len(ids)} of which this part declares") + problems.append( + f"{part}: references axId {', '.join(ids)}, {detail}, " + f"leaving fewer than two live axes; PowerPoint discards the chart. {hint}" + ) + return problems + + +def _ext_lst_spans(text: str) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + depth = 0 + start = 0 + for match in re.finditer(r"<(/?)c:extLst\b[^>]*?(/?)>", text): + closing, self_closing = match.group(1), match.group(2) + if self_closing: + continue + if closing: + depth -= 1 + if depth == 0: + spans.append((start, match.end())) + else: + if depth == 0: + start = match.start() + depth += 1 + return spans + + +CHART_CHECKS = (_check_stacked_label_positions, _check_chart_axis_references) + + +def find_chart_problems(files: Mapping[str, bytes]) -> list[str]: + problems: list[str] = [] + for part in sorted(n for n in files if _CHART_PART_RE.fullmatch(n)): + xml = part_text(files[part]) + for check in CHART_CHECKS: + problems.extend(check(part, xml)) + return problems diff --git a/docx/scripts/office/helpers/pptx_slide.py b/docx/scripts/office/helpers/pptx_slide.py new file mode 100644 index 0000000..22f9aee --- /dev/null +++ b/docx/scripts/office/helpers/pptx_slide.py @@ -0,0 +1,60 @@ +"""Pick the slide-XML schema errors PowerPoint refuses the file over. + +A denylist over lxml's messages, so an unrecognised error class is a miss rather +than a false alarm. +""" + + +from __future__ import annotations + +import re + +SLIDE_PART_RE = re.compile( + r"ppt/(slides|slideLayouts|slideMasters|notesSlides|notesMasters|handoutMasters)" + r"/[^/]+\.xml" +) + +FATAL_SLIDE_ERRORS: tuple[tuple[re.Pattern[str], str], ...] = ( + ( + re.compile(r"\}tableStyleId': This element is not expected"), + "two in one (the schema allows one)", + ), + ( + re.compile(r"\}srgbClr', attribute 'val'"), + "a colour that is not six hex digits", + ), + ( + re.compile(r"\}txBody': Missing child element"), + "a with no children", + ), + ( + re.compile(r"\}miter', attribute 'lim'"), + 'a line join with lim="NaN"', + ), + ( + re.compile(r"\}uLnTx': This element is not expected"), + " in a position the schema forbids", + ), + ( + re.compile(r"\}overrideClrMapping': This element is not expected"), + " in a position the schema forbids", + ), + ( + re.compile(r"\}nvGrpSpPr': Missing child element"), + "a with no children", + ), +) + + +def is_schema_verdict(error: str) -> bool: + return error.startswith("Element ") + + +def fatal_slide_errors(errors: set[str]) -> list[str]: + out = [] + for error in sorted(errors): + for pattern, meaning in FATAL_SLIDE_ERRORS: + if pattern.search(error): + out.append(f"{meaning}: {error}") + break + return out diff --git a/docx/scripts/office/helpers/pptx_theme.py b/docx/scripts/office/helpers/pptx_theme.py new file mode 100644 index 0000000..8446620 --- /dev/null +++ b/docx/scripts/office/helpers/pptx_theme.py @@ -0,0 +1,114 @@ +"""Find masters sharing a theme part in the way PowerPoint refuses to open. + +Reports only; the fix is to move back to directly after + in ppt/presentation.xml. +""" + + +from __future__ import annotations + +import posixpath +import re +from typing import Mapping + +from . import part_text + +THEME_REL_TYPE = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/theme" + +_MASTER_RE = re.compile( + r"^ppt/(?PslideMasters|notesMasters|handoutMasters)/" + r"(?:slide|notes|handout)Master(?P\d+)\.xml$" +) +_GROUP_ORDER = {"slideMasters": 0, "notesMasters": 1, "handoutMasters": 2} + +_RELATIONSHIP_RE = re.compile( + r"]*?(?:/>|>.*?)", re.DOTALL +) + + +def _sort_key(name: str) -> tuple[int, int]: + m = _MASTER_RE.match(name) + assert m is not None + return (_GROUP_ORDER[m.group("group")], int(m.group("num"))) + + +def _rels_path(part: str) -> str: + directory, base = posixpath.split(part) + return f"{directory}/_rels/{base}.rels" + + +def _resolve(rels_path: str, target: str) -> str: + if target.startswith("/"): + return target.lstrip("/") + part_dir = posixpath.dirname(posixpath.dirname(rels_path)) + return posixpath.normpath(posixpath.join(part_dir, target)) + + +def _theme_rel(files: Mapping[str, bytes], master: str): + rels_path = _rels_path(master) + rels = files.get(rels_path) + if rels is None: + return None + for element in _RELATIONSHIP_RE.findall(part_text(rels)): + if f'Type="{THEME_REL_TYPE}"' not in element: + continue + target = re.search(r'\bTarget="([^"]+)"', element) + if target is None: + continue + return rels_path, element, _resolve(rels_path, target.group(1)) + return None + + +def _masters(files: Mapping[str, bytes]) -> list[str]: + return sorted((n for n in files if _MASTER_RE.match(n)), key=_sort_key) + + +_PRESENTATION = "ppt/presentation.xml" +_NOTES_MASTERS = "ppt/notesMasters/" +_IGNORABLE_RE = re.compile(r"|<\?.*?\?>", re.DOTALL) +_AFTER_SLDIDLST_RE = re.compile( + r"]*/>|[^>]*>.*?)\s*(<[^>\s/]+)", re.DOTALL +) + + +def _notes_master_share_is_inert(files: Mapping[str, bytes]) -> bool: + data = files.get(_PRESENTATION) + if data is None: + return False + match = _AFTER_SLDIDLST_RE.search(_IGNORABLE_RE.sub("", part_text(data))) + return match is not None and match.group(1) == " bool: + return inert_notes and master.startswith(_NOTES_MASTERS) + + +def find_shared_master_themes(files: Mapping[str, bytes]) -> list[str]: + return [ + f"{master} shares {theme} with {first}" + for master, _, _, theme, first in _shares(files) + ] + + +def live_shared_master_themes(files: Mapping[str, bytes]) -> list[str]: + inert_notes = _notes_master_share_is_inert(files) + return [ + f"{master} shares {theme} with {first}" + for master, _, _, theme, first in _shares(files) + if not _is_inert(master, inert_notes) + ] diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd new file mode 100644 index 0000000..6454ef9 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd @@ -0,0 +1,1499 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd new file mode 100644 index 0000000..afa4f46 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd @@ -0,0 +1,146 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd new file mode 100644 index 0000000..64e66b8 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd @@ -0,0 +1,1085 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd new file mode 100644 index 0000000..687eea8 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd @@ -0,0 +1,11 @@ + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd new file mode 100644 index 0000000..6ac81b0 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd @@ -0,0 +1,3081 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd new file mode 100644 index 0000000..1dbf051 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd @@ -0,0 +1,23 @@ + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd new file mode 100644 index 0000000..f1af17d --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd @@ -0,0 +1,185 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd new file mode 100644 index 0000000..0a185ab --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd @@ -0,0 +1,287 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd new file mode 100644 index 0000000..14ef488 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd @@ -0,0 +1,1676 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd new file mode 100644 index 0000000..c20f3bf --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd @@ -0,0 +1,28 @@ + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd new file mode 100644 index 0000000..ac60252 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd @@ -0,0 +1,144 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd new file mode 100644 index 0000000..424b8ba --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd @@ -0,0 +1,174 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd new file mode 100644 index 0000000..2bddce2 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd @@ -0,0 +1,25 @@ + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd new file mode 100644 index 0000000..8a8c18b --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd new file mode 100644 index 0000000..5c42706 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd @@ -0,0 +1,59 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd new file mode 100644 index 0000000..853c341 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd @@ -0,0 +1,56 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd new file mode 100644 index 0000000..da835ee --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd @@ -0,0 +1,195 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd new file mode 100644 index 0000000..87ad265 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd @@ -0,0 +1,582 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd new file mode 100644 index 0000000..9e86f1b --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd @@ -0,0 +1,25 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd new file mode 100644 index 0000000..d0be42e --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd @@ -0,0 +1,4439 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd new file mode 100644 index 0000000..8821dd1 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd @@ -0,0 +1,570 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd new file mode 100644 index 0000000..ca2575c --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd @@ -0,0 +1,509 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd new file mode 100644 index 0000000..dd079e6 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd @@ -0,0 +1,12 @@ + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd new file mode 100644 index 0000000..3dd6cf6 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd @@ -0,0 +1,108 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd new file mode 100644 index 0000000..f1041e3 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd @@ -0,0 +1,96 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd new file mode 100644 index 0000000..9c5b7a6 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd @@ -0,0 +1,3646 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd new file mode 100644 index 0000000..0f13678 --- /dev/null +++ b/docx/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd @@ -0,0 +1,116 @@ + + + + + + See http://www.w3.org/XML/1998/namespace.html and + http://www.w3.org/TR/REC-xml for information about this namespace. + + This schema document describes the XML namespace, in a form + suitable for import by other schema documents. + + Note that local names in this namespace are intended to be defined + only by the World Wide Web Consortium or its subgroups. The + following names are currently defined in this namespace and should + not be used with conflicting semantics by any Working Group, + specification, or document instance: + + base (as an attribute name): denotes an attribute whose value + provides a URI to be used as the base for interpreting any + relative URIs in the scope of the element on which it + appears; its value is inherited. This name is reserved + by virtue of its definition in the XML Base specification. + + lang (as an attribute name): denotes an attribute whose value + is a language code for the natural language of the content of + any element; its value is inherited. This name is reserved + by virtue of its definition in the XML specification. + + space (as an attribute name): denotes an attribute whose + value is a keyword indicating what whitespace processing + discipline is intended for the content of the element; its + value is inherited. This name is reserved by virtue of its + definition in the XML specification. + + Father (in any context at all): denotes Jon Bosak, the chair of + the original XML Working Group. This name is reserved by + the following decision of the W3C XML Plenary and + XML Coordination groups: + + In appreciation for his vision, leadership and dedication + the W3C XML Plenary on this 10th day of February, 2000 + reserves for Jon Bosak in perpetuity the XML name + xml:Father + + + + + This schema defines attributes and an attribute group + suitable for use by + schemas wishing to allow xml:base, xml:lang or xml:space attributes + on elements they define. + + To enable this, such a schema must import this schema + for the XML namespace, e.g. as follows: + <schema . . .> + . . . + <import namespace="http://www.w3.org/XML/1998/namespace" + schemaLocation="http://www.w3.org/2001/03/xml.xsd"/> + + Subsequently, qualified reference to any of the attributes + or the group defined below will have the desired effect, e.g. + + <type . . .> + . . . + <attributeGroup ref="xml:specialAttrs"/> + + will define a type which will schema-validate an instance + element with any of those attributes + + + + In keeping with the XML Schema WG's standard versioning + policy, this schema document will persist at + http://www.w3.org/2001/03/xml.xsd. + At the date of issue it can also be found at + http://www.w3.org/2001/xml.xsd. + The schema document at that URI may however change in the future, + in order to remain compatible with the latest version of XML Schema + itself. In other words, if the XML Schema namespace changes, the version + of this document at + http://www.w3.org/2001/xml.xsd will change + accordingly; the version at + http://www.w3.org/2001/03/xml.xsd will not change. + + + + + + In due course, we should install the relevant ISO 2- and 3-letter + codes as the enumerated possible values . . . + + + + + + + + + + + + + + + See http://www.w3.org/TR/xmlbase/ for + information about this attribute. + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ecma/fourth-edition/opc-contentTypes.xsd b/docx/scripts/office/schemas/ecma/fourth-edition/opc-contentTypes.xsd new file mode 100644 index 0000000..a6de9d2 --- /dev/null +++ b/docx/scripts/office/schemas/ecma/fourth-edition/opc-contentTypes.xsd @@ -0,0 +1,42 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ecma/fourth-edition/opc-coreProperties.xsd b/docx/scripts/office/schemas/ecma/fourth-edition/opc-coreProperties.xsd new file mode 100644 index 0000000..10e978b --- /dev/null +++ b/docx/scripts/office/schemas/ecma/fourth-edition/opc-coreProperties.xsd @@ -0,0 +1,50 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ecma/fourth-edition/opc-digSig.xsd b/docx/scripts/office/schemas/ecma/fourth-edition/opc-digSig.xsd new file mode 100644 index 0000000..4248bf7 --- /dev/null +++ b/docx/scripts/office/schemas/ecma/fourth-edition/opc-digSig.xsd @@ -0,0 +1,49 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/ecma/fourth-edition/opc-relationships.xsd b/docx/scripts/office/schemas/ecma/fourth-edition/opc-relationships.xsd new file mode 100644 index 0000000..5649746 --- /dev/null +++ b/docx/scripts/office/schemas/ecma/fourth-edition/opc-relationships.xsd @@ -0,0 +1,33 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/mce/mc.xsd b/docx/scripts/office/schemas/mce/mc.xsd new file mode 100644 index 0000000..ef72545 --- /dev/null +++ b/docx/scripts/office/schemas/mce/mc.xsd @@ -0,0 +1,75 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/microsoft/wml-2010.xsd b/docx/scripts/office/schemas/microsoft/wml-2010.xsd new file mode 100644 index 0000000..f65f777 --- /dev/null +++ b/docx/scripts/office/schemas/microsoft/wml-2010.xsd @@ -0,0 +1,560 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/microsoft/wml-2012.xsd b/docx/scripts/office/schemas/microsoft/wml-2012.xsd new file mode 100644 index 0000000..6b00755 --- /dev/null +++ b/docx/scripts/office/schemas/microsoft/wml-2012.xsd @@ -0,0 +1,67 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/microsoft/wml-2018.xsd b/docx/scripts/office/schemas/microsoft/wml-2018.xsd new file mode 100644 index 0000000..f321d33 --- /dev/null +++ b/docx/scripts/office/schemas/microsoft/wml-2018.xsd @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd b/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd new file mode 100644 index 0000000..364c6a9 --- /dev/null +++ b/docx/scripts/office/schemas/microsoft/wml-cex-2018.xsd @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd b/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd new file mode 100644 index 0000000..fed9d15 --- /dev/null +++ b/docx/scripts/office/schemas/microsoft/wml-cid-2016.xsd @@ -0,0 +1,13 @@ + + + + + + + + + + + + + diff --git a/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd b/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd new file mode 100644 index 0000000..680cf15 --- /dev/null +++ b/docx/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd @@ -0,0 +1,4 @@ + + + + diff --git a/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd b/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd new file mode 100644 index 0000000..89ada90 --- /dev/null +++ b/docx/scripts/office/schemas/microsoft/wml-symex-2015.xsd @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/docx/scripts/office/soffice.py b/docx/scripts/office/soffice.py new file mode 100644 index 0000000..eec1687 --- /dev/null +++ b/docx/scripts/office/soffice.py @@ -0,0 +1,192 @@ +""" +Helper for running LibreOffice (soffice) in environments where AF_UNIX +sockets may be blocked (e.g., sandboxed VMs). Detects the restriction +at runtime and applies an LD_PRELOAD shim if needed. + +Usage: + from office.soffice import run_soffice + + result = run_soffice(["--headless", "--convert-to", "pdf", "input.docx"]) + +Call soffice through run_soffice, not through subprocess with get_soffice_env(): +the env dict carries the shim but names no user profile, and a non-root sandbox +cannot bootstrap the default one -- soffice aborts with "User installation could +not be completed" and converts nothing. get_soffice_env() stays public for the +callers that build their own argv (they must pass -env:UserInstallation too). +""" + +import contextlib +import os +import socket +import subprocess +import tempfile +from collections.abc import Iterable +from pathlib import Path + + +def get_soffice_env() -> dict: + env = os.environ.copy() + env["SAL_USE_VCLPLUGIN"] = "svp" + + if _needs_shim(): + shim = _ensure_shim() + env["LD_PRELOAD"] = str(shim) + + return env + + +def run_soffice(args: Iterable[str], **kwargs) -> subprocess.CompletedProcess: + args = list(args) + with contextlib.ExitStack() as stack: + if not any(str(a).startswith("-env:UserInstallation") for a in args): + profile = stack.enter_context( + tempfile.TemporaryDirectory(prefix="lo_profile_", ignore_cleanup_errors=True) + ) + args = [f"-env:UserInstallation={Path(profile).as_uri()}"] + args + return subprocess.run(["soffice"] + args, env=get_soffice_env(), **kwargs) + + + +_SHIM_SO = Path(tempfile.gettempdir()) / "lo_socket_shim.so" + + +def _needs_shim() -> bool: + try: + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s.close() + return False + except OSError: + return True + + +def _ensure_shim() -> Path: + if _SHIM_SO.exists(): + return _SHIM_SO + + src = Path(tempfile.gettempdir()) / "lo_socket_shim.c" + src.write_text(_SHIM_SOURCE, encoding="utf-8") + subprocess.run( + ["gcc", "-shared", "-fPIC", "-o", str(_SHIM_SO), str(src), "-ldl"], + check=True, + capture_output=True, + ) + src.unlink() + return _SHIM_SO + + + +_SHIM_SOURCE = r""" +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include + +static int (*real_socket)(int, int, int); +static int (*real_socketpair)(int, int, int, int[2]); +static int (*real_listen)(int, int); +static int (*real_accept)(int, struct sockaddr *, socklen_t *); +static int (*real_close)(int); +static int (*real_read)(int, void *, size_t); + +/* Per-FD bookkeeping (FDs >= 1024 are passed through unshimmed). */ +static int is_shimmed[1024]; +static int peer_of[1024]; +static int wake_r[1024]; /* accept() blocks reading this */ +static int wake_w[1024]; /* close() writes to this */ +static int listener_fd = -1; /* FD that received listen() */ + +__attribute__((constructor)) +static void init(void) { + real_socket = dlsym(RTLD_NEXT, "socket"); + real_socketpair = dlsym(RTLD_NEXT, "socketpair"); + real_listen = dlsym(RTLD_NEXT, "listen"); + real_accept = dlsym(RTLD_NEXT, "accept"); + real_close = dlsym(RTLD_NEXT, "close"); + real_read = dlsym(RTLD_NEXT, "read"); + for (int i = 0; i < 1024; i++) { + peer_of[i] = -1; + wake_r[i] = -1; + wake_w[i] = -1; + } +} + +/* ---- socket ---------------------------------------------------------- */ +int socket(int domain, int type, int protocol) { + if (domain == AF_UNIX) { + int fd = real_socket(domain, type, protocol); + if (fd >= 0) return fd; + /* socket(AF_UNIX) blocked – fall back to socketpair(). */ + int sv[2]; + if (real_socketpair(domain, type, protocol, sv) == 0) { + if (sv[0] >= 0 && sv[0] < 1024) { + is_shimmed[sv[0]] = 1; + peer_of[sv[0]] = sv[1]; + int wp[2]; + if (pipe(wp) == 0) { + wake_r[sv[0]] = wp[0]; + wake_w[sv[0]] = wp[1]; + } + } + return sv[0]; + } + errno = EPERM; + return -1; + } + return real_socket(domain, type, protocol); +} + +/* ---- listen ---------------------------------------------------------- */ +int listen(int sockfd, int backlog) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + listener_fd = sockfd; + return 0; + } + return real_listen(sockfd, backlog); +} + +/* ---- accept ---------------------------------------------------------- */ +int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + /* Block until close() writes to the wake pipe. */ + if (wake_r[sockfd] >= 0) { + char buf; + real_read(wake_r[sockfd], &buf, 1); + } + errno = ECONNABORTED; + return -1; + } + return real_accept(sockfd, addr, addrlen); +} + +/* ---- close ----------------------------------------------------------- */ +int close(int fd) { + if (fd >= 0 && fd < 1024 && is_shimmed[fd]) { + int was_listener = (fd == listener_fd); + is_shimmed[fd] = 0; + + if (wake_w[fd] >= 0) { /* unblock accept() */ + char c = 0; + write(wake_w[fd], &c, 1); + real_close(wake_w[fd]); + wake_w[fd] = -1; + } + if (wake_r[fd] >= 0) { real_close(wake_r[fd]); wake_r[fd] = -1; } + if (peer_of[fd] >= 0) { real_close(peer_of[fd]); peer_of[fd] = -1; } + + if (was_listener) + _exit(0); /* conversion done – exit */ + } + return real_close(fd); +} +""" + + + +if __name__ == "__main__": + import sys + result = run_soffice(sys.argv[1:]) + sys.exit(result.returncode) diff --git a/docx/scripts/office/validate.py b/docx/scripts/office/validate.py new file mode 100755 index 0000000..29ca186 --- /dev/null +++ b/docx/scripts/office/validate.py @@ -0,0 +1,173 @@ +""" +Command line tool to validate Office document XML files against XSD schemas and tracked changes. + +Usage: + python validate.py [--original ] [--auto-repair] [--author NAME] + +The first argument can be either: +- An unpacked directory containing the Office document XML files +- A packed Office file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx template) which will be unpacked to a temp directory + +Auto-repair fixes: +- paraId/durableId values that exceed OOXML limits +- Missing xml:space="preserve" on w:t elements with whitespace +""" + +import argparse +import sys +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.ElementTree as ET +from defusedxml.common import DefusedXmlException + +from helpers import OOXML_FAMILY, rezip, safe_extract +from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator + +WORD_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + + +def _fail(message: str): + print(f"Error: {message}", file=sys.stderr) + sys.exit(2) + + +def _has_tracked_changes(unpacked_dir: Path) -> bool: + document = unpacked_dir / "word" / "document.xml" + if not document.is_file(): + return False + try: + root = ET.parse(document).getroot() + except (ET.ParseError, DefusedXmlException): + return False + tracked = {f"{{{WORD_NS}}}ins", f"{{{WORD_NS}}}del"} + return any(elem.tag in tracked for elem in root.iter()) + + +def main(): + parser = argparse.ArgumentParser(description="Validate Office document XML files") + parser.add_argument( + "path", + help="Path to unpacked directory or packed Office file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx)", + ) + parser.add_argument( + "--original", + required=False, + default=None, + help="Path to original file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx). If omitted, all XSD errors are reported and redlining validation is skipped.", + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + help="Enable verbose output", + ) + parser.add_argument( + "--auto-repair", + action="store_true", + help="Automatically repair common issues (hex IDs, whitespace preservation). " + "Modifies the input in place: repairs to a packed file are written back to it.", + ) + parser.add_argument( + "--author", + default=None, + help="The name you are redlining under. Passing it turns on the " + "tracked-change check: any text differing from --original without a " + "/ recording it is reported. Untracked edits carry no " + "author, so the check covers them whoever made them — the name marks " + "the run as redlining work and is not used to filter. Requires " + "--original; docx only.", + ) + args = parser.parse_args() + + if args.author is not None and not args.original: + _fail("--author requires --original") + + path = Path(args.path) + if not path.exists(): + _fail(f"{path} does not exist") + + original_file = None + if args.original: + original_file = Path(args.original) + if not original_file.is_file(): + _fail(f"{original_file} is not a file") + if original_file.suffix.lower() not in OOXML_FAMILY: + _fail(f"{original_file} must be one of: {', '.join(sorted(OOXML_FAMILY))}") + + family = OOXML_FAMILY.get((original_file or path).suffix.lower()) + if family is None: + _fail( + f"Cannot determine file type from {path}. Use --original or provide one of: {', '.join(sorted(OOXML_FAMILY))}." + ) + + if args.author is not None and family != "docx": + _fail(f"--author only applies to docx files, not {family}") + + packed_file = None + temp_dir_ctx = None + if path.is_file() and path.suffix.lower() in OOXML_FAMILY: + packed_file = path + temp_dir_ctx = tempfile.TemporaryDirectory() + unpacked_dir = Path(temp_dir_ctx.name) + try: + with zipfile.ZipFile(path, "r") as zf: + safe_extract(zf, unpacked_dir) + except (zipfile.BadZipFile, ValueError, OSError) as e: + _fail(f"cannot unpack {path}: {e}") + else: + if not path.is_dir(): + _fail(f"{path} is not a directory or Office file") + unpacked_dir = path + + match family: + case "docx": + validators = [ + DOCXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), + ] + if args.author is not None: + validators.append( + RedliningValidator(unpacked_dir, original_file, verbose=args.verbose) + ) + elif original_file and _has_tracked_changes(unpacked_dir): + print( + "Note: this document has tracked changes; they were not " + "checked against the original (pass --author to check)." + ) + case "pptx": + validators = [ + PPTXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), + ] + case "xlsx": + exts = ", ".join(k for k, v in sorted(OOXML_FAMILY.items()) if v == "xlsx") + print( + f"No XSD schema validation is performed for xlsx-family files ({exts}). " + "For formula-error checking, use scripts/recalc.py instead." + ) + sys.exit(0) + case _: + print(f"Error: Validation not supported for file type {family}") + sys.exit(1) + + if args.auto_repair: + total_repairs = sum(v.repair() for v in validators) + if total_repairs: + print(f"Auto-repaired {total_repairs} issue(s)") + if packed_file is not None: + rezip(unpacked_dir, packed_file) + print(f"Wrote repaired file to {packed_file}") + + success = all([v.validate() for v in validators]) + + if temp_dir_ctx is not None: + temp_dir_ctx.cleanup() + + if success: + print("All validations PASSED!") + + sys.exit(0 if success else 1) + + +if __name__ == "__main__": + main() diff --git a/docx/scripts/office/validators/__init__.py b/docx/scripts/office/validators/__init__.py new file mode 100644 index 0000000..db092ec --- /dev/null +++ b/docx/scripts/office/validators/__init__.py @@ -0,0 +1,15 @@ +""" +Validation modules for Word document processing. +""" + +from .base import BaseSchemaValidator +from .docx import DOCXSchemaValidator +from .pptx import PPTXSchemaValidator +from .redlining import RedliningValidator + +__all__ = [ + "BaseSchemaValidator", + "DOCXSchemaValidator", + "PPTXSchemaValidator", + "RedliningValidator", +] diff --git a/docx/scripts/office/validators/base.py b/docx/scripts/office/validators/base.py new file mode 100644 index 0000000..f928b2c --- /dev/null +++ b/docx/scripts/office/validators/base.py @@ -0,0 +1,875 @@ +""" +Base validator with common validation logic for document files. +""" + +import re +from pathlib import Path + +import defusedxml.minidom +from functools import lru_cache + +import lxml.etree + +from helpers import safe_extract + + +@lru_cache(maxsize=None) +def _load_schema(schema_path: str): + with open(schema_path, "rb") as xsd_file: + xsd_doc = lxml.etree.parse( + xsd_file, parser=lxml.etree.XMLParser(), base_url=schema_path + ) + return lxml.etree.XMLSchema(xsd_doc) + +class BaseSchemaValidator: + + IGNORED_VALIDATION_ERRORS = [ + "hyphenationZone", + "purl.org/dc/terms", + ] + + UNIQUE_ID_REQUIREMENTS = { + "comment": ("id", "file"), + "commentrangestart": ("id", "file"), + "commentrangeend": ("id", "file"), + "bookmarkstart": ("id", "file"), + "bookmarkend": ("id", "file"), + "sldid": ("id", "file"), + "sldmasterid": ("id", "global"), + "sldlayoutid": ("id", "global"), + "cm": ("authorid", "file"), + "sheet": ("sheetid", "file"), + "definedname": ("id", "file"), + "cxnsp": ("id", "file"), + "sp": ("id", "file"), + "pic": ("id", "file"), + "grpsp": ("id", "file"), + } + + EXCLUDED_ID_CONTAINERS = { + "sectionlst", + } + + ELEMENT_RELATIONSHIP_TYPES = {} + + SCHEMA_MAPPINGS = { + "word": "ISO-IEC29500-4_2016/wml.xsd", + "ppt": "ISO-IEC29500-4_2016/pml.xsd", + "xl": "ISO-IEC29500-4_2016/sml.xsd", + "[Content_Types].xml": "ecma/fourth-edition/opc-contentTypes.xsd", + "app.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd", + "core.xml": "ecma/fourth-edition/opc-coreProperties.xsd", + "custom.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd", + ".rels": "ecma/fourth-edition/opc-relationships.xsd", + "people.xml": "microsoft/wml-2012.xsd", + "commentsIds.xml": "microsoft/wml-cid-2016.xsd", + "commentsExtensible.xml": "microsoft/wml-cex-2018.xsd", + "commentsExtended.xml": "microsoft/wml-2012.xsd", + "chart": "ISO-IEC29500-4_2016/dml-chart.xsd", + "theme": "ISO-IEC29500-4_2016/dml-main.xsd", + "drawing": "ISO-IEC29500-4_2016/dml-main.xsd", + } + + MC_NAMESPACE = "http://schemas.openxmlformats.org/markup-compatibility/2006" + XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace" + + PACKAGE_RELATIONSHIPS_NAMESPACE = ( + "http://schemas.openxmlformats.org/package/2006/relationships" + ) + OFFICE_RELATIONSHIPS_NAMESPACE = ( + "http://schemas.openxmlformats.org/officeDocument/2006/relationships" + ) + CONTENT_TYPES_NAMESPACE = ( + "http://schemas.openxmlformats.org/package/2006/content-types" + ) + + MAIN_CONTENT_FOLDERS = {"word", "ppt", "xl"} + + OOXML_NAMESPACES = { + "http://schemas.openxmlformats.org/officeDocument/2006/math", + "http://schemas.openxmlformats.org/officeDocument/2006/relationships", + "http://schemas.openxmlformats.org/schemaLibrary/2006/main", + "http://schemas.openxmlformats.org/drawingml/2006/main", + "http://schemas.openxmlformats.org/drawingml/2006/chart", + "http://schemas.openxmlformats.org/drawingml/2006/chartDrawing", + "http://schemas.openxmlformats.org/drawingml/2006/diagram", + "http://schemas.openxmlformats.org/drawingml/2006/picture", + "http://schemas.openxmlformats.org/drawingml/2006/spreadsheetDrawing", + "http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing", + "http://schemas.openxmlformats.org/wordprocessingml/2006/main", + "http://schemas.openxmlformats.org/presentationml/2006/main", + "http://schemas.openxmlformats.org/spreadsheetml/2006/main", + "http://schemas.openxmlformats.org/officeDocument/2006/sharedTypes", + "http://www.w3.org/XML/1998/namespace", + } + + def __init__(self, unpacked_dir, original_file=None, verbose=False): + self.unpacked_dir = Path(unpacked_dir).resolve() + self.original_file = Path(original_file) if original_file else None + self.verbose = verbose + + self.schemas_dir = Path(__file__).parent.parent / "schemas" + + patterns = ["*.xml", "*.rels"] + self.xml_files = [ + f for pattern in patterns for f in self.unpacked_dir.rglob(pattern) + ] + + if not self.xml_files: + print(f"Warning: No XML files found in {self.unpacked_dir}") + + def validate(self): + raise NotImplementedError("Subclasses must implement the validate method") + + def repair(self) -> int: + return self.repair_whitespace_preservation() + + def repair_whitespace_preservation(self) -> int: + repairs = 0 + + for xml_file in self.xml_files: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + pending = [] + + for elem in dom.getElementsByTagName("*"): + local_name = elem.tagName.rsplit(":", 1)[-1] + if local_name in ("t", "delText", "instrText", "delInstrText"): + text = "".join( + child.data + for child in elem.childNodes + if child.nodeType in (child.TEXT_NODE, child.CDATA_SECTION_NODE) + ) + ws = (" ", "\t", "\n", "\r") + if text and (text.startswith(ws) or text.endswith(ws)): + if elem.getAttribute("xml:space") != "preserve": + elem.setAttribute("xml:space", "preserve") + text_preview = repr(text[:30]) + "..." if len(text) > 30 else repr(text) + pending.append(f" Repaired: {xml_file.name}: Added xml:space='preserve' to {elem.tagName}: {text_preview}") + + if pending: + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + for message in pending: + print(message) + repairs += len(pending) + + except Exception: + pass + + return repairs + + def validate_xml(self): + errors = [] + + for xml_file in self.xml_files: + try: + lxml.etree.parse(str(xml_file)) + except lxml.etree.XMLSyntaxError as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {e.lineno}: {e.msg}" + ) + except Exception as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Unexpected error: {str(e)}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} XML violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All XML files are well-formed") + return True + + def validate_namespaces(self): + errors = [] + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + declared = set(root.nsmap.keys()) - {None} + + for attr_val in [ + v for k, v in root.attrib.items() if k.endswith("Ignorable") + ]: + undeclared = set(attr_val.split()) - declared + errors.extend( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Namespace '{ns}' in Ignorable but not declared" + for ns in undeclared + ) + except lxml.etree.XMLSyntaxError: + continue + + if errors: + print(f"FAILED - {len(errors)} namespace issues:") + for error in errors: + print(error) + return False + if self.verbose: + print("PASSED - All namespace prefixes properly declared") + return True + + def validate_unique_ids(self): + errors = [] + global_ids = {} + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + file_ids = {} + + mc_elements = root.xpath( + ".//mc:AlternateContent", namespaces={"mc": self.MC_NAMESPACE} + ) + for elem in mc_elements: + elem.getparent().remove(elem) + + for elem in root.iter(): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + tag = ( + elem.tag.split("}")[-1].lower() + if "}" in elem.tag + else elem.tag.lower() + ) + + if tag in self.UNIQUE_ID_REQUIREMENTS: + in_excluded_container = any( + ancestor.tag.split("}")[-1].lower() in self.EXCLUDED_ID_CONTAINERS + for ancestor in elem.iterancestors() + ) + if in_excluded_container: + continue + + attr_name, scope = self.UNIQUE_ID_REQUIREMENTS[tag] + + id_value = None + for attr, value in elem.attrib.items(): + attr_local = ( + attr.split("}")[-1].lower() + if "}" in attr + else attr.lower() + ) + if attr_local == attr_name: + id_value = value + break + + if id_value is not None: + if scope == "global": + if id_value in global_ids: + prev_file, prev_line, prev_tag = global_ids[ + id_value + ] + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: Global ID '{id_value}' in <{tag}> " + f"already used in {prev_file} at line {prev_line} in <{prev_tag}>" + ) + else: + global_ids[id_value] = ( + xml_file.relative_to(self.unpacked_dir), + elem.sourceline, + tag, + ) + elif scope == "file": + key = (tag, attr_name) + if key not in file_ids: + file_ids[key] = {} + + if id_value in file_ids[key]: + prev_line = file_ids[key][id_value] + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: Duplicate {attr_name}='{id_value}' in <{tag}> " + f"(first occurrence at line {prev_line})" + ) + else: + file_ids[key][id_value] = elem.sourceline + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} ID uniqueness violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All required IDs are unique") + return True + + def validate_file_references(self): + errors = [] + + rels_files = list(self.unpacked_dir.rglob("*.rels")) + + if not rels_files: + if self.verbose: + print("PASSED - No .rels files found") + return True + + all_files = [] + for file_path in self.unpacked_dir.rglob("*"): + if ( + file_path.is_file() + and file_path.name != "[Content_Types].xml" + and not file_path.name.endswith(".rels") + ): + all_files.append(file_path.resolve()) + + all_referenced_files = set() + + if self.verbose: + print( + f"Found {len(rels_files)} .rels files and {len(all_files)} target files" + ) + + for rels_file in rels_files: + try: + rels_root = lxml.etree.parse(str(rels_file)).getroot() + + rels_dir = rels_file.parent + + referenced_files = set() + broken_refs = [] + + for rel in rels_root.findall( + ".//ns:Relationship", + namespaces={"ns": self.PACKAGE_RELATIONSHIPS_NAMESPACE}, + ): + target = rel.get("Target") + if rel.get("TargetMode") == "External": + continue + if target and not target.startswith( + ("http", "mailto:") + ): + if target.startswith("/"): + target_path = self.unpacked_dir / target.lstrip("/") + elif rels_file.name == ".rels": + target_path = self.unpacked_dir / target + else: + base_dir = rels_dir.parent + target_path = base_dir / target + + try: + target_path = target_path.resolve() + if target_path.exists() and target_path.is_file(): + referenced_files.add(target_path) + all_referenced_files.add(target_path) + else: + broken_refs.append((target, rel.sourceline)) + except (OSError, ValueError): + broken_refs.append((target, rel.sourceline)) + + if broken_refs: + rel_path = rels_file.relative_to(self.unpacked_dir) + for broken_ref, line_num in broken_refs: + errors.append( + f" {rel_path}: Line {line_num}: Broken reference to {broken_ref}" + ) + + except Exception as e: + rel_path = rels_file.relative_to(self.unpacked_dir) + errors.append(f" Error parsing {rel_path}: {e}") + + unreferenced_files = set(all_files) - all_referenced_files + + if unreferenced_files: + for unref_file in sorted(unreferenced_files): + unref_rel_path = unref_file.relative_to(self.unpacked_dir) + errors.append(f" Unreferenced file: {unref_rel_path}") + + if errors: + print(f"FAILED - Found {len(errors)} relationship validation errors:") + for error in errors: + print(error) + print( + "CRITICAL: These errors will cause the document to appear corrupt. " + + "Broken references MUST be fixed, " + + "and unreferenced files MUST be referenced or removed." + ) + return False + else: + if self.verbose: + print( + "PASSED - All references are valid and all files are properly referenced" + ) + return True + + def validate_all_relationship_ids(self): + import lxml.etree + + errors = [] + + for xml_file in self.xml_files: + if xml_file.suffix == ".rels": + continue + + rels_dir = xml_file.parent / "_rels" + rels_file = rels_dir / f"{xml_file.name}.rels" + + if not rels_file.exists(): + continue + + try: + rels_root = lxml.etree.parse(str(rels_file)).getroot() + rid_to_type = {} + + for rel in rels_root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rid = rel.get("Id") + rel_type = rel.get("Type", "") + if rid: + if rid in rid_to_type: + rels_rel_path = rels_file.relative_to(self.unpacked_dir) + errors.append( + f" {rels_rel_path}: Line {rel.sourceline}: " + f"Duplicate relationship ID '{rid}' (IDs must be unique)" + ) + type_name = ( + rel_type.split("/")[-1] if "/" in rel_type else rel_type + ) + rid_to_type[rid] = type_name + + xml_root = lxml.etree.parse(str(xml_file)).getroot() + + r_ns = self.OFFICE_RELATIONSHIPS_NAMESPACE + rid_attrs_to_check = ["id", "embed", "link"] + for elem in xml_root.iter(): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + for attr_name in rid_attrs_to_check: + rid_attr = elem.get(f"{{{r_ns}}}{attr_name}") + if not rid_attr: + continue + xml_rel_path = xml_file.relative_to(self.unpacked_dir) + elem_name = ( + elem.tag.split("}")[-1] if "}" in elem.tag else elem.tag + ) + + if rid_attr not in rid_to_type: + errors.append( + f" {xml_rel_path}: Line {elem.sourceline}: " + f"<{elem_name}> r:{attr_name} references non-existent relationship '{rid_attr}' " + f"(valid IDs: {', '.join(sorted(rid_to_type.keys())[:5])}{'...' if len(rid_to_type) > 5 else ''})" + ) + elif attr_name == "id" and self.ELEMENT_RELATIONSHIP_TYPES: + expected_type = self._get_expected_relationship_type( + elem_name + ) + if expected_type: + actual_type = rid_to_type[rid_attr] + if expected_type not in actual_type.lower(): + errors.append( + f" {xml_rel_path}: Line {elem.sourceline}: " + f"<{elem_name}> references '{rid_attr}' which points to '{actual_type}' " + f"but should point to a '{expected_type}' relationship" + ) + + except Exception as e: + xml_rel_path = xml_file.relative_to(self.unpacked_dir) + errors.append(f" Error processing {xml_rel_path}: {e}") + + if errors: + print(f"FAILED - Found {len(errors)} relationship ID reference errors:") + for error in errors: + print(error) + print("\nThese ID mismatches will cause the document to appear corrupt!") + return False + else: + if self.verbose: + print("PASSED - All relationship ID references are valid") + return True + + def _get_expected_relationship_type(self, element_name): + elem_lower = element_name.lower() + + if elem_lower in self.ELEMENT_RELATIONSHIP_TYPES: + return self.ELEMENT_RELATIONSHIP_TYPES[elem_lower] + + if elem_lower.endswith("id") and len(elem_lower) > 2: + prefix = elem_lower[:-2] + if prefix.endswith("master"): + return prefix.lower() + elif prefix.endswith("layout"): + return prefix.lower() + else: + if prefix == "sld": + return "slide" + return prefix.lower() + + if elem_lower.endswith("reference") and len(elem_lower) > 9: + prefix = elem_lower[:-9] + return prefix.lower() + + return None + + def validate_content_types(self): + errors = [] + + content_types_file = self.unpacked_dir / "[Content_Types].xml" + if not content_types_file.exists(): + print("FAILED - [Content_Types].xml file not found") + return False + + try: + root = lxml.etree.parse(str(content_types_file)).getroot() + declared_parts = set() + declared_extensions = set() + + for override in root.findall( + f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Override" + ): + part_name = override.get("PartName") + if part_name is not None: + declared_parts.add(part_name.lstrip("/")) + + for default in root.findall( + f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Default" + ): + extension = default.get("Extension") + if extension is not None: + declared_extensions.add(extension.lower()) + + declarable_roots = { + "sld", + "sldLayout", + "sldMaster", + "presentation", + "document", + "workbook", + "worksheet", + "theme", + } + + media_extensions = { + "png": "image/png", + "jpg": "image/jpeg", + "jpeg": "image/jpeg", + "gif": "image/gif", + "bmp": "image/bmp", + "tiff": "image/tiff", + "wmf": "image/x-wmf", + "emf": "image/x-emf", + } + + all_files = list(self.unpacked_dir.rglob("*")) + all_files = [f for f in all_files if f.is_file()] + + for xml_file in self.xml_files: + path_str = str(xml_file.relative_to(self.unpacked_dir)).replace( + "\\", "/" + ) + + if any( + skip in path_str + for skip in [".rels", "[Content_Types]", "docProps/", "_rels/"] + ): + continue + + try: + root_tag = lxml.etree.parse(str(xml_file)).getroot().tag + root_name = root_tag.split("}")[-1] if "}" in root_tag else root_tag + + if root_name in declarable_roots and path_str not in declared_parts: + errors.append( + f" {path_str}: File with <{root_name}> root not declared in [Content_Types].xml" + ) + + except Exception: + continue + + for file_path in all_files: + if file_path.suffix.lower() in {".xml", ".rels"}: + continue + if file_path.name == "[Content_Types].xml": + continue + if "_rels" in file_path.parts or "docProps" in file_path.parts: + continue + + extension = file_path.suffix.lstrip(".").lower() + if extension and extension not in declared_extensions: + if extension in media_extensions: + relative_path = file_path.relative_to(self.unpacked_dir) + errors.append( + f' {relative_path}: File with extension \'{extension}\' not declared in [Content_Types].xml - should add: ' + ) + + except Exception as e: + errors.append(f" Error parsing [Content_Types].xml: {e}") + + if errors: + print(f"FAILED - Found {len(errors)} content type declaration errors:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print( + "PASSED - All content files are properly declared in [Content_Types].xml" + ) + return True + + def validate_file_against_xsd(self, xml_file, verbose=False): + xml_file = Path(xml_file).resolve() + unpacked_dir = self.unpacked_dir.resolve() + + is_valid, current_errors = self._validate_single_file_xsd( + xml_file, unpacked_dir + ) + + if is_valid is None: + return None, set() + elif is_valid: + return True, set() + + original_errors = self._get_original_file_errors(xml_file) + + assert current_errors is not None + new_errors = current_errors - original_errors + + new_errors = { + e for e in new_errors + if not any(pattern in e for pattern in self.IGNORED_VALIDATION_ERRORS) + } + + if new_errors: + if verbose: + relative_path = xml_file.relative_to(unpacked_dir) + print(f"FAILED - {relative_path}: {len(new_errors)} new error(s)") + for error in list(new_errors)[:3]: + truncated = error[:250] + "..." if len(error) > 250 else error + print(f" - {truncated}") + return False, new_errors + else: + if verbose: + print( + f"PASSED - No new errors (original had {len(current_errors)} errors)" + ) + return True, set() + + def validate_against_xsd(self): + new_errors = [] + original_error_count = 0 + valid_count = 0 + skipped_count = 0 + + for xml_file in self.xml_files: + relative_path = str(xml_file.relative_to(self.unpacked_dir)) + is_valid, new_file_errors = self.validate_file_against_xsd( + xml_file, verbose=False + ) + + if is_valid is None: + skipped_count += 1 + continue + elif is_valid and not new_file_errors: + valid_count += 1 + continue + elif is_valid: + original_error_count += 1 + valid_count += 1 + continue + + new_errors.append(f" {relative_path}: {len(new_file_errors)} new error(s)") + for error in list(new_file_errors)[:3]: + new_errors.append( + f" - {error[:250]}..." if len(error) > 250 else f" - {error}" + ) + + if self.verbose: + print(f"Validated {len(self.xml_files)} files:") + print(f" - Valid: {valid_count}") + print(f" - Skipped (no schema): {skipped_count}") + if original_error_count: + print(f" - With original errors (ignored): {original_error_count}") + print( + f" - With NEW errors: {len(new_errors) > 0 and len([e for e in new_errors if not e.startswith(' ')]) or 0}" + ) + + if new_errors: + print("\nFAILED - Found NEW validation errors:") + for error in new_errors: + print(error) + return False + else: + if self.verbose: + print("\nPASSED - No new XSD validation errors introduced") + return True + + def _get_schema_path(self, xml_file): + if xml_file.name in self.SCHEMA_MAPPINGS: + return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.name] + + if xml_file.suffix == ".rels": + return self.schemas_dir / self.SCHEMA_MAPPINGS[".rels"] + + if "charts/" in str(xml_file) and xml_file.name.startswith("chart"): + return self.schemas_dir / self.SCHEMA_MAPPINGS["chart"] + + if "theme/" in str(xml_file) and xml_file.name.startswith("theme"): + return self.schemas_dir / self.SCHEMA_MAPPINGS["theme"] + + if xml_file.parent.name in self.MAIN_CONTENT_FOLDERS: + return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.parent.name] + + return None + + def _clean_ignorable_namespaces(self, xml_doc): + xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") + xml_copy = lxml.etree.fromstring(xml_string) + + for elem in xml_copy.iter(): + attrs_to_remove = [] + + for attr in elem.attrib: + if "{" in attr: + ns = attr.split("}")[0][1:] + if ns not in self.OOXML_NAMESPACES: + attrs_to_remove.append(attr) + + for attr in attrs_to_remove: + del elem.attrib[attr] + + self._remove_ignorable_elements(xml_copy) + + return lxml.etree.ElementTree(xml_copy) + + def _remove_ignorable_elements(self, root): + elements_to_remove = [] + + for elem in list(root): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + + tag_str = str(elem.tag) + if tag_str.startswith("{"): + ns = tag_str.split("}")[0][1:] + if ns not in self.OOXML_NAMESPACES: + elements_to_remove.append(elem) + continue + + self._remove_ignorable_elements(elem) + + for elem in elements_to_remove: + root.remove(elem) + + def _preprocess_for_mc_ignorable(self, xml_doc): + root = xml_doc.getroot() + + if f"{{{self.MC_NAMESPACE}}}Ignorable" in root.attrib: + del root.attrib[f"{{{self.MC_NAMESPACE}}}Ignorable"] + + return xml_doc + + def _preprocess_for_schema(self, xml_doc, relative_path): + return xml_doc + + def _validate_single_file_xsd(self, xml_file, base_path, schema_path=None): + schema_path = schema_path or self._get_schema_path(xml_file) + if not schema_path: + return None, None + + try: + schema = _load_schema(str(schema_path)) + + with open(xml_file, "rb") as f: + xml_doc = lxml.etree.parse(f) + + xml_doc, _ = self._remove_template_tags_from_text_nodes(xml_doc) + xml_doc = self._preprocess_for_mc_ignorable(xml_doc) + + relative_path = xml_file.relative_to(base_path) + if ( + relative_path.parts + and relative_path.parts[0] in self.MAIN_CONTENT_FOLDERS + ): + xml_doc = self._clean_ignorable_namespaces(xml_doc) + + xml_doc = self._preprocess_for_schema(xml_doc, relative_path) + + if schema.validate(xml_doc): + return True, set() + else: + errors = set() + for error in schema.error_log: + errors.add(error.message) + return False, errors + + except Exception as e: + return False, {str(e)} + + def _get_original_file_errors(self, xml_file, schema_path=None): + if self.original_file is None: + return set() + + import tempfile + import zipfile + + xml_file = Path(xml_file).resolve() + unpacked_dir = self.unpacked_dir.resolve() + relative_path = xml_file.relative_to(unpacked_dir) + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + try: + with zipfile.ZipFile(self.original_file, "r") as zip_ref: + safe_extract(zip_ref, temp_path) + except (zipfile.BadZipFile, ValueError, OSError): + return set() + + original_xml_file = temp_path / relative_path + + if not original_xml_file.exists(): + return set() + + is_valid, errors = self._validate_single_file_xsd( + original_xml_file, temp_path, schema_path=schema_path + ) + return errors if errors else set() + + def _remove_template_tags_from_text_nodes(self, xml_doc): + warnings = [] + template_pattern = re.compile(r"\{\{[^}]*\}\}") + + xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") + xml_copy = lxml.etree.fromstring(xml_string) + + def process_text_content(text, content_type): + if not text: + return text + matches = list(template_pattern.finditer(text)) + if matches: + for match in matches: + warnings.append( + f"Found template tag in {content_type}: {match.group()}" + ) + return template_pattern.sub("", text) + return text + + for elem in xml_copy.iter(): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + tag_str = str(elem.tag) + if tag_str.endswith("}t") or tag_str == "t": + continue + + elem.text = process_text_content(elem.text, "text content") + elem.tail = process_text_content(elem.tail, "tail content") + + return lxml.etree.ElementTree(xml_copy), warnings + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/docx/scripts/office/validators/docx.py b/docx/scripts/office/validators/docx.py new file mode 100644 index 0000000..b181499 --- /dev/null +++ b/docx/scripts/office/validators/docx.py @@ -0,0 +1,466 @@ +""" +Validator for Word document XML files against XSD schemas. +""" + +import random +import re +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.minidom +import lxml.etree + +from helpers import safe_extract + +from .base import BaseSchemaValidator + + +class DOCXSchemaValidator(BaseSchemaValidator): + + WORD_2006_NAMESPACE = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + W14_NAMESPACE = "http://schemas.microsoft.com/office/word/2010/wordml" + W16CID_NAMESPACE = "http://schemas.microsoft.com/office/word/2016/wordml/cid" + + ELEMENT_RELATIONSHIP_TYPES = {} + + def validate(self): + if not self.validate_xml(): + return False + + all_valid = True + if not self.validate_namespaces(): + all_valid = False + + if not self.validate_unique_ids(): + all_valid = False + + if not self.validate_file_references(): + all_valid = False + + if not self.validate_content_types(): + all_valid = False + + if not self.validate_against_xsd(): + all_valid = False + + if not self.validate_whitespace_preservation(): + all_valid = False + + if not self.validate_deletions(): + all_valid = False + + if not self.validate_insertions(): + all_valid = False + + if not self.validate_all_relationship_ids(): + all_valid = False + + if not self.validate_id_constraints(): + all_valid = False + + if not self.validate_comment_markers(): + all_valid = False + + self.compare_paragraph_counts() + + return all_valid + + def validate_whitespace_preservation(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + + for elem in root.iter(f"{{{self.WORD_2006_NAMESPACE}}}t"): + if elem.text: + text = elem.text + if re.search(r"^[ \t\n\r]", text) or re.search( + r"[ \t\n\r]$", text + ): + xml_space_attr = f"{{{self.XML_NAMESPACE}}}space" + if ( + xml_space_attr not in elem.attrib + or elem.attrib[xml_space_attr] != "preserve" + ): + text_preview = ( + repr(text)[:50] + "..." + if len(repr(text)) > 50 + else repr(text) + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: w:t element with whitespace missing xml:space='preserve': {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} whitespace preservation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All whitespace is properly preserved") + return True + + def validate_deletions(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + for t_elem in root.xpath(".//w:del//w:t", namespaces=namespaces): + if t_elem.text: + text_preview = ( + repr(t_elem.text)[:50] + "..." + if len(repr(t_elem.text)) > 50 + else repr(t_elem.text) + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {t_elem.sourceline}: found within : {text_preview}" + ) + + for instr_elem in root.xpath( + ".//w:del//w:instrText", namespaces=namespaces + ): + text_preview = ( + repr(instr_elem.text or "")[:50] + "..." + if len(repr(instr_elem.text or "")) > 50 + else repr(instr_elem.text or "") + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {instr_elem.sourceline}: found within (use ): {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} deletion validation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - No w:t elements found within w:del elements") + return True + + def count_paragraphs_in_unpacked(self): + count = 0 + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") + count = len(paragraphs) + except Exception as e: + print(f"Error counting paragraphs in unpacked document: {e}") + + return count + + def count_paragraphs_in_original(self): + original = self.original_file + if original is None: + return 0 + + count = 0 + + try: + with tempfile.TemporaryDirectory() as temp_dir: + with zipfile.ZipFile(original, "r") as zip_ref: + safe_extract(zip_ref, Path(temp_dir)) + + doc_xml_path = temp_dir + "/word/document.xml" + root = lxml.etree.parse(doc_xml_path).getroot() + + paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") + count = len(paragraphs) + + except Exception as e: + print(f"Error counting paragraphs in original document: {e}") + + return count + + def validate_insertions(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + invalid_elements = root.xpath( + ".//w:ins//w:delText[not(ancestor::w:del)]", namespaces=namespaces + ) + + for elem in invalid_elements: + text_preview = ( + repr(elem.text or "")[:50] + "..." + if len(repr(elem.text or "")) > 50 + else repr(elem.text or "") + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: within : {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} insertion validation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - No w:delText elements within w:ins elements") + return True + + def compare_paragraph_counts(self): + new_count = self.count_paragraphs_in_unpacked() + if self.original_file is None: + print(f"\nParagraphs: {new_count}") + return + + original_count = self.count_paragraphs_in_original() + diff = new_count - original_count + diff_str = f"+{diff}" if diff > 0 else str(diff) + print(f"\nParagraphs: {original_count} → {new_count} ({diff_str})") + + def _parse_id_value(self, val: str, base: int = 16) -> int: + return int(val, base) + + def validate_id_constraints(self): + errors = [] + para_id_attr = f"{{{self.W14_NAMESPACE}}}paraId" + durable_id_attr = f"{{{self.W16CID_NAMESPACE}}}durableId" + + for xml_file in self.xml_files: + try: + for elem in lxml.etree.parse(str(xml_file)).iter(): + if val := elem.get(para_id_attr): + try: + if self._parse_id_value(val, base=16) >= 0x80000000: + errors.append( + f" {xml_file.name}:{elem.sourceline}: paraId={val} >= 0x80000000" + ) + except ValueError: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"paraId={val} is not valid hex" + ) + + if val := elem.get(durable_id_attr): + if xml_file.name == "numbering.xml": + try: + if self._parse_id_value(val, base=10) >= 0x7FFFFFFF: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} >= 0x7FFFFFFF" + ) + except ValueError: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} must be decimal in numbering.xml" + ) + else: + try: + if self._parse_id_value(val, base=16) >= 0x7FFFFFFF: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} >= 0x7FFFFFFF" + ) + except ValueError: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} is not valid hex" + ) + except lxml.etree.XMLSyntaxError: + continue + + if errors: + print(f"FAILED - {len(errors)} ID constraint violations:") + for e in errors: + print(e) + elif self.verbose: + print("PASSED - All paraId/durableId values within constraints") + return not errors + + def validate_comment_markers(self): + errors = [] + + document_xml = None + comments_xml = None + for xml_file in self.xml_files: + if xml_file.name == "document.xml" and "word" in str(xml_file): + document_xml = xml_file + elif xml_file.name == "comments.xml": + comments_xml = xml_file + + if not document_xml: + if self.verbose: + print("PASSED - No document.xml found (skipping comment validation)") + return True + + try: + doc_root = lxml.etree.parse(str(document_xml)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + range_starts = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentRangeStart", namespaces=namespaces + ) + } + range_ends = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentRangeEnd", namespaces=namespaces + ) + } + references = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentReference", namespaces=namespaces + ) + } + + orphaned_ends = range_ends - range_starts + for comment_id in sorted( + orphaned_ends, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + errors.append( + f' document.xml: commentRangeEnd id="{comment_id}" has no matching commentRangeStart' + ) + + orphaned_starts = range_starts - range_ends + for comment_id in sorted( + orphaned_starts, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + errors.append( + f' document.xml: commentRangeStart id="{comment_id}" has no matching commentRangeEnd' + ) + + comment_ids = set() + if comments_xml and comments_xml.exists(): + comments_root = lxml.etree.parse(str(comments_xml)).getroot() + comment_ids = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in comments_root.xpath( + ".//w:comment", namespaces=namespaces + ) + } + + marker_ids = range_starts | range_ends | references + invalid_refs = marker_ids - comment_ids + for comment_id in sorted( + invalid_refs, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + if comment_id: + errors.append( + f' document.xml: marker id="{comment_id}" references non-existent comment' + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append(f" Error parsing XML: {e}") + + if errors: + print(f"FAILED - {len(errors)} comment marker violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All comment markers properly paired") + return True + + def repair(self) -> int: + repairs = super().repair() + repairs += self.repair_durableId() + return repairs + + def repair_durableId(self) -> int: + DURABLE_ID_ATTRS = ("w16cid:durableId", "w16cex:durableId") + repairs = 0 + renames: dict = {} + + for xml_file in self.xml_files: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + is_numbering = xml_file.name == "numbering.xml" + base = 10 if is_numbering else 16 + pending = [] + seen_in_file = set() + modified = False + + for elem in dom.getElementsByTagName("*"): + for attr_name in DURABLE_ID_ATTRS: + if not elem.hasAttribute(attr_name): + continue + + durable_id = elem.getAttribute(attr_name) + try: + key = self._parse_id_value(durable_id, base=base) + needs_repair = key >= 0x7FFFFFFF + except ValueError: + key = durable_id + needs_repair = True + + if needs_repair: + if key in seen_in_file: + value = random.randint(1, 0x7FFFFFFE) + else: + seen_in_file.add(key) + if key not in renames: + renames[key] = random.randint(1, 0x7FFFFFFE) + value = renames[key] + new_id = str(value) if is_numbering else f"{value:08X}" + + elem.setAttribute(attr_name, new_id) + pending.append( + f" Repaired: {xml_file.name}: durableId {durable_id} → {new_id}" + ) + modified = True + + if modified: + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + for message in pending: + print(message) + repairs += len(pending) + + except Exception: + pass + + return repairs + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/docx/scripts/office/validators/pptx.py b/docx/scripts/office/validators/pptx.py new file mode 100644 index 0000000..318f0e6 --- /dev/null +++ b/docx/scripts/office/validators/pptx.py @@ -0,0 +1,441 @@ +""" +Validator for PowerPoint presentation XML files against XSD schemas. +""" + +import re +from pathlib import Path + +from helpers import opc_target, rels_source_part, safe_extract + +from .base import BaseSchemaValidator + + +class PPTXSchemaValidator(BaseSchemaValidator): + + PRESENTATIONML_NAMESPACE = ( + "http://schemas.openxmlformats.org/presentationml/2006/main" + ) + + ELEMENT_RELATIONSHIP_TYPES = { + "sldid": "slide", + "sldmasterid": "slidemaster", + "notesmasterid": "notesmaster", + "sldlayoutid": "slidelayout", + "themeid": "theme", + "tablestyleid": "tablestyles", + } + + def validate(self): + if not self.validate_xml(): + return False + + all_valid = True + if not self.validate_namespaces(): + all_valid = False + + if not self.validate_unique_ids(): + all_valid = False + + if not self.validate_uuid_ids(): + all_valid = False + + if not self.validate_file_references(): + all_valid = False + + if not self.validate_slide_layout_ids(): + all_valid = False + + if not self.validate_content_types(): + all_valid = False + + if not self.validate_against_xsd(): + all_valid = False + + if not self.validate_notes_slide_references(): + all_valid = False + + if not self.validate_all_relationship_ids(): + all_valid = False + + if not self.validate_no_duplicate_slide_layouts(): + all_valid = False + + if not self.validate_master_theme_uniqueness(): + all_valid = False + + if not self.validate_charts(): + all_valid = False + + if not self.validate_slides(): + all_valid = False + + return all_valid + + def _package_map(self) -> dict: + wanted = [] + wanted += list(self.unpacked_dir.glob("[[]Content_Types[]].xml")) + wanted += list(self.unpacked_dir.glob("ppt/presentation.xml")) + wanted += list(self.unpacked_dir.glob("ppt/theme/*.xml")) + wanted += list(self.unpacked_dir.glob("ppt/theme/_rels/*.rels")) + wanted += list(self.unpacked_dir.glob("ppt/charts/chart*.xml")) + for group in ("slideMasters", "notesMasters", "handoutMasters"): + wanted += list(self.unpacked_dir.glob(f"ppt/{group}/*.xml")) + wanted += list(self.unpacked_dir.glob(f"ppt/{group}/_rels/*.rels")) + return { + p.relative_to(self.unpacked_dir).as_posix(): p.read_bytes() + for p in wanted + if p.is_file() + } + + def validate_master_theme_uniqueness(self): + from helpers.pptx_theme import _NOTES_MASTERS, live_shared_master_themes + + shared = live_shared_master_themes(self._package_map()) + if shared: + print(f"FAILED - Found {len(shared)} master(s) sharing a theme part:") + for message in shared: + print(f" {message}") + if any(m.startswith(_NOTES_MASTERS) for m in shared): + print(" Fix: in ppt/presentation.xml, move back to " + "directly after . PowerPoint reads that happily.") + else: + print(" Fix: give each master its own theme part.") + return False + + if self.verbose: + print("PASSED - No master shares a theme part in a way PowerPoint refuses") + return True + + def validate_charts(self): + from helpers.pptx_chart import find_chart_problems + + problems = find_chart_problems(self._package_map()) + if problems: + print(f"FAILED - Found {len(problems)} chart problem(s) PowerPoint rejects:") + for message in problems: + print(f" {message}") + return False + + if self.verbose: + print("PASSED - Charts satisfy the constraints PowerPoint enforces") + return True + + def _original_slide_defects(self, schema) -> set[str]: + import tempfile + import zipfile + + from helpers.pptx_slide import SLIDE_PART_RE, fatal_slide_errors + + if self.original_file is None: + return set() + + found: set[str] = set() + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + try: + with zipfile.ZipFile(self.original_file, "r") as zf: + safe_extract(zf, temp_path) + except (zipfile.BadZipFile, ValueError, OSError): + return set() + + for part in sorted(temp_path.rglob("*.xml")): + relative = part.relative_to(temp_path).as_posix() + if not SLIDE_PART_RE.fullmatch(relative): + continue + ok, errors = self._validate_single_file_xsd( + part.resolve(), temp_path.resolve(), schema_path=schema + ) + if ok is None or ok or not errors: + continue + found |= set(fatal_slide_errors(set(errors))) + return found + + def validate_slides(self): + from helpers.pptx_slide import ( + SLIDE_PART_RE, + fatal_slide_errors, + is_schema_verdict, + ) + + schema = self.schemas_dir / self.SCHEMA_MAPPINGS["ppt"] + inherited = self._original_slide_defects(schema) + problems: list[str] = [] + broken: list[str] = [] + + for xml_file in self.xml_files: + relative = xml_file.relative_to(self.unpacked_dir).as_posix() + if not SLIDE_PART_RE.fullmatch(relative): + continue + ok, errors = self._validate_single_file_xsd( + xml_file.resolve(), self.unpacked_dir.resolve(), schema_path=schema + ) + if ok is None or not errors: + continue + + unreadable = [f"{relative}: {e}" for e in errors if not is_schema_verdict(e)] + if unreadable: + broken.extend(unreadable) + continue + if ok: + continue + + for message in fatal_slide_errors(set(errors)): + if message in inherited: + continue + problems.append(f"{relative}: {message}") + + if broken: + print(f"FAILED - Could not check {len(broken)} slide part(s):") + for message in sorted(broken): + print(f" {message[:240]}") + + if problems: + print(f"FAILED - Found {len(problems)} slide problem(s) PowerPoint rejects:") + for message in sorted(problems): + print(f" {message[:240]}") + + if broken or problems: + return False + + if self.verbose: + print("PASSED - Slide XML has none of the defects PowerPoint refuses") + return True + + def _get_schema_path(self, xml_file): + if xml_file.parent.name == "charts" and xml_file.name.startswith("chart"): + return None + return super()._get_schema_path(xml_file) + + def _preprocess_for_schema(self, xml_doc, relative_path): + if relative_path.as_posix() != "ppt/presentation.xml": + return xml_doc + + root = xml_doc.getroot() + ns = f"{{{self.PRESENTATIONML_NAMESPACE}}}" + notes = root.find(f"{ns}notesMasterIdLst") + slides = root.find(f"{ns}sldIdLst") + if notes is None or slides is None: + return xml_doc + + children = list(root) + if children.index(notes) < children.index(slides): + return xml_doc + + root.remove(notes) + root.insert(list(root).index(slides), notes) + return xml_doc + + def validate_uuid_ids(self): + import lxml.etree + + errors = [] + uuid_pattern = re.compile( + r"^[\{\(]?[0-9A-Fa-f]{8}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{12}[\}\)]?$" + ) + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + + for elem in root.iter(): + for attr, value in elem.attrib.items(): + attr_name = attr.split("}")[-1].lower() + if attr_name == "id" or attr_name.endswith("id"): + if self._looks_like_uuid(value): + if not uuid_pattern.match(value): + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: ID '{value}' appears to be a UUID but contains invalid hex characters" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} UUID ID validation errors:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All UUID-like IDs contain valid hex values") + return True + + def _looks_like_uuid(self, value): + clean_value = value.strip("{}()").replace("-", "") + return len(clean_value) == 32 and all(c.isalnum() for c in clean_value) + + def validate_slide_layout_ids(self): + import lxml.etree + + errors = [] + + slide_masters = list(self.unpacked_dir.glob("ppt/slideMasters/*.xml")) + + if not slide_masters: + if self.verbose: + print("PASSED - No slide masters found") + return True + + for slide_master in slide_masters: + try: + root = lxml.etree.parse(str(slide_master)).getroot() + + rels_file = slide_master.parent / "_rels" / f"{slide_master.name}.rels" + + if not rels_file.exists(): + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: " + f"Missing relationships file: {rels_file.relative_to(self.unpacked_dir)}" + ) + continue + + rels_root = lxml.etree.parse(str(rels_file)).getroot() + + valid_layout_rids = set() + for rel in rels_root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rel_type = rel.get("Type", "") + if "slideLayout" in rel_type: + valid_layout_rids.add(rel.get("Id")) + + for sld_layout_id in root.findall( + f".//{{{self.PRESENTATIONML_NAMESPACE}}}sldLayoutId" + ): + r_id = sld_layout_id.get( + f"{{{self.OFFICE_RELATIONSHIPS_NAMESPACE}}}id" + ) + layout_id = sld_layout_id.get("id") + + if r_id and r_id not in valid_layout_rids: + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: " + f"Line {sld_layout_id.sourceline}: sldLayoutId with id='{layout_id}' " + f"references r:id='{r_id}' which is not found in slide layout relationships" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} slide layout ID validation errors:") + for error in errors: + print(error) + print( + "Remove invalid references or add missing slide layouts to the relationships file." + ) + return False + else: + if self.verbose: + print("PASSED - All slide layout IDs reference valid slide layouts") + return True + + def validate_no_duplicate_slide_layouts(self): + import lxml.etree + + errors = [] + slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) + + for rels_file in slide_rels_files: + try: + root = lxml.etree.parse(str(rels_file)).getroot() + + layout_rels = [ + rel + for rel in root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ) + if "slideLayout" in rel.get("Type", "") + ] + + if len(layout_rels) > 1: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: has {len(layout_rels)} slideLayout references" + ) + + except Exception as e: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print("FAILED - Found slides with duplicate slideLayout references:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All slides have exactly one slideLayout reference") + return True + + def validate_notes_slide_references(self): + import lxml.etree + + errors = [] + notes_slide_references = {} + + slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) + + if not slide_rels_files: + if self.verbose: + print("PASSED - No slide relationship files found") + return True + + for rels_file in slide_rels_files: + try: + root = lxml.etree.parse(str(rels_file)).getroot() + + for rel in root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rel_type = rel.get("Type", "") + if "notesSlide" in rel_type: + part = opc_target( + rel.get("Target", ""), + rels_source_part(rels_file, self.unpacked_dir), + rel.get("TargetMode", ""), + ) + if part: + slide_name = rels_file.stem.replace( + ".xml", "" + ) + + notes_slide_references.setdefault(part, []).append( + (slide_name, rels_file) + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + for target, references in notes_slide_references.items(): + if len(references) > 1: + slide_names = [ref[0] for ref in references] + errors.append( + f" Notes slide '{target}' is referenced by multiple slides: {', '.join(slide_names)}" + ) + for slide_name, rels_file in references: + errors.append(f" - {rels_file.relative_to(self.unpacked_dir)}") + + if errors: + print( + f"FAILED - Found {len([e for e in errors if not e.startswith(' ')])} notes slide reference validation errors:" + ) + for error in errors: + print(error) + print("Each slide may optionally have its own slide file.") + return False + else: + if self.verbose: + print("PASSED - All notes slide references are unique") + return True + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/docx/scripts/office/validators/redlining.py b/docx/scripts/office/validators/redlining.py new file mode 100644 index 0000000..7d3a082 --- /dev/null +++ b/docx/scripts/office/validators/redlining.py @@ -0,0 +1,299 @@ +""" +Validator for tracked changes in Word documents. + +Detects untracked edits in word/document.xml: text that differs from the +original without a / wrapper recording it. The tracked changes +that are new relative to the original are undone, and the result is compared +against the original; whatever text still differs was edited without being +tracked. + +Only the document body is compared. Headers, footers, footnotes and endnotes +are separate parts and are not checked. +""" + +import subprocess +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.ElementTree as ET +from defusedxml.common import DefusedXmlException + +from helpers import rendered_text, safe_extract + + +class RedliningValidator: + + def __init__(self, unpacked_dir, original_docx, verbose=False): + self.unpacked_dir = Path(unpacked_dir) + self.original_docx = Path(original_docx) + self.verbose = verbose + self.namespaces = { + "w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + } + + def repair(self) -> int: + return 0 + + def validate(self): + modified_file = self.unpacked_dir / "word" / "document.xml" + if not modified_file.exists(): + print(f"FAILED - Modified document.xml not found at {modified_file}") + return False + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + try: + with zipfile.ZipFile(self.original_docx, "r") as zip_ref: + safe_extract(zip_ref, temp_path) + except Exception as e: + print(f"FAILED - Error unpacking original docx: {e}") + return False + + original_file = temp_path / "word" / "document.xml" + if not original_file.exists(): + print( + f"FAILED - Original document.xml not found in {self.original_docx}" + ) + return False + + try: + modified_tree = ET.parse(modified_file) + modified_root = modified_tree.getroot() + original_tree = ET.parse(original_file) + original_root = original_tree.getroot() + except (ET.ParseError, DefusedXmlException) as e: + print(f"FAILED - Error parsing XML files: {e}") + return False + + new_changes = self._new_tracked_changes(original_root, modified_root) + self._remove_tracked_changes(modified_root, new_changes) + + modified_text = self._extract_text_content(modified_root) + original_text = self._extract_text_content(original_root) + + if modified_text != original_text: + error_message = self._generate_detailed_diff( + original_text, modified_text + ) + print(error_message) + return False + + if self.verbose: + print( + f"PASSED - All {len(new_changes)} change(s) against the original " + "are properly tracked" + ) + return True + + def _tracked_change_elements(self, root): + ins_tag = f"{{{self.namespaces['w']}}}ins" + del_tag = f"{{{self.namespaces['w']}}}del" + return [elem for elem in root.iter() if elem.tag in (ins_tag, del_tag)] + + def _rendered_text(self, elem): + preserve = elem.get("{http://www.w3.org/XML/1998/namespace}space") == "preserve" + return rendered_text(elem.text or "", preserve) + + def _text_elements(self, elem): + w = self.namespaces["w"] + return [ + node + for node in elem.iter() + if node.tag in (f"{{{w}}}t", f"{{{w}}}delText") + ] + + def _tracked_change_key(self, elem): + w = self.namespaces["w"] + text = "".join(self._rendered_text(node) for node in self._text_elements(elem)) + return (elem.tag, elem.get(f"{{{w}}}author"), elem.get(f"{{{w}}}date"), text) + + def _new_tracked_changes(self, original_root, modified_root): + original = self._tracked_change_elements(original_root) + modified = self._tracked_change_elements(modified_root) + + pool = {} + for elem in original: + pool.setdefault(self._tracked_change_key(elem), []).append(elem) + + matched, leftover = set(), [] + for elem in modified: + bucket = pool.get(self._tracked_change_key(elem)) + if bucket: + matched.add(bucket.pop()) + else: + leftover.append(elem) + + def group(elem): + return self._tracked_change_key(elem)[:3] + + def text_of(elems): + return "".join(self._tracked_change_key(e)[3] for e in elems) + + unmatched_original = {} + for elem in original: + if elem not in matched: + unmatched_original.setdefault(group(elem), []).append(elem) + + by_group = {} + for elem in leftover: + by_group.setdefault(group(elem), []).append(elem) + + new = set() + for key, elems in by_group.items(): + rebuilt = text_of(elems) + if rebuilt and rebuilt == text_of(unmatched_original.get(key, [])): + continue + new.update(elems) + return new + + def _generate_detailed_diff(self, original_text, modified_text): + error_parts = [ + "FAILED - Document text doesn't match after removing the tracked changes", + "", + "Likely causes:", + " 1. Modified text inside another author's or tags", + " 2. Made edits without proper tracked changes", + " 3. Didn't nest inside when deleting another's insertion", + " 4. Rewrote another author's / and changed its text on", + " the way. A tracked change from the original is recognised by its", + " author, date and text; anything that doesn't reproduce one exactly", + " reads as new, and the text it carried is reported missing.", + "", + "For pre-redlined documents, use correct patterns:", + " - To reject another's INSERTION: Nest inside their ", + " - To reject PART of one: nest around only the runs you reject.", + " Their may be split around it, so long as the pieces keep", + " their author and date and still spell out the same text.", + " - To restore another's DELETION: Add new AFTER their ", + "", + ] + + git_diff = self._get_git_word_diff(original_text, modified_text) + if git_diff: + error_parts.extend(["Differences:", "============", git_diff]) + else: + error_parts.append("Unable to generate word diff (git not available)") + + return "\n".join(error_parts) + + def _get_git_word_diff(self, original_text, modified_text): + try: + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + original_file = temp_path / "original.txt" + modified_file = temp_path / "modified.txt" + + original_file.write_text(original_text, encoding="utf-8") + modified_file.write_text(modified_text, encoding="utf-8") + + result = subprocess.run( + [ + "git", + "diff", + "--word-diff=plain", + "--word-diff-regex=.", + "-U0", + "--no-index", + str(original_file), + str(modified_file), + ], + capture_output=True, + text=True, encoding="utf-8", errors="replace", + ) + + if result.stdout.strip(): + lines = result.stdout.split("\n") + content_lines = [] + in_content = False + for line in lines: + if line.startswith("@@"): + in_content = True + continue + if in_content and line.strip(): + content_lines.append(line) + + if content_lines: + return "\n".join(content_lines) + + result = subprocess.run( + [ + "git", + "diff", + "--word-diff=plain", + "-U0", + "--no-index", + str(original_file), + str(modified_file), + ], + capture_output=True, + text=True, encoding="utf-8", errors="replace", + ) + + if result.stdout.strip(): + lines = result.stdout.split("\n") + content_lines = [] + in_content = False + for line in lines: + if line.startswith("@@"): + in_content = True + continue + if in_content and line.strip(): + content_lines.append(line) + return "\n".join(content_lines) + + except (subprocess.CalledProcessError, FileNotFoundError, Exception): + pass + + return None + + def _remove_tracked_changes(self, root, targets): + ins_tag = f"{{{self.namespaces['w']}}}ins" + del_tag = f"{{{self.namespaces['w']}}}del" + + for parent in root.iter(): + to_remove = [] + for child in parent: + if child.tag == ins_tag and child in targets: + to_remove.append(child) + for elem in to_remove: + parent.remove(elem) + + deltext_tag = f"{{{self.namespaces['w']}}}delText" + t_tag = f"{{{self.namespaces['w']}}}t" + + for parent in root.iter(): + to_process = [] + for child in parent: + if child.tag == del_tag and child in targets: + to_process.append((child, list(parent).index(child))) + + for del_elem, del_index in reversed(to_process): + for elem in del_elem.iter(): + if elem.tag == deltext_tag: + elem.tag = t_tag + + for child in reversed(list(del_elem)): + parent.insert(del_index, child) + parent.remove(del_elem) + + def _extract_text_content(self, root): + p_tag = f"{{{self.namespaces['w']}}}p" + t_tag = f"{{{self.namespaces['w']}}}t" + + paragraphs = [] + for p_elem in root.findall(f".//{p_tag}"): + text_parts = [] + for t_elem in p_elem.findall(f".//{t_tag}"): + text_parts.append(self._rendered_text(t_elem)) + paragraph_text = "".join(text_parts) + if paragraph_text: + paragraphs.append(paragraph_text) + + return "\n".join(paragraphs) + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/docx/scripts/templates/comments.xml b/docx/scripts/templates/comments.xml new file mode 100644 index 0000000..cd01a7d --- /dev/null +++ b/docx/scripts/templates/comments.xml @@ -0,0 +1,3 @@ + + + diff --git a/docx/scripts/templates/commentsExtended.xml b/docx/scripts/templates/commentsExtended.xml new file mode 100644 index 0000000..411003c --- /dev/null +++ b/docx/scripts/templates/commentsExtended.xml @@ -0,0 +1,3 @@ + + + diff --git a/docx/scripts/templates/commentsExtensible.xml b/docx/scripts/templates/commentsExtensible.xml new file mode 100644 index 0000000..f5572d7 --- /dev/null +++ b/docx/scripts/templates/commentsExtensible.xml @@ -0,0 +1,3 @@ + + + diff --git a/docx/scripts/templates/commentsIds.xml b/docx/scripts/templates/commentsIds.xml new file mode 100644 index 0000000..32f1629 --- /dev/null +++ b/docx/scripts/templates/commentsIds.xml @@ -0,0 +1,3 @@ + + + diff --git a/docx/scripts/templates/people.xml b/docx/scripts/templates/people.xml new file mode 100644 index 0000000..3803d2d --- /dev/null +++ b/docx/scripts/templates/people.xml @@ -0,0 +1,3 @@ + + + diff --git a/google-workspace/SKILL.md b/google-workspace/SKILL.md new file mode 100644 index 0000000..5668d80 --- /dev/null +++ b/google-workspace/SKILL.md @@ -0,0 +1,335 @@ +--- +name: google-workspace +description: "Gmail, Calendar, Drive, Docs, Sheets via gws CLI or Python." +version: 1.1.0 +author: Nous Research +license: MIT +platforms: [linux, macos, windows] +required_credential_files: + - path: google_token.json + description: Google OAuth2 token (created by setup script) + - path: google_client_secret.json + description: Google OAuth2 client credentials (downloaded from Google Cloud Console) +metadata: + hermes: + tags: [Google, Gmail, Calendar, Drive, Sheets, Docs, Contacts, Email, OAuth] + homepage: https://github.com/NousResearch/hermes-agent + related_skills: [himalaya] +--- + +# Google Workspace + +Gmail, Calendar, Drive, Contacts, Sheets, and Docs — through Hermes-managed OAuth and a thin CLI wrapper. When `gws` is installed, the skill uses it as the execution backend for broader Google Workspace coverage; otherwise it falls back to the bundled Python client implementation. + +## References + +- `references/gmail-search-syntax.md` — Gmail search operators (is:unread, from:, newer_than:, etc.) + +## Scripts + +- `scripts/setup.py` — OAuth2 setup (run once to authorize) +- `scripts/google_api.py` — compatibility wrapper CLI. It prefers `gws` for operations when available, while preserving Hermes' existing JSON output contract. + +## First-Time Setup + +The setup is fully non-interactive — you drive it step by step so it works +on CLI, Telegram, Discord, or any platform. + +Define a shorthand first: + +```bash +GSETUP="python ${HERMES_HOME:-$HOME/.hermes}/skills/productivity/google-workspace/scripts/setup.py" +``` + +### Step 0: Check if already set up + +```bash +$GSETUP --check +``` + +If it prints `AUTHENTICATED`, skip to Usage — setup is already done. + +### Step 1: Triage — ask the user what they need + +Before starting OAuth setup, ask the user TWO questions: + +**Question 1: "What Google services do you need? Just email, or also +Calendar/Drive/Sheets/Docs?"** + +- **Email only** → They don't need this skill at all. Use the `himalaya` skill + instead — it works with a Gmail App Password (Settings → Security → App + Passwords) and takes 2 minutes to set up. No Google Cloud project needed. + Load the himalaya skill and follow its setup instructions. + +- **Email + Calendar** → Continue with this skill, but use + `--services email,calendar` during auth so the consent screen only asks for + the scopes they actually need. + +- **Calendar/Drive/Sheets/Docs only** → Continue with this skill and use a + narrower `--services` set like `calendar,drive,sheets,docs`. + +- **Full Workspace access** → Continue with this skill and use the default + `all` service set. + +**Question 2: "Does your Google account use Advanced Protection (hardware +security keys required to sign in)? If you're not sure, you probably don't +— it's something you would have explicitly enrolled in."** + +- **No / Not sure** → Normal setup. Continue below. +- **Yes** → Their Workspace admin must add the OAuth client ID to the org's + allowed apps list before Step 4 will work. Let them know upfront. + +### Step 2: Create OAuth credentials (one-time, ~5 minutes) + +Tell the user: + +> You need a Google Cloud OAuth client. This is a one-time setup: +> +> 1. Create or select a project: +> https://console.cloud.google.com/projectselector2/home/dashboard +> 2. Enable the required APIs from the API Library: +> https://console.cloud.google.com/apis/library +> Enable: Gmail API, Google Calendar API, Google Drive API, +> Google Sheets API, Google Docs API, People API +> 3. Create the OAuth client here: +> https://console.cloud.google.com/apis/credentials +> Credentials → Create Credentials → OAuth 2.0 Client ID +> 4. Application type: "Desktop app" → Create +> 5. If the app is still in Testing, add the user's Google account as a test user here: +> https://console.cloud.google.com/auth/audience +> Audience → Test users → Add users +> 6. Download the JSON file and tell me the file path +> +> Important Hermes CLI note: if the file path starts with `/`, do NOT send only the bare path as its own message in the CLI, because it can be mistaken for a slash command. Send it in a sentence instead, like: +> `The JSON file path is: /home/user/Downloads/client_secret_....json` + +Once they provide the path: + +```bash +$GSETUP --client-secret /path/to/client_secret.json +``` + +If they paste the raw client ID / client secret values instead of a file path, +write a valid Desktop OAuth JSON file for them yourself, save it somewhere +explicit (for example `~/Downloads/hermes-google-client-secret.json`), then run +`--client-secret` against that file. + +### Step 3: Get authorization URL + +Use the service set chosen in Step 1. Examples: + +```bash +$GSETUP --auth-url --services email,calendar --format json +$GSETUP --auth-url --services calendar,drive,sheets,docs --format json +$GSETUP --auth-url --services all --format json +``` + +This returns JSON with an `auth_url` field and also saves the exact URL to +`~/.hermes/google_oauth_last_url.txt`. + +Agent rules for this step: +- Extract the `auth_url` field and send that exact URL to the user as a single line. +- Tell the user that the browser will likely fail on `http://localhost:1` after approval, and that this is expected. +- Tell them to copy the ENTIRE redirected URL from the browser address bar. +- If the user gets `Error 403: access_denied`, send them directly to `https://console.cloud.google.com/auth/audience` to add themselves as a test user. + +### Step 4: Exchange the code + +The user will paste back either a URL like `http://localhost:1/?code=4/0A...&scope=...` +or just the code string. Either works. The `--auth-url` step stores a temporary +pending OAuth session locally so `--auth-code` can complete the PKCE exchange +later, even on headless systems: + +```bash +$GSETUP --auth-code "THE_URL_OR_CODE_THE_USER_PASTED" --format json +``` + +If `--auth-code` fails because the code expired, was already used, or came from +an older browser tab, it now returns a fresh `fresh_auth_url`. In that case, +immediately send the new URL to the user and have them retry with the newest +browser redirect only. + +### Step 5: Verify + +```bash +$GSETUP --check +``` + +Should print `AUTHENTICATED`. Setup is complete — token refreshes automatically from now on. + +### Notes + +- Token is stored at `~/.hermes/google_token.json` and auto-refreshes. +- Pending OAuth session state/verifier are stored temporarily at `~/.hermes/google_oauth_pending.json` until exchange completes. +- If `gws` is installed, `google_api.py` points it at the same `~/.hermes/google_token.json` credentials file. Users do not need to run a separate `gws auth login` flow. +- To revoke: `$GSETUP --revoke` + +## Usage + +All commands go through the API script. Set `GAPI` as a shorthand: + +```bash +GAPI="python ${HERMES_HOME:-$HOME/.hermes}/skills/productivity/google-workspace/scripts/google_api.py" +``` + +### Gmail + +```bash +# Search (returns JSON array with id, from, subject, date, snippet) +$GAPI gmail search "is:unread" --max 10 +$GAPI gmail search "from:boss@company.com newer_than:1d" +$GAPI gmail search "has:attachment filename:pdf newer_than:7d" + +# Read full message (returns JSON with body text) +$GAPI gmail get MESSAGE_ID + +# Send +$GAPI gmail send --to user@example.com --subject "Hello" --body "Message text" +$GAPI gmail send --to user@example.com --subject "Report" --body "

Q4

Details...

" --html +$GAPI gmail send --to user@example.com --subject "Hello" --from '"Research Agent" ' --body "Message text" + +# Reply (automatically threads and sets In-Reply-To) +$GAPI gmail reply MESSAGE_ID --body "Thanks, that works for me." +$GAPI gmail reply MESSAGE_ID --from '"Support Bot" ' --body "Thanks" + +# Labels +$GAPI gmail labels +$GAPI gmail modify MESSAGE_ID --add-labels LABEL_ID +$GAPI gmail modify MESSAGE_ID --remove-labels UNREAD +``` + +### Calendar + +```bash +# List events (defaults to next 7 days) +$GAPI calendar list +$GAPI calendar list --start 2026-03-01T00:00:00Z --end 2026-03-07T23:59:59Z + +# Create event (ISO 8601 with timezone required) +$GAPI calendar create --summary "Team Standup" --start 2026-03-01T10:00:00-06:00 --end 2026-03-01T10:30:00-06:00 +$GAPI calendar create --summary "Lunch" --start 2026-03-01T12:00:00Z --end 2026-03-01T13:00:00Z --location "Cafe" +$GAPI calendar create --summary "Review" --start 2026-03-01T14:00:00Z --end 2026-03-01T15:00:00Z --attendees "alice@co.com,bob@co.com" + +# Delete event +$GAPI calendar delete EVENT_ID +``` + +### Drive + +```bash +# Search existing files +$GAPI drive search "quarterly report" --max 10 +$GAPI drive search "mimeType='application/pdf'" --raw-query --max 5 + +# Get metadata for a single file +$GAPI drive get FILE_ID + +# Upload a local file (auto-detects MIME type) +$GAPI drive upload /path/to/report.pdf +$GAPI drive upload /path/to/image.png --name "Logo.png" --parent FOLDER_ID + +# Download (binary files download as-is; Google-native files export to a +# sensible default — Docs→pdf, Sheets→csv, Slides→pdf, Drawings→png) +$GAPI drive download FILE_ID +$GAPI drive download DOC_ID --output ~/doc.pdf +$GAPI drive download DOC_ID --export-mime text/plain --output ~/doc.txt + +# Create a folder +$GAPI drive create-folder "Reports" +$GAPI drive create-folder "Q4" --parent FOLDER_ID + +# Share +$GAPI drive share FILE_ID --email alice@example.com --role reader +$GAPI drive share FILE_ID --email alice@example.com --role writer --notify +$GAPI drive share FILE_ID --type anyone --role reader # anyone with link +$GAPI drive share FILE_ID --type domain --domain example.com --role reader + +# Delete — defaults to trash (reversible). Use --permanent to skip the trash. +$GAPI drive delete FILE_ID +$GAPI drive delete FILE_ID --permanent +``` + +### Contacts + +```bash +$GAPI contacts list --max 20 +``` + +### Sheets + +```bash +# Create a new spreadsheet +$GAPI sheets create --title "Q4 Budget" +$GAPI sheets create --title "Inventory" --sheet-name "Stock" + +# Read +$GAPI sheets get SHEET_ID "Sheet1!A1:D10" + +# Write +$GAPI sheets update SHEET_ID "Sheet1!A1:B2" --values '[["Name","Score"],["Alice","95"]]' + +# Append rows +$GAPI sheets append SHEET_ID "Sheet1!A:C" --values '[["new","row","data"]]' +``` + +### Docs + +```bash +# Read +$GAPI docs get DOC_ID + +# Create a new Doc (optionally seeded with body text) +$GAPI docs create --title "Meeting Notes" +$GAPI docs create --title "Draft" --body "First paragraph..." + +# Append text to the end of an existing Doc +$GAPI docs append DOC_ID --text "Additional content to append" +``` + +## Output Format + +All commands return JSON. Parse with `jq` or read directly. Key fields: + +- **Gmail search**: `[{id, threadId, from, to, subject, date, snippet, labels}]` +- **Gmail get**: `{id, threadId, from, to, subject, date, labels, body}` +- **Gmail send/reply**: `{status: "sent", id, threadId}` +- **Calendar list**: `[{id, summary, start, end, location, description, htmlLink}]` +- **Calendar create**: `{status: "created", id, summary, htmlLink}` +- **Drive search**: `[{id, name, mimeType, modifiedTime, webViewLink}]` +- **Drive get**: `{id, name, mimeType, modifiedTime, size, webViewLink, parents, owners}` +- **Drive upload**: `{status: "uploaded", id, name, mimeType, webViewLink}` +- **Drive download**: `{status: "downloaded", id, name, path, mimeType}` +- **Drive create-folder**: `{status: "created", id, name, webViewLink}` +- **Drive share**: `{status: "shared", permissionId, fileId, role, type}` +- **Drive delete**: `{status: "trashed" | "deleted", fileId, permanent}` +- **Contacts list**: `[{name, emails: [...], phones: [...]}]` +- **Sheets get**: `[[cell, cell, ...], ...]` +- **Sheets create**: `{status: "created", spreadsheetId, title, spreadsheetUrl}` +- **Docs create**: `{status: "created", documentId, title, url}` +- **Docs append**: `{status: "appended", documentId, inserted_at, characters}` + +## Rules + +1. **Never send email, create/delete calendar events, delete Drive files, share files, or modify Docs/Sheets without confirming with the user first.** Show what will be done (recipients, file IDs, content, share role) and ask for approval. For `drive delete`, prefer the default trash (reversible) over `--permanent`. +2. **Check auth before first use** — run `setup.py --check`. If it fails, guide the user through setup. +3. **Use the Gmail search syntax reference** for complex queries — load it with `skill_view("google-workspace", file_path="references/gmail-search-syntax.md")`. +4. **Calendar times must include timezone** — always use ISO 8601 with offset (e.g., `2026-03-01T10:00:00-06:00`) or UTC (`Z`). +5. **Respect rate limits** — avoid rapid-fire sequential API calls. Batch reads when possible. + +## Troubleshooting + +| Problem | Fix | +|---------|-----| +| `NOT_AUTHENTICATED` | Run setup Steps 2-5 above | +| `REFRESH_FAILED` | Token revoked or expired — redo Steps 3-5 | +| `HttpError 403: Insufficient Permission` | Missing API scope — `$GSETUP --revoke` then redo Steps 3-5 | +| `AUTHENTICATED (partial)` or "Token missing scopes" | New write capabilities (Drive write/delete, Docs create/edit) require re-authorization. `$GSETUP --revoke` then redo Steps 3-5 to grant the upgraded scopes. | +| `HttpError 403: Access Not Configured` | API not enabled — user needs to enable it in Google Cloud Console | +| `ModuleNotFoundError` | Run `$GSETUP --install-deps` | +| Advanced Protection blocks auth | Workspace admin must allowlist the OAuth client ID | + +## Revoking Access + +```bash +$GSETUP --revoke +``` diff --git a/google-workspace/references/gmail-search-syntax.md b/google-workspace/references/gmail-search-syntax.md new file mode 100644 index 0000000..f662346 --- /dev/null +++ b/google-workspace/references/gmail-search-syntax.md @@ -0,0 +1,63 @@ +# Gmail Search Syntax + +Standard Gmail search operators work in the `query` argument. + +## Common Operators + +| Operator | Example | Description | +|----------|---------|-------------| +| `is:unread` | `is:unread` | Unread messages | +| `is:starred` | `is:starred` | Starred messages | +| `is:important` | `is:important` | Important messages | +| `in:inbox` | `in:inbox` | Inbox only | +| `in:sent` | `in:sent` | Sent folder | +| `in:drafts` | `in:drafts` | Drafts | +| `in:trash` | `in:trash` | Trash | +| `in:anywhere` | `in:anywhere` | All mail including spam/trash | +| `from:` | `from:alice@example.com` | Sender | +| `to:` | `to:bob@example.com` | Recipient | +| `cc:` | `cc:team@example.com` | CC recipient | +| `subject:` | `subject:invoice` | Subject contains | +| `label:` | `label:work` | Has label | +| `has:attachment` | `has:attachment` | Has attachments | +| `filename:` | `filename:pdf` | Attachment filename/type | +| `larger:` | `larger:5M` | Larger than size | +| `smaller:` | `smaller:1M` | Smaller than size | + +## Date Operators + +| Operator | Example | Description | +|----------|---------|-------------| +| `newer_than:` | `newer_than:7d` | Within last N days (d), months (m), years (y) | +| `older_than:` | `older_than:30d` | Older than N days/months/years | +| `after:` | `after:2026/02/01` | After date (YYYY/MM/DD) | +| `before:` | `before:2026/03/01` | Before date | + +## Combining + +| Syntax | Example | Description | +|--------|---------|-------------| +| space | `from:alice subject:meeting` | AND (implicit) | +| `OR` | `from:alice OR from:bob` | OR | +| `-` | `-from:noreply@` | NOT (exclude) | +| `()` | `(from:alice OR from:bob) subject:meeting` | Grouping | +| `""` | `"exact phrase"` | Exact phrase match | + +## Common Patterns + +``` +# Unread emails from the last day +is:unread newer_than:1d + +# Emails with PDF attachments from a specific sender +from:accounting@company.com has:attachment filename:pdf + +# Important unread emails (not promotions/social) +is:unread -category:promotions -category:social + +# Emails in a thread about a topic +subject:"Q4 budget" newer_than:30d + +# Large attachments to clean up +has:attachment larger:10M older_than:90d +``` diff --git a/google-workspace/scripts/_hermes_home.py b/google-workspace/scripts/_hermes_home.py new file mode 100644 index 0000000..456eaa9 --- /dev/null +++ b/google-workspace/scripts/_hermes_home.py @@ -0,0 +1,42 @@ +"""Resolve HERMES_HOME for standalone skill scripts. + +Skill scripts may run outside the Hermes process (e.g. system Python, +nix env, CI) where ``hermes_constants`` is not importable. This module +provides the same ``get_hermes_home()`` and ``display_hermes_home()`` +contracts as ``hermes_constants`` without requiring it on ``sys.path``. + +When ``hermes_constants`` IS available it is used directly so that any +future enhancements (profile resolution, Docker detection, etc.) are +picked up automatically. The fallback path replicates the core logic +from ``hermes_constants.py`` using only the stdlib. + +All scripts under ``google-workspace/scripts/`` should import from here +instead of duplicating the ``HERMES_HOME = Path(os.getenv(...))`` pattern. +""" + +from __future__ import annotations + +import os +from pathlib import Path + +try: + from hermes_constants import display_hermes_home as display_hermes_home + from hermes_constants import get_hermes_home as get_hermes_home +except (ModuleNotFoundError, ImportError): + + def get_hermes_home() -> Path: + """Return the Hermes home directory (default: ~/.hermes). + + Mirrors ``hermes_constants.get_hermes_home()``.""" + val = os.environ.get("HERMES_HOME", "").strip() + return Path(val) if val else Path.home() / ".hermes" + + def display_hermes_home() -> str: + """Return a user-friendly ``~/``-shortened display string. + + Mirrors ``hermes_constants.display_hermes_home()``.""" + home = get_hermes_home() + try: + return "~/" + str(home.relative_to(Path.home())) + except ValueError: + return str(home) diff --git a/google-workspace/scripts/google_api.py b/google-workspace/scripts/google_api.py new file mode 100644 index 0000000..5c2e823 --- /dev/null +++ b/google-workspace/scripts/google_api.py @@ -0,0 +1,1225 @@ +#!/usr/bin/env python3 +"""Google Workspace API CLI for Hermes Agent. + +Uses the Google Workspace CLI (`gws`) when available, but preserves the +existing Hermes-facing JSON contract and falls back to the Python client +libraries if `gws` is not installed. + +Usage: + python google_api.py gmail search "is:unread" [--max 10] + python google_api.py gmail get MESSAGE_ID + python google_api.py gmail send --to user@example.com --subject "Hi" --body "Hello" + python google_api.py gmail reply MESSAGE_ID --body "Thanks" + python google_api.py calendar list [--from DATE] [--to DATE] [--calendar primary] + python google_api.py calendar create --summary "Meeting" --start DATETIME --end DATETIME + python google_api.py drive search "budget report" [--max 10] + python google_api.py contacts list [--max 20] + python google_api.py sheets get SHEET_ID RANGE + python google_api.py sheets update SHEET_ID RANGE --values '[[...]]' + python google_api.py sheets append SHEET_ID RANGE --values '[[...]]' + python google_api.py docs get DOC_ID +""" + +import argparse +import base64 +import json +import os +import shutil +import subprocess +import sys +from datetime import datetime, timedelta, timezone +from email.mime.text import MIMEText +from pathlib import Path + +# Ensure sibling modules (_hermes_home) are importable when run standalone. +_SCRIPTS_DIR = str(Path(__file__).resolve().parent) +if _SCRIPTS_DIR not in sys.path: + sys.path.insert(0, _SCRIPTS_DIR) + +from _hermes_home import get_hermes_home + +HERMES_HOME = get_hermes_home() +TOKEN_PATH = HERMES_HOME / "google_token.json" +CLIENT_SECRET_PATH = HERMES_HOME / "google_client_secret.json" + +SCOPES = [ + "https://www.googleapis.com/auth/gmail.readonly", + "https://www.googleapis.com/auth/gmail.send", + "https://www.googleapis.com/auth/gmail.modify", + "https://www.googleapis.com/auth/calendar", + "https://www.googleapis.com/auth/drive", + "https://www.googleapis.com/auth/contacts.readonly", + "https://www.googleapis.com/auth/spreadsheets", + "https://www.googleapis.com/auth/documents", +] + + +def _normalize_authorized_user_payload(payload: dict) -> dict: + normalized = dict(payload) + if not normalized.get("type"): + normalized["type"] = "authorized_user" + return normalized + + +def _ensure_authenticated(): + if not TOKEN_PATH.exists(): + print("Not authenticated. Run the setup script first:", file=sys.stderr) + print(f" python {Path(__file__).parent / 'setup.py'}", file=sys.stderr) + sys.exit(1) + + +def _stored_token_scopes() -> list[str]: + try: + data = json.loads(TOKEN_PATH.read_text(encoding="utf-8")) + except Exception: + return list(SCOPES) + scopes = data.get("scopes") + if isinstance(scopes, list) and scopes: + return scopes + return list(SCOPES) + + +def _gws_binary() -> str | None: + override = os.getenv("HERMES_GWS_BIN") + if override: + return override + return shutil.which("gws") + + +def _gws_env() -> dict[str, str]: + env = os.environ.copy() + env["GOOGLE_WORKSPACE_CLI_CREDENTIALS_FILE"] = str(TOKEN_PATH) + return env + + +def _run_gws(parts: list[str], *, params: dict | None = None, body: dict | None = None): + binary = _gws_binary() + if not binary: + raise RuntimeError("gws not installed") + + _ensure_authenticated() + + cmd = [binary, *parts] + if params is not None: + cmd.extend(["--params", json.dumps(params)]) + if body is not None: + cmd.extend(["--json", json.dumps(body)]) + + result = subprocess.run( + cmd, + capture_output=True, + text=True, encoding='utf-8', errors='replace', + env=_gws_env(), + ) + if result.returncode != 0: + err = result.stderr.strip() or result.stdout.strip() or "Unknown gws error" + print(err, file=sys.stderr) + sys.exit(result.returncode or 1) + + stdout = result.stdout.strip() + if not stdout: + return {} + + try: + return json.loads(stdout) + except json.JSONDecodeError: + print("ERROR: Unexpected non-JSON output from gws:", file=sys.stderr) + print(stdout, file=sys.stderr) + sys.exit(1) + + +def _headers_dict(msg: dict) -> dict[str, str]: + return { + h["name"].lower(): h["value"] + for h in msg.get("payload", {}).get("headers", []) + if h.get("name") + } + + +def _extract_message_body(msg: dict) -> str: + body = "" + payload = msg.get("payload", {}) + if payload.get("body", {}).get("data"): + body = base64.urlsafe_b64decode(payload["body"]["data"]).decode("utf-8", errors="replace") + elif payload.get("parts"): + for part in payload["parts"]: + if part.get("mimeType") == "text/plain" and part.get("body", {}).get("data"): + body = base64.urlsafe_b64decode(part["body"]["data"]).decode("utf-8", errors="replace") + break + if not body: + for part in payload["parts"]: + if part.get("mimeType") == "text/html" and part.get("body", {}).get("data"): + body = base64.urlsafe_b64decode(part["body"]["data"]).decode("utf-8", errors="replace") + break + return body + + +def _extract_doc_text(doc: dict) -> str: + text_parts = [] + for element in doc.get("body", {}).get("content", []): + paragraph = element.get("paragraph", {}) + for pe in paragraph.get("elements", []): + text_run = pe.get("textRun", {}) + if text_run.get("content"): + text_parts.append(text_run["content"]) + return "".join(text_parts) + + +def _datetime_with_timezone(value: str) -> str: + if not value: + return value + if "T" not in value: + return value + if value.endswith("Z"): + return value + tail = value[10:] + if "+" in tail or "-" in tail: + return value + return value + "Z" + + +def get_credentials(): + """Load and refresh credentials from token file.""" + _ensure_authenticated() + + from google.oauth2.credentials import Credentials + from google.auth.transport.requests import Request + + creds = Credentials.from_authorized_user_file(str(TOKEN_PATH), _stored_token_scopes()) + if creds.expired and creds.refresh_token: + creds.refresh(Request()) + TOKEN_PATH.write_text( + json.dumps( + _normalize_authorized_user_payload(json.loads(creds.to_json())), + indent=2, + ), encoding="utf-8" + ) + if not creds.valid: + print("Token is invalid. Re-run setup.", file=sys.stderr) + sys.exit(1) + return creds + + +def build_service(api, version): + from googleapiclient.discovery import build + + return build(api, version, credentials=get_credentials()) + + +# ========================================================================= +# Gmail +# ========================================================================= + + +def gmail_search(args): + if _gws_binary(): + results = _run_gws( + ["gmail", "users", "messages", "list"], + params={"userId": "me", "q": args.query, "maxResults": args.max}, + ) + messages = results.get("messages", []) + output = [] + for msg_meta in messages: + msg = _run_gws( + ["gmail", "users", "messages", "get"], + params={ + "userId": "me", + "id": msg_meta["id"], + "format": "metadata", + "metadataHeaders": ["From", "To", "Subject", "Date"], + }, + ) + headers = _headers_dict(msg) + output.append( + { + "id": msg["id"], + "threadId": msg["threadId"], + "from": headers.get("from", ""), + "to": headers.get("to", ""), + "subject": headers.get("subject", ""), + "date": headers.get("date", ""), + "snippet": msg.get("snippet", ""), + "labels": msg.get("labelIds", []), + } + ) + print(json.dumps(output, indent=2, ensure_ascii=False)) + return + + service = build_service("gmail", "v1") + results = service.users().messages().list( + userId="me", q=args.query, maxResults=args.max + ).execute() + messages = results.get("messages", []) + if not messages: + print("No messages found.") + return + + output = [] + for msg_meta in messages: + msg = service.users().messages().get( + userId="me", id=msg_meta["id"], format="metadata", + metadataHeaders=["From", "To", "Subject", "Date"], + ).execute() + headers = _headers_dict(msg) + output.append({ + "id": msg["id"], + "threadId": msg["threadId"], + "from": headers.get("from", ""), + "to": headers.get("to", ""), + "subject": headers.get("subject", ""), + "date": headers.get("date", ""), + "snippet": msg.get("snippet", ""), + "labels": msg.get("labelIds", []), + }) + print(json.dumps(output, indent=2, ensure_ascii=False)) + + + +def gmail_get(args): + if _gws_binary(): + msg = _run_gws( + ["gmail", "users", "messages", "get"], + params={"userId": "me", "id": args.message_id, "format": "full"}, + ) + headers = _headers_dict(msg) + result = { + "id": msg["id"], + "threadId": msg["threadId"], + "from": headers.get("from", ""), + "to": headers.get("to", ""), + "subject": headers.get("subject", ""), + "date": headers.get("date", ""), + "labels": msg.get("labelIds", []), + "body": _extract_message_body(msg), + } + print(json.dumps(result, indent=2, ensure_ascii=False)) + return + + service = build_service("gmail", "v1") + msg = service.users().messages().get( + userId="me", id=args.message_id, format="full" + ).execute() + + headers = _headers_dict(msg) + result = { + "id": msg["id"], + "threadId": msg["threadId"], + "from": headers.get("from", ""), + "to": headers.get("to", ""), + "subject": headers.get("subject", ""), + "date": headers.get("date", ""), + "labels": msg.get("labelIds", []), + "body": _extract_message_body(msg), + } + print(json.dumps(result, indent=2, ensure_ascii=False)) + + + +def gmail_send(args): + if _gws_binary(): + message = MIMEText(args.body, "html" if args.html else "plain") + message["To"] = args.to + message["Subject"] = args.subject + if args.cc: + message["Cc"] = args.cc + if args.from_header: + message["From"] = args.from_header + + raw = base64.urlsafe_b64encode(message.as_bytes()).decode() + body = {"raw": raw} + if args.thread_id: + body["threadId"] = args.thread_id + + result = _run_gws( + ["gmail", "users", "messages", "send"], + params={"userId": "me"}, + body=body, + ) + print(json.dumps({"status": "sent", "id": result["id"], "threadId": result.get("threadId", "")}, indent=2)) + return + + service = build_service("gmail", "v1") + message = MIMEText(args.body, "html" if args.html else "plain") + message["To"] = args.to + message["Subject"] = args.subject + if args.cc: + message["Cc"] = args.cc + if args.from_header: + message["From"] = args.from_header + + raw = base64.urlsafe_b64encode(message.as_bytes()).decode() + body = {"raw": raw} + + if args.thread_id: + body["threadId"] = args.thread_id + + result = service.users().messages().send(userId="me", body=body).execute() + print(json.dumps({"status": "sent", "id": result["id"], "threadId": result.get("threadId", "")}, indent=2)) + + + +def gmail_reply(args): + if _gws_binary(): + original = _run_gws( + ["gmail", "users", "messages", "get"], + params={ + "userId": "me", + "id": args.message_id, + "format": "metadata", + "metadataHeaders": ["From", "Subject", "Message-ID"], + }, + ) + headers = _headers_dict(original) + + subject = headers.get("subject", "") + if not subject.startswith("Re:"): + subject = f"Re: {subject}" + + message = MIMEText(args.body) + message["To"] = headers.get("from", "") + message["Subject"] = subject + if args.from_header: + message["From"] = args.from_header + if headers.get("message-id"): + message["In-Reply-To"] = headers["message-id"] + message["References"] = headers["message-id"] + + raw = base64.urlsafe_b64encode(message.as_bytes()).decode() + result = _run_gws( + ["gmail", "users", "messages", "send"], + params={"userId": "me"}, + body={"raw": raw, "threadId": original["threadId"]}, + ) + print(json.dumps({"status": "sent", "id": result["id"], "threadId": result.get("threadId", "")}, indent=2)) + return + + service = build_service("gmail", "v1") + original = service.users().messages().get( + userId="me", id=args.message_id, format="metadata", + metadataHeaders=["From", "Subject", "Message-ID"], + ).execute() + headers = _headers_dict(original) + + subject = headers.get("subject", "") + if not subject.startswith("Re:"): + subject = f"Re: {subject}" + + message = MIMEText(args.body) + message["To"] = headers.get("from", "") + message["Subject"] = subject + if args.from_header: + message["From"] = args.from_header + if headers.get("message-id"): + message["In-Reply-To"] = headers["message-id"] + message["References"] = headers["message-id"] + + raw = base64.urlsafe_b64encode(message.as_bytes()).decode() + body = {"raw": raw, "threadId": original["threadId"]} + + result = service.users().messages().send(userId="me", body=body).execute() + print(json.dumps({"status": "sent", "id": result["id"], "threadId": result.get("threadId", "")}, indent=2)) + + + +def gmail_labels(args): + if _gws_binary(): + results = _run_gws(["gmail", "users", "labels", "list"], params={"userId": "me"}) + labels = [{"id": l["id"], "name": l["name"], "type": l.get("type", "")} for l in results.get("labels", [])] + print(json.dumps(labels, indent=2)) + return + + service = build_service("gmail", "v1") + results = service.users().labels().list(userId="me").execute() + labels = [{"id": l["id"], "name": l["name"], "type": l.get("type", "")} for l in results.get("labels", [])] + print(json.dumps(labels, indent=2)) + + + +def gmail_modify(args): + body = {} + if args.add_labels: + body["addLabelIds"] = args.add_labels.split(",") + if args.remove_labels: + body["removeLabelIds"] = args.remove_labels.split(",") + + if _gws_binary(): + result = _run_gws( + ["gmail", "users", "messages", "modify"], + params={"userId": "me", "id": args.message_id}, + body=body, + ) + print(json.dumps({"id": result["id"], "labels": result.get("labelIds", [])}, indent=2)) + return + + service = build_service("gmail", "v1") + result = service.users().messages().modify(userId="me", id=args.message_id, body=body).execute() + print(json.dumps({"id": result["id"], "labels": result.get("labelIds", [])}, indent=2)) + + +# ========================================================================= +# Calendar +# ========================================================================= + + +def calendar_list(args): + now = datetime.now(timezone.utc) + time_min = _datetime_with_timezone(args.start or now.isoformat()) + time_max = _datetime_with_timezone(args.end or (now + timedelta(days=7)).isoformat()) + + if _gws_binary(): + results = _run_gws( + ["calendar", "events", "list"], + params={ + "calendarId": args.calendar, + "timeMin": time_min, + "timeMax": time_max, + "maxResults": args.max, + "singleEvents": True, + "orderBy": "startTime", + }, + ) + events = [] + for e in results.get("items", []): + events.append({ + "id": e["id"], + "summary": e.get("summary", "(no title)"), + "start": e.get("start", {}).get("dateTime", e.get("start", {}).get("date", "")), + "end": e.get("end", {}).get("dateTime", e.get("end", {}).get("date", "")), + "location": e.get("location", ""), + "description": e.get("description", ""), + "status": e.get("status", ""), + "htmlLink": e.get("htmlLink", ""), + }) + print(json.dumps(events, indent=2, ensure_ascii=False)) + return + + service = build_service("calendar", "v3") + results = service.events().list( + calendarId=args.calendar, timeMin=time_min, timeMax=time_max, + maxResults=args.max, singleEvents=True, orderBy="startTime", + ).execute() + + events = [] + for e in results.get("items", []): + events.append({ + "id": e["id"], + "summary": e.get("summary", "(no title)"), + "start": e.get("start", {}).get("dateTime", e.get("start", {}).get("date", "")), + "end": e.get("end", {}).get("dateTime", e.get("end", {}).get("date", "")), + "location": e.get("location", ""), + "description": e.get("description", ""), + "status": e.get("status", ""), + "htmlLink": e.get("htmlLink", ""), + }) + print(json.dumps(events, indent=2, ensure_ascii=False)) + + + +def calendar_create(args): + event = { + "summary": args.summary, + "start": {"dateTime": args.start}, + "end": {"dateTime": args.end}, + } + if args.location: + event["location"] = args.location + if args.description: + event["description"] = args.description + if args.attendees: + event["attendees"] = [{"email": e.strip()} for e in args.attendees.split(",") if e.strip()] + + if _gws_binary(): + result = _run_gws( + ["calendar", "events", "insert"], + params={"calendarId": args.calendar}, + body=event, + ) + print(json.dumps({ + "status": "created", + "id": result["id"], + "summary": result.get("summary", ""), + "htmlLink": result.get("htmlLink", ""), + }, indent=2)) + return + + service = build_service("calendar", "v3") + result = service.events().insert(calendarId=args.calendar, body=event).execute() + print(json.dumps({ + "status": "created", + "id": result["id"], + "summary": result.get("summary", ""), + "htmlLink": result.get("htmlLink", ""), + }, indent=2)) + + + +def calendar_delete(args): + if _gws_binary(): + _run_gws(["calendar", "events", "delete"], params={"calendarId": args.calendar, "eventId": args.event_id}) + print(json.dumps({"status": "deleted", "eventId": args.event_id})) + return + + service = build_service("calendar", "v3") + service.events().delete(calendarId=args.calendar, eventId=args.event_id).execute() + print(json.dumps({"status": "deleted", "eventId": args.event_id})) + + +# ========================================================================= +# Drive +# ========================================================================= + + +def drive_search(args): + query = args.query if args.raw_query else f"fullText contains '{args.query}'" + if _gws_binary(): + results = _run_gws( + ["drive", "files", "list"], + params={ + "q": query, + "pageSize": args.max, + "fields": "files(id, name, mimeType, modifiedTime, webViewLink)", + }, + ) + print(json.dumps(results.get("files", []), indent=2, ensure_ascii=False)) + return + + service = build_service("drive", "v3") + results = service.files().list( + q=query, pageSize=args.max, fields="files(id, name, mimeType, modifiedTime, webViewLink)", + ).execute() + files = results.get("files", []) + print(json.dumps(files, indent=2, ensure_ascii=False)) + + +def drive_get(args): + """Get metadata for a single Drive file by ID.""" + fields = "id, name, mimeType, modifiedTime, size, webViewLink, parents, owners(emailAddress)" + if _gws_binary(): + result = _run_gws( + ["drive", "files", "get"], + params={"fileId": args.file_id, "fields": fields}, + ) + print(json.dumps(result, indent=2, ensure_ascii=False)) + return + + service = build_service("drive", "v3") + result = service.files().get(fileId=args.file_id, fields=fields).execute() + print(json.dumps(result, indent=2, ensure_ascii=False)) + + +def drive_upload(args): + """Upload a local file to Drive. Falls through to Python client even when gws + is installed, because gws doesn't do multipart uploads.""" + import mimetypes + from googleapiclient.http import MediaFileUpload + + local_path = Path(args.path).expanduser() + if not local_path.exists(): + print(f"ERROR: file not found: {local_path}", file=sys.stderr) + sys.exit(1) + + mime = args.mime_type or mimetypes.guess_type(str(local_path))[0] or "application/octet-stream" + metadata = {"name": args.name or local_path.name} + if args.parent: + metadata["parents"] = [args.parent] + + service = build_service("drive", "v3") + media = MediaFileUpload(str(local_path), mimetype=mime, resumable=True) + result = service.files().create( + body=metadata, + media_body=media, + fields="id, name, mimeType, webViewLink", + ).execute() + print(json.dumps({ + "status": "uploaded", + "id": result["id"], + "name": result.get("name", ""), + "mimeType": result.get("mimeType", ""), + "webViewLink": result.get("webViewLink", ""), + }, indent=2, ensure_ascii=False)) + + +def drive_download(args): + """Download a Drive file to a local path. Google-native files (Docs/Sheets/Slides) + must be exported; binary files are downloaded as-is.""" + import io + from googleapiclient.http import MediaIoBaseDownload + + service = build_service("drive", "v3") + + # Look up the file to decide download vs export. + meta = service.files().get(fileId=args.file_id, fields="id, name, mimeType").execute() + mime = meta.get("mimeType", "") + name = meta.get("name", args.file_id) + + # Map Google-native MIME types to a sensible export default. + native_export_map = { + "application/vnd.google-apps.document": ("application/pdf", ".pdf"), + "application/vnd.google-apps.spreadsheet": ("text/csv", ".csv"), + "application/vnd.google-apps.presentation": ("application/pdf", ".pdf"), + "application/vnd.google-apps.drawing": ("image/png", ".png"), + } + + out_path = Path(args.output).expanduser() if args.output else Path.cwd() / name + + if mime in native_export_map: + export_mime = args.export_mime or native_export_map[mime][0] + default_ext = native_export_map[mime][1] + if not args.output and not out_path.suffix: + out_path = out_path.with_suffix(default_ext) + request = service.files().export_media(fileId=args.file_id, mimeType=export_mime) + else: + request = service.files().get_media(fileId=args.file_id) + + out_path.parent.mkdir(parents=True, exist_ok=True) + fh = io.FileIO(str(out_path), "wb") + downloader = MediaIoBaseDownload(fh, request) + done = False + while not done: + _, done = downloader.next_chunk() + fh.close() + + print(json.dumps({ + "status": "downloaded", + "id": args.file_id, + "name": name, + "path": str(out_path), + "mimeType": mime, + }, indent=2, ensure_ascii=False)) + + +def drive_create_folder(args): + body = { + "name": args.name, + "mimeType": "application/vnd.google-apps.folder", + } + if args.parent: + body["parents"] = [args.parent] + + if _gws_binary(): + result = _run_gws( + ["drive", "files", "create"], + params={"fields": "id, name, webViewLink"}, + body=body, + ) + print(json.dumps({ + "status": "created", + "id": result["id"], + "name": result.get("name", ""), + "webViewLink": result.get("webViewLink", ""), + }, indent=2, ensure_ascii=False)) + return + + service = build_service("drive", "v3") + result = service.files().create(body=body, fields="id, name, webViewLink").execute() + print(json.dumps({ + "status": "created", + "id": result["id"], + "name": result.get("name", ""), + "webViewLink": result.get("webViewLink", ""), + }, indent=2, ensure_ascii=False)) + + +def drive_share(args): + permission = { + "type": args.type, + "role": args.role, + } + if args.type in {"user", "group"}: + if not args.email: + print("ERROR: --email is required for type=user or type=group", file=sys.stderr) + sys.exit(1) + permission["emailAddress"] = args.email + elif args.type == "domain": + if not args.domain: + print("ERROR: --domain is required for type=domain", file=sys.stderr) + sys.exit(1) + permission["domain"] = args.domain + + if _gws_binary(): + result = _run_gws( + ["drive", "permissions", "create"], + params={ + "fileId": args.file_id, + "sendNotificationEmail": args.notify, + }, + body=permission, + ) + print(json.dumps({ + "status": "shared", + "permissionId": result.get("id", ""), + "fileId": args.file_id, + "role": permission["role"], + "type": permission["type"], + }, indent=2, ensure_ascii=False)) + return + + service = build_service("drive", "v3") + result = service.permissions().create( + fileId=args.file_id, + body=permission, + sendNotificationEmail=args.notify, + fields="id", + ).execute() + print(json.dumps({ + "status": "shared", + "permissionId": result.get("id", ""), + "fileId": args.file_id, + "role": permission["role"], + "type": permission["type"], + }, indent=2, ensure_ascii=False)) + + +def drive_delete(args): + """Trash or permanently delete a Drive file. Defaults to trash (reversible).""" + if args.permanent: + if _gws_binary(): + _run_gws(["drive", "files", "delete"], params={"fileId": args.file_id}) + print(json.dumps({"status": "deleted", "fileId": args.file_id, "permanent": True})) + return + service = build_service("drive", "v3") + service.files().delete(fileId=args.file_id).execute() + print(json.dumps({"status": "deleted", "fileId": args.file_id, "permanent": True})) + return + + # Trash (reversible). Use files.update with trashed=True. + body = {"trashed": True} + if _gws_binary(): + _run_gws( + ["drive", "files", "update"], + params={"fileId": args.file_id}, + body=body, + ) + print(json.dumps({"status": "trashed", "fileId": args.file_id, "permanent": False})) + return + + service = build_service("drive", "v3") + service.files().update(fileId=args.file_id, body=body).execute() + print(json.dumps({"status": "trashed", "fileId": args.file_id, "permanent": False})) + + +# ========================================================================= +# Contacts +# ========================================================================= + + +def contacts_list(args): + if _gws_binary(): + results = _run_gws( + ["people", "people", "connections", "list"], + params={ + "resourceName": "people/me", + "pageSize": args.max, + "personFields": "names,emailAddresses,phoneNumbers", + }, + ) + contacts = [] + for person in results.get("connections", []): + names = person.get("names", [{}]) + emails = person.get("emailAddresses", []) + phones = person.get("phoneNumbers", []) + contacts.append({ + "name": names[0].get("displayName", "") if names else "", + "emails": [e.get("value", "") for e in emails], + "phones": [p.get("value", "") for p in phones], + }) + print(json.dumps(contacts, indent=2, ensure_ascii=False)) + return + + service = build_service("people", "v1") + results = service.people().connections().list( + resourceName="people/me", + pageSize=args.max, + personFields="names,emailAddresses,phoneNumbers", + ).execute() + contacts = [] + for person in results.get("connections", []): + names = person.get("names", [{}]) + emails = person.get("emailAddresses", []) + phones = person.get("phoneNumbers", []) + contacts.append({ + "name": names[0].get("displayName", "") if names else "", + "emails": [e.get("value", "") for e in emails], + "phones": [p.get("value", "") for p in phones], + }) + print(json.dumps(contacts, indent=2, ensure_ascii=False)) + + +# ========================================================================= +# Sheets +# ========================================================================= + + +def sheets_get(args): + if _gws_binary(): + result = _run_gws( + ["sheets", "spreadsheets", "values", "get"], + params={"spreadsheetId": args.sheet_id, "range": args.range}, + ) + print(json.dumps(result.get("values", []), indent=2, ensure_ascii=False)) + return + + service = build_service("sheets", "v4") + result = service.spreadsheets().values().get( + spreadsheetId=args.sheet_id, range=args.range, + ).execute() + print(json.dumps(result.get("values", []), indent=2, ensure_ascii=False)) + + + +def sheets_update(args): + values = json.loads(args.values) + body = {"values": values} + + if _gws_binary(): + result = _run_gws( + ["sheets", "spreadsheets", "values", "update"], + params={ + "spreadsheetId": args.sheet_id, + "range": args.range, + "valueInputOption": "USER_ENTERED", + }, + body=body, + ) + print(json.dumps({"updatedCells": result.get("updatedCells", 0), "updatedRange": result.get("updatedRange", "")}, indent=2)) + return + + service = build_service("sheets", "v4") + result = service.spreadsheets().values().update( + spreadsheetId=args.sheet_id, range=args.range, + valueInputOption="USER_ENTERED", body=body, + ).execute() + print(json.dumps({"updatedCells": result.get("updatedCells", 0), "updatedRange": result.get("updatedRange", "")}, indent=2)) + + + +def sheets_append(args): + values = json.loads(args.values) + body = {"values": values} + + if _gws_binary(): + result = _run_gws( + ["sheets", "spreadsheets", "values", "append"], + params={ + "spreadsheetId": args.sheet_id, + "range": args.range, + "valueInputOption": "USER_ENTERED", + "insertDataOption": "INSERT_ROWS", + }, + body=body, + ) + print(json.dumps({"updatedCells": result.get("updates", {}).get("updatedCells", 0)}, indent=2)) + return + + service = build_service("sheets", "v4") + result = service.spreadsheets().values().append( + spreadsheetId=args.sheet_id, range=args.range, + valueInputOption="USER_ENTERED", insertDataOption="INSERT_ROWS", body=body, + ).execute() + print(json.dumps({"updatedCells": result.get("updates", {}).get("updatedCells", 0)}, indent=2)) + + +def sheets_create(args): + """Create a new spreadsheet. Returns the new spreadsheet ID and URL.""" + body = {"properties": {"title": args.title}} + if args.sheet_name: + body["sheets"] = [{"properties": {"title": args.sheet_name}}] + + if _gws_binary(): + result = _run_gws(["sheets", "spreadsheets", "create"], body=body) + print(json.dumps({ + "status": "created", + "spreadsheetId": result.get("spreadsheetId", ""), + "title": result.get("properties", {}).get("title", ""), + "spreadsheetUrl": result.get("spreadsheetUrl", ""), + }, indent=2, ensure_ascii=False)) + return + + service = build_service("sheets", "v4") + result = service.spreadsheets().create( + body=body, fields="spreadsheetId,properties,spreadsheetUrl", + ).execute() + print(json.dumps({ + "status": "created", + "spreadsheetId": result.get("spreadsheetId", ""), + "title": result.get("properties", {}).get("title", ""), + "spreadsheetUrl": result.get("spreadsheetUrl", ""), + }, indent=2, ensure_ascii=False)) + + +# ========================================================================= +# Docs +# ========================================================================= + + +def docs_get(args): + if _gws_binary(): + doc = _run_gws(["docs", "documents", "get"], params={"documentId": args.doc_id}) + result = { + "title": doc.get("title", ""), + "documentId": doc.get("documentId", ""), + "body": _extract_doc_text(doc), + } + print(json.dumps(result, indent=2, ensure_ascii=False)) + return + + service = build_service("docs", "v1") + doc = service.documents().get(documentId=args.doc_id).execute() + result = { + "title": doc.get("title", ""), + "documentId": doc.get("documentId", ""), + "body": _extract_doc_text(doc), + } + print(json.dumps(result, indent=2, ensure_ascii=False)) + + +def docs_create(args): + """Create a new Doc. Optionally seed it with initial body text.""" + body = {"title": args.title} + + if _gws_binary(): + doc = _run_gws(["docs", "documents", "create"], body=body) + else: + service = build_service("docs", "v1") + doc = service.documents().create(body=body).execute() + + doc_id = doc.get("documentId", "") + + if args.body and doc_id: + _docs_insert_text(doc_id, args.body, index=1) + + print(json.dumps({ + "status": "created", + "documentId": doc_id, + "title": doc.get("title", ""), + "url": f"https://docs.google.com/document/d/{doc_id}/edit" if doc_id else "", + }, indent=2, ensure_ascii=False)) + + +def docs_append(args): + """Append text to the end of an existing Doc.""" + if _gws_binary(): + doc = _run_gws(["docs", "documents", "get"], params={"documentId": args.doc_id}) + else: + service = build_service("docs", "v1") + doc = service.documents().get(documentId=args.doc_id).execute() + + # The end-of-body index is one less than the segment endIndex of the body + # (trailing newline is always at length-1). Docs indexes are 1-based; use + # endIndex - 1 to insert before the final newline. + content = doc.get("body", {}).get("content", []) + end_index = 1 + for element in content: + ei = element.get("endIndex") + if isinstance(ei, int) and ei > end_index: + end_index = ei + insert_index = max(end_index - 1, 1) + + text = args.text if args.text.endswith("\n") else args.text + "\n" + _docs_insert_text(args.doc_id, text, index=insert_index) + + print(json.dumps({ + "status": "appended", + "documentId": args.doc_id, + "inserted_at": insert_index, + "characters": len(text), + }, indent=2, ensure_ascii=False)) + + +def _docs_insert_text(doc_id: str, text: str, index: int) -> None: + """Send a batchUpdate with a single insertText request.""" + requests = [{ + "insertText": { + "location": {"index": index}, + "text": text, + } + }] + if _gws_binary(): + _run_gws( + ["docs", "documents", "batchUpdate"], + params={"documentId": doc_id}, + body={"requests": requests}, + ) + return + + service = build_service("docs", "v1") + service.documents().batchUpdate(documentId=doc_id, body={"requests": requests}).execute() + + +# ========================================================================= +# CLI parser +# ========================================================================= + + +def main(): + parser = argparse.ArgumentParser(description="Google Workspace API for Hermes Agent") + sub = parser.add_subparsers(dest="service", required=True) + + # --- Gmail --- + gmail = sub.add_parser("gmail") + gmail_sub = gmail.add_subparsers(dest="action", required=True) + + p = gmail_sub.add_parser("search") + p.add_argument("query", help="Gmail search query (e.g. 'is:unread')") + p.add_argument("--max", type=int, default=10) + p.set_defaults(func=gmail_search) + + p = gmail_sub.add_parser("get") + p.add_argument("message_id") + p.set_defaults(func=gmail_get) + + p = gmail_sub.add_parser("send") + p.add_argument("--to", required=True) + p.add_argument("--subject", required=True) + p.add_argument("--body", required=True) + p.add_argument("--cc", default="") + p.add_argument("--from", dest="from_header", default="", help="Custom From header (e.g. '\"Agent Name\" ')") + p.add_argument("--html", action="store_true", help="Send body as HTML") + p.add_argument("--thread-id", default="", help="Thread ID for threading") + p.set_defaults(func=gmail_send) + + p = gmail_sub.add_parser("reply") + p.add_argument("message_id", help="Message ID to reply to") + p.add_argument("--body", required=True) + p.add_argument("--from", dest="from_header", default="", help="Custom From header (e.g. '\"Agent Name\" ')") + p.set_defaults(func=gmail_reply) + + p = gmail_sub.add_parser("labels") + p.set_defaults(func=gmail_labels) + + p = gmail_sub.add_parser("modify") + p.add_argument("message_id") + p.add_argument("--add-labels", default="", help="Comma-separated label IDs to add") + p.add_argument("--remove-labels", default="", help="Comma-separated label IDs to remove") + p.set_defaults(func=gmail_modify) + + # --- Calendar --- + cal = sub.add_parser("calendar") + cal_sub = cal.add_subparsers(dest="action", required=True) + + p = cal_sub.add_parser("list") + p.add_argument("--start", default="", help="Start time (ISO 8601)") + p.add_argument("--end", default="", help="End time (ISO 8601)") + p.add_argument("--max", type=int, default=25) + p.add_argument("--calendar", default="primary") + p.set_defaults(func=calendar_list) + + p = cal_sub.add_parser("create") + p.add_argument("--summary", required=True) + p.add_argument("--start", required=True, help="Start (ISO 8601 with timezone)") + p.add_argument("--end", required=True, help="End (ISO 8601 with timezone)") + p.add_argument("--location", default="") + p.add_argument("--description", default="") + p.add_argument("--attendees", default="", help="Comma-separated email addresses") + p.add_argument("--calendar", default="primary") + p.set_defaults(func=calendar_create) + + p = cal_sub.add_parser("delete") + p.add_argument("event_id") + p.add_argument("--calendar", default="primary") + p.set_defaults(func=calendar_delete) + + # --- Drive --- + drv = sub.add_parser("drive") + drv_sub = drv.add_subparsers(dest="action", required=True) + + p = drv_sub.add_parser("search") + p.add_argument("query") + p.add_argument("--max", type=int, default=10) + p.add_argument("--raw-query", action="store_true", help="Use query as raw Drive API query") + p.set_defaults(func=drive_search) + + p = drv_sub.add_parser("get") + p.add_argument("file_id") + p.set_defaults(func=drive_get) + + p = drv_sub.add_parser("upload") + p.add_argument("path", help="Local file path to upload") + p.add_argument("--name", default="", help="Override file name in Drive (defaults to local filename)") + p.add_argument("--parent", default="", help="Parent folder ID") + p.add_argument("--mime-type", default="", help="Override MIME type (auto-detected if omitted)") + p.set_defaults(func=drive_upload) + + p = drv_sub.add_parser("download") + p.add_argument("file_id") + p.add_argument("--output", default="", help="Local output path (defaults to ./ in cwd)") + p.add_argument("--export-mime", default="", help="Export MIME for Google-native files (overrides defaults: pdf for Docs/Slides, csv for Sheets, png for Drawings)") + p.set_defaults(func=drive_download) + + p = drv_sub.add_parser("create-folder") + p.add_argument("name") + p.add_argument("--parent", default="", help="Parent folder ID (defaults to root)") + p.set_defaults(func=drive_create_folder) + + p = drv_sub.add_parser("share") + p.add_argument("file_id") + p.add_argument("--role", default="reader", choices=["reader", "commenter", "writer", "fileOrganizer", "organizer", "owner"]) + p.add_argument("--type", default="user", choices=["user", "group", "domain", "anyone"]) + p.add_argument("--email", default="", help="Email address (required for type=user or type=group)") + p.add_argument("--domain", default="", help="Domain (required for type=domain)") + p.add_argument("--notify", action="store_true", help="Send notification email") + p.set_defaults(func=drive_share) + + p = drv_sub.add_parser("delete") + p.add_argument("file_id") + p.add_argument("--permanent", action="store_true", help="Permanently delete (default is trash, which is reversible)") + p.set_defaults(func=drive_delete) + + # --- Contacts --- + con = sub.add_parser("contacts") + con_sub = con.add_subparsers(dest="action", required=True) + + p = con_sub.add_parser("list") + p.add_argument("--max", type=int, default=50) + p.set_defaults(func=contacts_list) + + # --- Sheets --- + sh = sub.add_parser("sheets") + sh_sub = sh.add_subparsers(dest="action", required=True) + + p = sh_sub.add_parser("get") + p.add_argument("sheet_id") + p.add_argument("range") + p.set_defaults(func=sheets_get) + + p = sh_sub.add_parser("update") + p.add_argument("sheet_id") + p.add_argument("range") + p.add_argument("--values", required=True, help="JSON array of arrays") + p.set_defaults(func=sheets_update) + + p = sh_sub.add_parser("append") + p.add_argument("sheet_id") + p.add_argument("range") + p.add_argument("--values", required=True, help="JSON array of arrays") + p.set_defaults(func=sheets_append) + + p = sh_sub.add_parser("create") + p.add_argument("--title", required=True, help="Spreadsheet title") + p.add_argument("--sheet-name", default="", help="Name of the first tab (defaults to 'Sheet1')") + p.set_defaults(func=sheets_create) + + # --- Docs --- + docs = sub.add_parser("docs") + docs_sub = docs.add_subparsers(dest="action", required=True) + + p = docs_sub.add_parser("get") + p.add_argument("doc_id") + p.set_defaults(func=docs_get) + + p = docs_sub.add_parser("create") + p.add_argument("--title", required=True, help="Document title") + p.add_argument("--body", default="", help="Initial body text (optional)") + p.set_defaults(func=docs_create) + + p = docs_sub.add_parser("append") + p.add_argument("doc_id") + p.add_argument("--text", required=True, help="Text to append to the end of the document") + p.set_defaults(func=docs_append) + + args = parser.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/google-workspace/scripts/gws_bridge.py b/google-workspace/scripts/gws_bridge.py new file mode 100755 index 0000000..5fa7a9d --- /dev/null +++ b/google-workspace/scripts/gws_bridge.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +"""Bridge between Hermes OAuth token and gws CLI. + +Refreshes the token if expired, then executes gws with the valid access token. +""" +import json +import os +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +# Ensure sibling modules (_hermes_home) are importable when run standalone. +_SCRIPTS_DIR = str(Path(__file__).resolve().parent) +if _SCRIPTS_DIR not in sys.path: + sys.path.insert(0, _SCRIPTS_DIR) + +from _hermes_home import get_hermes_home + + +def get_token_path() -> Path: + return get_hermes_home() / "google_token.json" + + +def _normalize_authorized_user_payload(payload: dict) -> dict: + normalized = dict(payload) + if not normalized.get("type"): + normalized["type"] = "authorized_user" + return normalized + + +def refresh_token(token_data: dict) -> dict: + """Refresh the access token using the refresh token.""" + import urllib.error + import urllib.parse + import urllib.request + + required_keys = ["client_id", "client_secret", "refresh_token", "token_uri"] + missing = [k for k in required_keys if k not in token_data] + if missing: + print(f"ERROR: google_token.json is missing required fields: {', '.join(missing)}", file=sys.stderr) + print("Please re-authenticate by running the Google Workspace setup script.", file=sys.stderr) + sys.exit(1) + + params = urllib.parse.urlencode({ + "client_id": token_data["client_id"], + "client_secret": token_data["client_secret"], + "refresh_token": token_data["refresh_token"], + "grant_type": "refresh_token", + }).encode() + + req = urllib.request.Request(token_data["token_uri"], data=params) + try: + with urllib.request.urlopen(req, timeout=15) as resp: + result = json.loads(resp.read()) + except urllib.error.HTTPError as e: + body = e.read().decode("utf-8", errors="replace") + print(f"ERROR: Token refresh failed (HTTP {e.code}): {body}", file=sys.stderr) + print("Re-run setup.py to re-authenticate.", file=sys.stderr) + sys.exit(1) + except (urllib.error.URLError, TimeoutError) as e: + print(f"ERROR: Token refresh failed (network): {e}", file=sys.stderr) + sys.exit(1) + + token_data["token"] = result["access_token"] + token_data["expiry"] = datetime.fromtimestamp( + datetime.now(timezone.utc).timestamp() + result["expires_in"], + tz=timezone.utc, + ).isoformat() + + get_token_path().write_text( + json.dumps(_normalize_authorized_user_payload(token_data), indent=2), encoding="utf-8" + ) + return token_data + + +def get_valid_token() -> str: + """Return a valid access token, refreshing if needed.""" + token_path = get_token_path() + if not token_path.exists(): + print("ERROR: No Google token found. Run setup.py --auth-url first.", file=sys.stderr) + sys.exit(1) + + token_data = json.loads(token_path.read_text(encoding="utf-8")) + + expiry = token_data.get("expiry", "") + if expiry: + exp_dt = datetime.fromisoformat(expiry.replace("Z", "+00:00")) + now = datetime.now(timezone.utc) + if now >= exp_dt: + token_data = refresh_token(token_data) + + return token_data["token"] + + +def main(): + """Refresh token if needed, then exec gws with remaining args.""" + if len(sys.argv) < 2: + print("Usage: gws_bridge.py ", file=sys.stderr) + sys.exit(1) + + access_token = get_valid_token() + env = os.environ.copy() + env["GOOGLE_WORKSPACE_CLI_TOKEN"] = access_token + + result = subprocess.run(["gws"] + sys.argv[1:], env=env) + sys.exit(result.returncode) + + +if __name__ == "__main__": + main() diff --git a/google-workspace/scripts/setup.py b/google-workspace/scripts/setup.py new file mode 100644 index 0000000..375fcd9 --- /dev/null +++ b/google-workspace/scripts/setup.py @@ -0,0 +1,514 @@ +#!/usr/bin/env python3 +"""Google Workspace OAuth2 setup for Hermes Agent. + +Fully non-interactive — designed to be driven by the agent via terminal commands. +The agent mediates between this script and the user (works on CLI, Telegram, Discord, etc.) + +Commands: + setup.py --check # Is auth valid? Exit 0 = yes, 1 = no + setup.py --client-secret /path/to.json # Store OAuth client credentials + setup.py --auth-url # Print the OAuth URL for user to visit + setup.py --auth-code CODE # Exchange auth code for token + setup.py --revoke # Revoke and delete stored token + setup.py --install-deps # Install Python dependencies only + +Agent workflow: + 1. Run --check. If exit 0, auth is good — skip setup. + 2. Ask user for client_secret.json path. Run --client-secret PATH. + 3. Run --auth-url. Send the printed URL to the user. + 4. User opens URL, authorizes, gets redirected to a page with a code. + 5. User pastes the code. Agent runs --auth-code CODE. + 6. Run --check to verify. Done. +""" + +from __future__ import annotations # allow PEP 604 `X | None` on Python 3.9+ + +import argparse +import json +import os +import shutil +import subprocess +import sys +from importlib.metadata import version as _distribution_version +from pathlib import Path + +# Ensure sibling modules (_hermes_home) are importable when run standalone. +_SCRIPTS_DIR = str(Path(__file__).resolve().parent) +if _SCRIPTS_DIR not in sys.path: + sys.path.insert(0, _SCRIPTS_DIR) + +from _hermes_home import display_hermes_home, get_hermes_home + +HERMES_HOME = get_hermes_home() +TOKEN_PATH = HERMES_HOME / "google_token.json" +CLIENT_SECRET_PATH = HERMES_HOME / "google_client_secret.json" +PENDING_AUTH_PATH = HERMES_HOME / "google_oauth_pending.json" + +SCOPES = [ + "https://www.googleapis.com/auth/gmail.readonly", + "https://www.googleapis.com/auth/gmail.send", + "https://www.googleapis.com/auth/gmail.modify", + "https://www.googleapis.com/auth/calendar", + "https://www.googleapis.com/auth/drive", + "https://www.googleapis.com/auth/contacts.readonly", + "https://www.googleapis.com/auth/spreadsheets", + "https://www.googleapis.com/auth/documents", +] + +# Exact pins: keep in sync with pyproject.toml [project.optional-dependencies].google +# and tools/lazy_deps.py LAZY_DEPS['skill.google_workspace']. +# Pinning all protects against version drift and ensures the security floors +# (httplib2 GHSA-j5g9-f88f-gfj3, stale pyasn1/google-auth) are honoured +# regardless of install path. +REQUIRED_PACKAGES = [ + "google-api-python-client==2.194.0", + "google-auth==2.55.1", + "google-auth-oauthlib==1.3.1", + "google-auth-httplib2==0.3.1", + # GHSA-j5g9-f88f-gfj3 — Decompression Bomb DoS via unbounded gzip/deflate + "httplib2==0.32.0", + "pyasn1==0.6.4", +] + +# OAuth redirect for "out of band" manual code copy flow. +# Google deprecated OOB, so we use a localhost redirect and tell the user to +# copy the code from the browser's URL bar (or the page body). +REDIRECT_URI = "http://localhost:1" + + +def _normalize_authorized_user_payload(payload: dict) -> dict: + normalized = dict(payload) + if not normalized.get("type"): + normalized["type"] = "authorized_user" + return normalized + + +def _load_token_payload(path: Path = TOKEN_PATH) -> dict: + try: + return json.loads(path.read_text(encoding="utf-8")) + except Exception: + return {} + + +def _missing_scopes_from_payload(payload: dict) -> list[str]: + raw = payload.get("scopes") or payload.get("scope") + if not raw: + return [] + granted = {s.strip() for s in (raw.split() if isinstance(raw, str) else raw) if s.strip()} + return sorted(scope for scope in SCOPES if scope not in granted) + + +def _format_missing_scopes(missing_scopes: list[str]) -> str: + bullets = "\n".join(f" - {scope}" for scope in missing_scopes) + return ( + "Token is valid but missing required Google Workspace scopes:\n" + f"{bullets}\n" + "Run the Google Workspace setup again from this same Hermes profile to refresh consent." + ) + + +def _missing_required_packages() -> list[str]: + """Return exact requirements absent or stale in this interpreter. + + All REQUIRED_PACKAGES entries are exact ``name==version`` pins, so a + direct version comparison is sufficient — no ``packaging`` dependency + needed in this standalone script. + """ + missing = [] + for spec in REQUIRED_PACKAGES: + name, _, wanted = spec.partition("==") + try: + if _distribution_version(name) != wanted: + missing.append(spec) + except Exception: + missing.append(spec) + return missing + + +def install_deps(): + """Install missing or stale Google API packages. Returns True on success.""" + missing = _missing_required_packages() + if not missing: + print("Dependencies already installed.") + return True + + print("Installing Google API dependencies...") + + # First choice: pip in the current interpreter. Works for most installs. + try: + subprocess.check_call( + [sys.executable, "-m", "pip", "install", "--quiet"] + missing, + stdout=subprocess.DEVNULL, + ) + remaining = _missing_required_packages() + if remaining: + print(f"ERROR: Dependencies remain stale after pip install: {' '.join(remaining)}") + return False + print("Dependencies installed.") + return True + except subprocess.CalledProcessError as e: + pip_error = e + + # Fallback: the interpreter has no pip (the Hermes Docker image's venv is + # built with `uv sync`, which does not bootstrap pip). `uv pip install + # --python ` installs into that exact interpreter without + # needing pip present. Targeting sys.executable keeps us on the venv the + # script is actually running under, rather than guessing. + uv = shutil.which("uv") + if uv: + try: + subprocess.check_call( + [uv, "pip", "install", "--python", sys.executable, "--quiet"] + + missing, + stdout=subprocess.DEVNULL, + ) + remaining = _missing_required_packages() + if remaining: + print(f"ERROR: Dependencies remain stale after uv install: {' '.join(remaining)}") + return False + print("Dependencies installed.") + return True + except subprocess.CalledProcessError as e: + print(f"ERROR: Failed to install dependencies via uv: {e}") + print(f"Manually: {uv} pip install --python {sys.executable} {' '.join(REQUIRED_PACKAGES)}") + return False + + print(f"ERROR: Failed to install dependencies: {pip_error}") + print( + "On environments without pip (e.g. Nix, or the Hermes Docker image's " + "uv-managed venv), install the optional extra instead:" + ) + print(" hermes setup") + print(f"Or manually: {sys.executable} -m pip install {' '.join(REQUIRED_PACKAGES)}") + return False + + +def _ensure_deps(): + """Check exact dependency versions, install if stale, exit on failure.""" + if _missing_required_packages() and not install_deps(): + sys.exit(1) + + +def check_auth_live(): + """Check auth with a real API call to detect disabled_client/account issues.""" + # quiet=True suppresses the "AUTHENTICATED" print from check_auth so the + # final status line reflects the live-call outcome (OK or FAILED). + if not check_auth(quiet=True): + return False + try: + from googleapiclient.discovery import build + from google.oauth2.credentials import Credentials + creds = Credentials.from_authorized_user_file(str(TOKEN_PATH)) + service = build("calendar", "v3", credentials=creds) + service.calendarList().list(maxResults=1).execute() + print("LIVE_CHECK_OK: Real API call succeeded.") + return True + except Exception as e: + err_str = str(e).lower() + if "disabled_client" in err_str or "invalid_client" in err_str: + print(f"LIVE_CHECK_FAILED: OAuth client or account disabled: {e}") + print(" 1. Check Google Cloud Console for disabled OAuth client") + print(" 2. Check myaccount.google.com for account status") + print(" 3. Do NOT retry with a disabled account") + else: + print(f"LIVE_CHECK_FAILED: {e}") + return False + + +def check_auth(quiet: bool = False): + """Check if stored credentials are valid. Prints status, exits 0 or 1.""" + if not TOKEN_PATH.exists(): + print(f"NOT_AUTHENTICATED: No token at {TOKEN_PATH}") + return False + + _ensure_deps() + from google.oauth2.credentials import Credentials + from google.auth.transport.requests import Request + + try: + # Don't pass scopes — user may have authorized only a subset. + # Passing scopes forces google-auth to validate them on refresh, + # which fails with invalid_scope if the token has fewer scopes + # than requested. + creds = Credentials.from_authorized_user_file(str(TOKEN_PATH)) + except Exception as e: + print(f"TOKEN_CORRUPT: {e}") + return False + + payload = _load_token_payload(TOKEN_PATH) + if creds.valid: + missing_scopes = _missing_scopes_from_payload(payload) + if missing_scopes: + print(f"AUTHENTICATED (partial): Token valid but missing {len(missing_scopes)} scopes:") + for s in missing_scopes: + print(f" - {s}") + if not quiet: + print(f"AUTHENTICATED: Token valid at {TOKEN_PATH}") + return True + + if creds.expired and creds.refresh_token: + try: + creds.refresh(Request()) + TOKEN_PATH.write_text( + json.dumps( + _normalize_authorized_user_payload(json.loads(creds.to_json())), + indent=2, + ), encoding="utf-8" + ) + missing_scopes = _missing_scopes_from_payload(_load_token_payload(TOKEN_PATH)) + if missing_scopes: + print(f"AUTHENTICATED (partial): Token refreshed but missing {len(missing_scopes)} scopes:") + for s in missing_scopes: + print(f" - {s}") + if not quiet: + print(f"AUTHENTICATED: Token refreshed at {TOKEN_PATH}") + return True + except Exception as e: + err_str = str(e).lower() + if "disabled_client" in err_str or "invalid_client" in err_str: + print(f"OAUTH_CLIENT_DISABLED: {e}") + print(" The OAuth client or Google account has been disabled.") + print(" Steps to resolve:") + print(" 1. Check your Google Cloud Console — verify the OAuth client is not disabled") + print(" 2. Check if your Google account itself has been disabled at myaccount.google.com") + print(" 3. If the account is disabled, you can appeal at accounts.google.com/signin/recovery") + print(" 4. Do NOT retry API calls with a disabled account — this may worsen the situation") + print(" 5. If the OAuth client is disabled, create a new one in Google Cloud Console") + elif "token_revoked" in err_str or "invalid_grant" in err_str: + print(f"TOKEN_REVOKED: {e}") + print(" Re-run setup to re-authenticate.") + else: + print(f"REFRESH_FAILED: {e}") + return False + + print("TOKEN_INVALID: Re-run setup.") + return False + + +def store_client_secret(path: str): + """Copy and validate client_secret.json to Hermes home.""" + src = Path(path).expanduser().resolve() + if not src.exists(): + print(f"ERROR: File not found: {src}") + sys.exit(1) + + try: + data = json.loads(src.read_text(encoding="utf-8")) + except json.JSONDecodeError: + print("ERROR: File is not valid JSON.") + sys.exit(1) + + if "installed" not in data and "web" not in data: + print("ERROR: Not a Google OAuth client secret file (missing 'installed' key).") + print("Download the correct file from: https://console.cloud.google.com/apis/credentials") + sys.exit(1) + + CLIENT_SECRET_PATH.write_text(json.dumps(data, indent=2), encoding="utf-8") + print(f"OK: Client secret saved to {CLIENT_SECRET_PATH}") + + +def _save_pending_auth(*, state: str, code_verifier: str): + """Persist the OAuth session bits needed for a later token exchange.""" + PENDING_AUTH_PATH.write_text( + json.dumps( + { + "state": state, + "code_verifier": code_verifier, + "redirect_uri": REDIRECT_URI, + }, + indent=2, + ), encoding="utf-8" + ) + + +def _load_pending_auth() -> dict: + """Load the pending OAuth session created by get_auth_url().""" + if not PENDING_AUTH_PATH.exists(): + print("ERROR: No pending OAuth session found. Run --auth-url first.") + sys.exit(1) + + try: + data = json.loads(PENDING_AUTH_PATH.read_text(encoding="utf-8")) + except Exception as e: + print(f"ERROR: Could not read pending OAuth session: {e}") + print("Run --auth-url again to start a fresh OAuth session.") + sys.exit(1) + + if not data.get("state") or not data.get("code_verifier"): + print("ERROR: Pending OAuth session is missing PKCE data.") + print("Run --auth-url again to start a fresh OAuth session.") + sys.exit(1) + + return data + + +def _extract_code_and_state(code_or_url: str) -> tuple[str, str | None]: + """Accept either a raw auth code or the full redirect URL pasted by the user.""" + if not code_or_url.startswith("http"): + return code_or_url, None + + from urllib.parse import parse_qs, urlparse + + parsed = urlparse(code_or_url) + params = parse_qs(parsed.query) + if "code" not in params: + print("ERROR: No 'code' parameter found in URL.") + sys.exit(1) + + state = params.get("state", [None])[0] + return params["code"][0], state + + +def get_auth_url(): + """Print the OAuth authorization URL. User visits this in a browser.""" + if not CLIENT_SECRET_PATH.exists(): + print("ERROR: No client secret stored. Run --client-secret first.") + sys.exit(1) + + _ensure_deps() + from google_auth_oauthlib.flow import Flow + + flow = Flow.from_client_secrets_file( + str(CLIENT_SECRET_PATH), + scopes=SCOPES, + redirect_uri=REDIRECT_URI, + autogenerate_code_verifier=True, + ) + auth_url, state = flow.authorization_url( + access_type="offline", + prompt="consent", + ) + _save_pending_auth(state=state, code_verifier=flow.code_verifier) + # Print just the URL so the agent can extract it cleanly + print(auth_url) + + +def exchange_auth_code(code: str): + """Exchange the authorization code for a token and save it.""" + if not CLIENT_SECRET_PATH.exists(): + print("ERROR: No client secret stored. Run --client-secret first.") + sys.exit(1) + + pending_auth = _load_pending_auth() + raw_callback = code + code, returned_state = _extract_code_and_state(code) + if returned_state and returned_state != pending_auth["state"]: + print("ERROR: OAuth state mismatch. Run --auth-url again to start a fresh session.") + sys.exit(1) + + _ensure_deps() + from google_auth_oauthlib.flow import Flow + from urllib.parse import parse_qs, urlparse + + # Extract granted scopes from the callback URL if the user pasted the full redirect URL. + granted_scopes = list(SCOPES) + if isinstance(raw_callback, str) and raw_callback.startswith("http"): + params = parse_qs(urlparse(raw_callback).query) + scope_val = (params.get("scope") or [""])[0].strip() + if scope_val: + granted_scopes = scope_val.split() + + flow = Flow.from_client_secrets_file( + str(CLIENT_SECRET_PATH), + scopes=granted_scopes, + redirect_uri=pending_auth.get("redirect_uri", REDIRECT_URI), + state=pending_auth["state"], + code_verifier=pending_auth["code_verifier"], + ) + + try: + # Accept partial scopes — user may deselect some permissions in the consent screen + os.environ["OAUTHLIB_RELAX_TOKEN_SCOPE"] = "1" + flow.fetch_token(code=code) + except Exception as e: + print(f"ERROR: Token exchange failed: {e}") + print("The code may have expired. Run --auth-url to get a fresh URL.") + sys.exit(1) + + creds = flow.credentials + token_payload = _normalize_authorized_user_payload(json.loads(creds.to_json())) + + # Store only the scopes actually granted by the user, not what was requested. + # creds.to_json() writes the requested scopes, which causes refresh to fail + # with invalid_scope if the user only authorized a subset. + actually_granted = list(creds.granted_scopes or []) if hasattr(creds, "granted_scopes") and creds.granted_scopes else [] + if actually_granted: + token_payload["scopes"] = actually_granted + elif granted_scopes != SCOPES: + # granted_scopes was extracted from the callback URL + token_payload["scopes"] = granted_scopes + + missing_scopes = _missing_scopes_from_payload(token_payload) + if missing_scopes: + print(f"WARNING: Token missing some Google Workspace scopes: {', '.join(missing_scopes)}") + print("Some services may not be available.") + + TOKEN_PATH.write_text(json.dumps(token_payload, indent=2), encoding="utf-8") + PENDING_AUTH_PATH.unlink(missing_ok=True) + print(f"OK: Authenticated. Token saved to {TOKEN_PATH}") + print(f"Profile-scoped token location: {display_hermes_home()}/google_token.json") + + +def revoke(): + """Revoke stored token and delete it.""" + if not TOKEN_PATH.exists(): + print("No token to revoke.") + return + + _ensure_deps() + from google.oauth2.credentials import Credentials + from google.auth.transport.requests import Request + + try: + creds = Credentials.from_authorized_user_file(str(TOKEN_PATH), SCOPES) + if creds.expired and creds.refresh_token: + creds.refresh(Request()) + + import urllib.request + urllib.request.urlopen( + urllib.request.Request( + f"https://oauth2.googleapis.com/revoke?token={creds.token}", + method="POST", + headers={"Content-Type": "application/x-www-form-urlencoded"}, + ), + timeout=15, + ) + print("Token revoked with Google.") + except Exception as e: + print(f"Remote revocation failed (token may already be invalid): {e}") + + TOKEN_PATH.unlink(missing_ok=True) + PENDING_AUTH_PATH.unlink(missing_ok=True) + print(f"Deleted {TOKEN_PATH}") + + +def main(): + parser = argparse.ArgumentParser(description="Google Workspace OAuth setup for Hermes") + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument("--check", action="store_true", help="Check if auth is valid (exit 0=yes, 1=no)") + group.add_argument("--check-live", action="store_true", help="Check auth with a real API call (detects disabled_client)") + group.add_argument("--client-secret", metavar="PATH", help="Store OAuth client_secret.json") + group.add_argument("--auth-url", action="store_true", help="Print OAuth URL for user to visit") + group.add_argument("--auth-code", metavar="CODE", help="Exchange auth code for token") + group.add_argument("--revoke", action="store_true", help="Revoke and delete stored token") + group.add_argument("--install-deps", action="store_true", help="Install Python dependencies") + args = parser.parse_args() + + if args.check: + sys.exit(0 if check_auth() else 1) + if getattr(args, "check_live", False): + sys.exit(0 if check_auth_live() else 1) + elif args.client_secret: + store_client_secret(args.client_secret) + elif args.auth_url: + get_auth_url() + elif args.auth_code: + exchange_auth_code(args.auth_code) + elif args.revoke: + revoke() + elif args.install_deps: + sys.exit(0 if install_deps() else 1) + + +if __name__ == "__main__": + main() diff --git a/linear/SKILL.md b/linear/SKILL.md new file mode 100644 index 0000000..a08a03e --- /dev/null +++ b/linear/SKILL.md @@ -0,0 +1,380 @@ +--- +name: linear +description: "Linear: manage issues, projects, teams via GraphQL + curl." +version: 1.0.0 +author: Hermes Agent +license: MIT +platforms: [linux, macos, windows] +prerequisites: + env_vars: [LINEAR_API_KEY] + commands: [curl] +metadata: + hermes: + tags: [Linear, Project Management, Issues, GraphQL, API, Productivity] +--- + +# Linear — Issue & Project Management + +Manage Linear issues, projects, and teams directly via the GraphQL API using `curl`. No MCP server, no OAuth flow, no extra dependencies. + +## Setup + +1. Get a personal API key from **Linear Settings > Account > Security & access > Personal API keys** (URL: https://linear.app/settings/account/security). Note: the org-level *Settings > API* page only shows OAuth apps and workspace-member keys, not personal keys. +2. Set `LINEAR_API_KEY` in your environment (via `hermes setup` or your env config) + +## API Basics + +- **Endpoint:** `https://api.linear.app/graphql` (POST) +- **Auth header:** `Authorization: $LINEAR_API_KEY` (no "Bearer" prefix for API keys) +- **All requests are POST** with `Content-Type: application/json` +- **Both UUIDs and short identifiers** (e.g., `ENG-123`) work for `issue(id:)` + +Base curl pattern: +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ viewer { id name } }"}' | python3 -m json.tool +``` + +## Python helper script (ergonomic alternative) + +For faster one-liners that don't need hand-written GraphQL, this skill ships a stdlib Python CLI at `scripts/linear_api.py`. Zero dependencies. Same auth (reads `LINEAR_API_KEY`). + +```bash +SCRIPT=$(dirname "$(find ~/.hermes -path '*skills/productivity/linear/scripts/linear_api.py' 2>/dev/null | head -1)")/linear_api.py + +python3 "$SCRIPT" whoami +python3 "$SCRIPT" list-teams +python3 "$SCRIPT" get-issue ENG-42 +python3 "$SCRIPT" get-document 38359beef67c # fetch a doc by slugId from the URL +python3 "$SCRIPT" raw 'query { viewer { name } }' +``` + +All subcommands: `whoami`, `list-teams`, `list-projects`, `list-states`, `list-issues`, `get-issue`, `search-issues`, `create-issue`, `update-issue`, `update-status`, `add-comment`, `list-documents`, `get-document`, `search-documents`, `raw`. Run with `--help` for flags. + +Use the script when: you want a quick answer without crafting GraphQL. Use curl when: you need a query the script doesn't wrap, or you want to compose filters inline. + +## Workflow States + +Linear uses `WorkflowState` objects with a `type` field. **6 state types:** + +| Type | Description | +|------|-------------| +| `triage` | Incoming issues needing review | +| `backlog` | Acknowledged but not yet planned | +| `unstarted` | Planned/ready but not started | +| `started` | Actively being worked on | +| `completed` | Done | +| `canceled` | Won't do | + +Each team has its own named states (e.g., "In Progress" is type `started`). To change an issue's status, you need the `stateId` (UUID) of the target state — query workflow states first. + +**Priority values:** 0 = None, 1 = Urgent, 2 = High, 3 = Medium, 4 = Low + +## Common Queries + +### Get current user +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ viewer { id name email } }"}' | python3 -m json.tool +``` + +### List teams +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ teams { nodes { id name key } } }"}' | python3 -m json.tool +``` + +### List workflow states for a team +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ workflowStates(filter: { team: { key: { eq: \"ENG\" } } }) { nodes { id name type } } }"}' | python3 -m json.tool +``` + +### List issues (first 20) +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ issues(first: 20) { nodes { identifier title priority state { name type } assignee { name } team { key } url } pageInfo { hasNextPage endCursor } } }"}' | python3 -m json.tool +``` + +### List my assigned issues +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ viewer { assignedIssues(first: 25) { nodes { identifier title state { name type } priority url } } } }"}' | python3 -m json.tool +``` + +### Get a single issue (by identifier like ENG-123) +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ issue(id: \"ENG-123\") { id identifier title description priority state { id name type } assignee { id name } team { key } project { name } labels { nodes { name } } comments { nodes { body user { name } createdAt } } url } }"}' | python3 -m json.tool +``` + +### Search issues by text +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ issueSearch(query: \"bug login\", first: 10) { nodes { identifier title state { name } assignee { name } url } } }"}' | python3 -m json.tool +``` + +### Filter issues by state type +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ issues(filter: { state: { type: { in: [\"started\"] } } }, first: 20) { nodes { identifier title state { name } assignee { name } } } }"}' | python3 -m json.tool +``` + +### Filter by team and assignee +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ issues(filter: { team: { key: { eq: \"ENG\" } }, assignee: { email: { eq: \"user@example.com\" } } }, first: 20) { nodes { identifier title state { name } priority } } }"}' | python3 -m json.tool +``` + +### List projects +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ projects(first: 20) { nodes { id name description progress lead { name } teams { nodes { key } } url } } }"}' | python3 -m json.tool +``` + +### List team members +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ users { nodes { id name email active } } }"}' | python3 -m json.tool +``` + +### List labels +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ issueLabels { nodes { id name color } } }"}' | python3 -m json.tool +``` + +## Common Mutations + +### Create an issue +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "query": "mutation($input: IssueCreateInput!) { issueCreate(input: $input) { success issue { id identifier title url } } }", + "variables": { + "input": { + "teamId": "TEAM_UUID", + "title": "Fix login bug", + "description": "Users cannot login with SSO", + "priority": 2 + } + } + }' | python3 -m json.tool +``` + +### Update issue status +First get the target state UUID from the workflow states query above, then: +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "mutation { issueUpdate(id: \"ENG-123\", input: { stateId: \"STATE_UUID\" }) { success issue { identifier state { name type } } } }"}' | python3 -m json.tool +``` + +### Assign an issue +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "mutation { issueUpdate(id: \"ENG-123\", input: { assigneeId: \"USER_UUID\" }) { success issue { identifier assignee { name } } } }"}' | python3 -m json.tool +``` + +### Set priority +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "mutation { issueUpdate(id: \"ENG-123\", input: { priority: 1 }) { success issue { identifier priority } } }"}' | python3 -m json.tool +``` + +### Add a comment +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "mutation { commentCreate(input: { issueId: \"ISSUE_UUID\", body: \"Investigated. Root cause is X.\" }) { success comment { id body } } }"}' | python3 -m json.tool +``` + +### Set due date +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "mutation { issueUpdate(id: \"ENG-123\", input: { dueDate: \"2026-04-01\" }) { success issue { identifier dueDate } } }"}' | python3 -m json.tool +``` + +### Add labels to an issue +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "mutation { issueUpdate(id: \"ENG-123\", input: { labelIds: [\"LABEL_UUID_1\", \"LABEL_UUID_2\"] }) { success issue { identifier labels { nodes { name } } } } }"}' | python3 -m json.tool +``` + +### Add issue to a project +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "mutation { issueUpdate(id: \"ENG-123\", input: { projectId: \"PROJECT_UUID\" }) { success issue { identifier project { name } } } }"}' | python3 -m json.tool +``` + +### Create a project +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "query": "mutation($input: ProjectCreateInput!) { projectCreate(input: $input) { success project { id name url } } }", + "variables": { + "input": { + "name": "Q2 Auth Overhaul", + "description": "Replace legacy auth with OAuth2 and PKCE", + "teamIds": ["TEAM_UUID"] + } + } + }' | python3 -m json.tool +``` + +## Documents + +Linear **Documents** are prose docs (RFCs, specs, notes) stored alongside issues. They have their own `documents` root query and `document(id:)` single-fetch. + +### Document URLs and `slugId` + +Document URLs look like: +``` +https://linear.app//document/- +``` + +The trailing hex segment is the `slugId`. Example: `https://linear.app/nousresearch/document/rfc-hermes-permission-gateway-discord-38359beef67c` → `slugId` is `38359beef67c`. + +**Important schema detail:** the Markdown body is in the `content` field. The ProseMirror JSON is in `contentState` (not `contentData` — that field does not exist and the API returns 400). + +### Fetch a document by slugId + +`document(id:)` only accepts UUIDs. To fetch by the URL's hex slug, filter the collection: + +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "query($s: String!) { documents(filter: { slugId: { eq: $s } }, first: 1) { nodes { id title content contentState slugId url creator { name } project { name } updatedAt } } }", "variables": {"s": "38359beef67c"}}' \ + | python3 -m json.tool +``` + +Or via the Python helper: +```bash +python3 scripts/linear_api.py get-document 38359beef67c +``` + +### Fetch a document by UUID + +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ document(id: \"11700cff-b514-4db3-afcc-3ed1afacba1c\") { title content url } }"}' \ + | python3 -m json.tool +``` + +### List recent documents + +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ documents(first: 25, orderBy: updatedAt) { nodes { id title slugId url updatedAt project { name } } } }"}' \ + | python3 -m json.tool +``` + +### Search documents by title + +Linear's schema has no `searchDocuments` root. Use a title-substring filter instead: + +```bash +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ documents(filter: { title: { containsIgnoreCase: \"RFC\" } }, first: 25) { nodes { title slugId url } } }"}' \ + | python3 -m json.tool +``` + +## Pagination + +Linear uses Relay-style cursor pagination: + +```bash +# First page +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ issues(first: 20) { nodes { identifier title } pageInfo { hasNextPage endCursor } } }"}' | python3 -m json.tool + +# Next page — use endCursor from previous response +curl -s -X POST https://api.linear.app/graphql \ + -H "Authorization: $LINEAR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "{ issues(first: 20, after: \"CURSOR_FROM_PREVIOUS\") { nodes { identifier title } pageInfo { hasNextPage endCursor } } }"}' | python3 -m json.tool +``` + +Default page size: 50. Max: 250. Always use `first: N` to limit results. + +## Filtering Reference + +Comparators: `eq`, `neq`, `in`, `nin`, `lt`, `lte`, `gt`, `gte`, `contains`, `startsWith`, `containsIgnoreCase` + +Combine filters with `or: [...]` for OR logic (default is AND within a filter object). + +## Typical Workflow + +1. **Query teams** to get team IDs and keys +2. **Query workflow states** for target team to get state UUIDs +3. **List or search issues** to find what needs work +4. **Create issues** with team ID, title, description, priority +5. **Update status** by setting `stateId` to the target workflow state +6. **Add comments** to track progress +7. **Mark complete** by setting `stateId` to the team's "completed" type state + +## Rate Limits + +- 5,000 requests/hour per API key +- 3,000,000 complexity points/hour +- Use `first: N` to limit results and reduce complexity cost +- Monitor `X-RateLimit-Requests-Remaining` response header + +## Important Notes + +- Always use `terminal` tool with `curl` for API calls — do NOT use `web_extract` or `browser` +- Always check the `errors` array in GraphQL responses — HTTP 200 can still contain errors +- If `stateId` is omitted when creating issues, Linear defaults to the first backlog state +- The `description` field supports Markdown +- Use `python3 -m json.tool` or `jq` to format JSON responses for readability diff --git a/linear/scripts/linear_api.py b/linear/scripts/linear_api.py new file mode 100644 index 0000000..cb8c5d8 --- /dev/null +++ b/linear/scripts/linear_api.py @@ -0,0 +1,445 @@ +#!/usr/bin/env python3 +"""Linear GraphQL API CLI — zero dependencies, stdlib only. + +Usage: + linear_api.py [args...] + +Commands: + whoami Show authenticated user + list-teams List all teams + list-projects [--team KEY] List projects (optionally filter by team) + list-states [--team KEY] List workflow states + list-issues [filters] List issues + --team KEY Filter by team key (e.g. ENG) + --status NAME Filter by workflow state name + --assignee NAME Filter by assignee name (exact) + --label NAME Filter by label name + --limit N Max results (default: 25) + get-issue Full issue details (e.g. ENG-42) + search-issues Full-text search across issues + create-issue [options] Create a new issue + --title TITLE Required + --team KEY Required + --description DESC + --priority 0-4 0=none, 1=urgent, 4=low + --label NAME + --assignee NAME + --parent IDENTIFIER Parent issue ID for sub-issues + update-issue [options] Update existing issue (same options as create) + update-status Move issue to workflow state (by state name) + add-comment Add comment to issue + + list-documents [--limit N] List documents (docs, not issues) + get-document Fetch a document by slugId (from URL) or UUID + search-documents Search documents by title + + raw [variables_json] Run an arbitrary GraphQL query + Use --vars '{"key":"value"}' for variables + +Auth: + Set LINEAR_API_KEY environment variable (from Linear Settings -> API). + Uses the personal API key header format: `Authorization: ` (no Bearer prefix). + +Output: + JSON to stdout. Errors to stderr with non-zero exit code. +""" +from __future__ import annotations + +import argparse +import json +import os +import sys +import urllib.error +import urllib.request +from typing import Any + +API_URL = "https://api.linear.app/graphql" + + +def _get_key() -> str: + key = os.environ.get("LINEAR_API_KEY", "").strip() + if not key: + sys.stderr.write( + "ERROR: LINEAR_API_KEY not set.\n" + "Create one at https://linear.app/settings/api and export it,\n" + "or add `LINEAR_API_KEY=lin_api_...` to ~/.hermes/.env\n" + ) + sys.exit(2) + return key + + +def gql(query: str, variables: dict[str, Any] | None = None) -> dict[str, Any]: + """Execute a GraphQL query against Linear. Raises on HTTP error or GraphQL errors.""" + key = _get_key() + payload = {"query": query} + if variables: + payload["variables"] = variables + data = json.dumps(payload).encode("utf-8") + req = urllib.request.Request( + API_URL, + data=data, + headers={ + "Content-Type": "application/json", + "Authorization": key, # Personal API key — NO `Bearer` prefix + "User-Agent": "hermes-agent-linear-skill/1.0", + }, + method="POST", + ) + try: + with urllib.request.urlopen(req, timeout=30) as resp: + body = resp.read().decode("utf-8") + except urllib.error.HTTPError as e: + sys.stderr.write(f"HTTP {e.code}: {e.read().decode('utf-8', 'replace')}\n") + sys.exit(1) + except urllib.error.URLError as e: + sys.stderr.write(f"Network error: {e}\n") + sys.exit(1) + + result = json.loads(body) + if "errors" in result and result["errors"]: + sys.stderr.write(f"GraphQL errors: {json.dumps(result['errors'], indent=2)}\n") + # Still return data if partial success; let caller decide + if not result.get("data"): + sys.exit(1) + return result.get("data", {}) or {} + + +def emit(obj: Any) -> None: + print(json.dumps(obj, indent=2, default=str)) + + +# ---------- Commands ---------- + +def cmd_whoami(_args: argparse.Namespace) -> None: + q = "query { viewer { id name email displayName } }" + emit(gql(q).get("viewer")) + + +def cmd_list_teams(_args: argparse.Namespace) -> None: + q = "query { teams(first: 100) { nodes { id key name description } } }" + emit(gql(q).get("teams", {}).get("nodes", [])) + + +def _resolve_team_id(key_or_name: str) -> str | None: + """Map a team key (ENG) or name to UUID.""" + q = "query { teams(first: 100) { nodes { id key name } } }" + teams = gql(q).get("teams", {}).get("nodes", []) + kl = key_or_name.lower() + for t in teams: + if t["key"].lower() == kl or t["name"].lower() == kl: + return t["id"] + return None + + +def cmd_list_projects(args: argparse.Namespace) -> None: + if args.team: + tid = _resolve_team_id(args.team) + if not tid: + sys.stderr.write(f"Team not found: {args.team}\n") + sys.exit(1) + q = """query($id: String!) { + team(id: $id) { projects(first: 100) { nodes { id name description state } } } + }""" + data = gql(q, {"id": tid}) + emit(data.get("team", {}).get("projects", {}).get("nodes", [])) + else: + q = "query { projects(first: 100) { nodes { id name description state } } }" + emit(gql(q).get("projects", {}).get("nodes", [])) + + +def cmd_list_states(args: argparse.Namespace) -> None: + if args.team: + tid = _resolve_team_id(args.team) + if not tid: + sys.stderr.write(f"Team not found: {args.team}\n") + sys.exit(1) + q = """query($id: String!) { + team(id: $id) { states(first: 100) { nodes { id name type color } } } + }""" + emit(gql(q, {"id": tid}).get("team", {}).get("states", {}).get("nodes", [])) + else: + q = "query { workflowStates(first: 200) { nodes { id name type team { key } } } }" + emit(gql(q).get("workflowStates", {}).get("nodes", [])) + + +def cmd_list_issues(args: argparse.Namespace) -> None: + filt: dict[str, Any] = {} + if args.team: + filt["team"] = {"key": {"eq": args.team}} + if args.status: + filt["state"] = {"name": {"eq": args.status}} + if args.assignee: + filt["assignee"] = {"name": {"eq": args.assignee}} + if args.label: + filt["labels"] = {"name": {"eq": args.label}} + + q = """query($filter: IssueFilter, $first: Int!) { + issues(filter: $filter, first: $first, orderBy: updatedAt) { + nodes { + id identifier title + state { name } priority + assignee { name } + team { key } + updatedAt url + } + } + }""" + data = gql(q, {"filter": filt or None, "first": args.limit}) + emit(data.get("issues", {}).get("nodes", [])) + + +def cmd_get_issue(args: argparse.Namespace) -> None: + q = """query($id: String!) { + issue(id: $id) { + id identifier title description + state { name type } + priority priorityLabel + assignee { name email } + creator { name } + team { key name } + project { name } + labels { nodes { name } } + parent { identifier title } + children { nodes { identifier title state { name } } } + comments { nodes { user { name } body createdAt } } + createdAt updatedAt url + } + }""" + emit(gql(q, {"id": args.identifier}).get("issue")) + + +def cmd_search_issues(args: argparse.Namespace) -> None: + q = """query($term: String!, $first: Int!) { + searchIssues(term: $term, first: $first) { + nodes { id identifier title state { name } url } + } + }""" + emit(gql(q, {"term": args.query, "first": args.limit}).get("searchIssues", {}).get("nodes", [])) + + +def cmd_create_issue(args: argparse.Namespace) -> None: + tid = _resolve_team_id(args.team) + if not tid: + sys.stderr.write(f"Team not found: {args.team}\n") + sys.exit(1) + inp: dict[str, Any] = {"title": args.title, "teamId": tid} + if args.description: + inp["description"] = args.description + if args.priority is not None: + inp["priority"] = args.priority + if args.parent: + inp["parentId"] = args.parent + # TODO: label + assignee name->id lookup (omitted for v1 brevity) + + q = """mutation($input: IssueCreateInput!) { + issueCreate(input: $input) { + success issue { id identifier title url } + } + }""" + emit(gql(q, {"input": inp}).get("issueCreate")) + + +def cmd_update_issue(args: argparse.Namespace) -> None: + inp: dict[str, Any] = {} + if args.title: + inp["title"] = args.title + if args.description: + inp["description"] = args.description + if args.priority is not None: + inp["priority"] = args.priority + if not inp: + sys.stderr.write("No update fields provided.\n") + sys.exit(1) + q = """mutation($id: String!, $input: IssueUpdateInput!) { + issueUpdate(id: $id, input: $input) { + success issue { identifier title url } + } + }""" + emit(gql(q, {"id": args.identifier, "input": inp}).get("issueUpdate")) + + +def cmd_update_status(args: argparse.Namespace) -> None: + # Resolve state name -> id within the issue's team + get_q = """query($id: String!) { + issue(id: $id) { team { id states(first: 100) { nodes { id name } } } } + }""" + issue = gql(get_q, {"id": args.identifier}).get("issue") + if not issue: + sys.stderr.write(f"Issue not found: {args.identifier}\n") + sys.exit(1) + sl = args.state.lower() + match = next((s for s in issue["team"]["states"]["nodes"] if s["name"].lower() == sl), None) + if not match: + sys.stderr.write( + f"State '{args.state}' not found. Available: " + f"{[s['name'] for s in issue['team']['states']['nodes']]}\n" + ) + sys.exit(1) + + q = """mutation($id: String!, $stateId: String!) { + issueUpdate(id: $id, input: { stateId: $stateId }) { + success issue { identifier state { name } url } + } + }""" + emit(gql(q, {"id": args.identifier, "stateId": match["id"]}).get("issueUpdate")) + + +def cmd_add_comment(args: argparse.Namespace) -> None: + q = """mutation($input: CommentCreateInput!) { + commentCreate(input: $input) { + success comment { id body createdAt } + } + }""" + emit(gql(q, {"input": {"issueId": args.identifier, "body": args.body}}).get("commentCreate")) + + +# ---- Documents ---- + +def cmd_list_documents(args: argparse.Namespace) -> None: + q = """query($first: Int!) { + documents(first: $first, orderBy: updatedAt) { + nodes { id title slugId updatedAt url project { name } creator { name } } + } + }""" + emit(gql(q, {"first": args.limit}).get("documents", {}).get("nodes", [])) + + +def cmd_get_document(args: argparse.Namespace) -> None: + """Fetch a document by slugId (from URL) OR full UUID. + + Linear document URLs look like: + https://linear.app//document/- + The part we want is the final hex segment (the slugId). + """ + ref = args.ref + # If it looks like a UUID, query by id. Otherwise, assume slugId. + is_uuid = len(ref) == 36 and ref.count("-") == 4 + if is_uuid: + q = """query($id: String!) { + document(id: $id) { + id title content contentState slugId + createdAt updatedAt url + creator { name } project { name } + } + }""" + emit(gql(q, {"id": ref}).get("document")) + else: + # Query the collection and filter by slugId — the doc() query only accepts UUIDs. + q = """query($slug: String!) { + documents(filter: { slugId: { eq: $slug } }, first: 1) { + nodes { + id title content contentState slugId + createdAt updatedAt url + creator { name } project { name } + } + } + }""" + nodes = gql(q, {"slug": ref}).get("documents", {}).get("nodes", []) + emit(nodes[0] if nodes else None) + + +def cmd_search_documents(args: argparse.Namespace) -> None: + # Linear doesn't have a first-class searchDocuments — use title filter as a fallback. + q = """query($term: String!, $first: Int!) { + documents(filter: { title: { containsIgnoreCase: $term } }, first: $first) { + nodes { id title slugId url updatedAt } + } + }""" + emit(gql(q, {"term": args.query, "first": args.limit}).get("documents", {}).get("nodes", [])) + + +def cmd_raw(args: argparse.Namespace) -> None: + variables = json.loads(args.vars) if args.vars else None + emit(gql(args.query, variables)) + + +# ---------- Arg parsing ---------- + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(prog="linear_api.py", description="Linear GraphQL CLI") + sub = p.add_subparsers(dest="cmd", required=True) + + sub.add_parser("whoami").set_defaults(func=cmd_whoami) + sub.add_parser("list-teams").set_defaults(func=cmd_list_teams) + + lp = sub.add_parser("list-projects") + lp.add_argument("--team") + lp.set_defaults(func=cmd_list_projects) + + ls = sub.add_parser("list-states") + ls.add_argument("--team") + ls.set_defaults(func=cmd_list_states) + + li = sub.add_parser("list-issues") + li.add_argument("--team") + li.add_argument("--status") + li.add_argument("--assignee") + li.add_argument("--label") + li.add_argument("--limit", type=int, default=25) + li.set_defaults(func=cmd_list_issues) + + gi = sub.add_parser("get-issue") + gi.add_argument("identifier") + gi.set_defaults(func=cmd_get_issue) + + si = sub.add_parser("search-issues") + si.add_argument("query") + si.add_argument("--limit", type=int, default=25) + si.set_defaults(func=cmd_search_issues) + + ci = sub.add_parser("create-issue") + ci.add_argument("--title", required=True) + ci.add_argument("--team", required=True) + ci.add_argument("--description") + ci.add_argument("--priority", type=int, choices=[0, 1, 2, 3, 4]) + ci.add_argument("--label") + ci.add_argument("--assignee") + ci.add_argument("--parent") + ci.set_defaults(func=cmd_create_issue) + + ui = sub.add_parser("update-issue") + ui.add_argument("identifier") + ui.add_argument("--title") + ui.add_argument("--description") + ui.add_argument("--priority", type=int, choices=[0, 1, 2, 3, 4]) + ui.set_defaults(func=cmd_update_issue) + + us = sub.add_parser("update-status") + us.add_argument("identifier") + us.add_argument("state") + us.set_defaults(func=cmd_update_status) + + ac = sub.add_parser("add-comment") + ac.add_argument("identifier") + ac.add_argument("body") + ac.set_defaults(func=cmd_add_comment) + + ld = sub.add_parser("list-documents") + ld.add_argument("--limit", type=int, default=50) + ld.set_defaults(func=cmd_list_documents) + + gd = sub.add_parser("get-document") + gd.add_argument("ref", help="slugId (hex suffix from URL) or full UUID") + gd.set_defaults(func=cmd_get_document) + + sd = sub.add_parser("search-documents") + sd.add_argument("query") + sd.add_argument("--limit", type=int, default=25) + sd.set_defaults(func=cmd_search_documents) + + r = sub.add_parser("raw") + r.add_argument("query") + r.add_argument("--vars", help="JSON string of variables") + r.set_defaults(func=cmd_raw) + + return p + + +def main(argv: list[str] | None = None) -> None: + parser = build_parser() + args = parser.parse_args(argv) + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/maps/SKILL.md b/maps/SKILL.md new file mode 100644 index 0000000..3c1e8af --- /dev/null +++ b/maps/SKILL.md @@ -0,0 +1,195 @@ +--- +name: maps +description: "Geocode, POIs, routes, timezones via OpenStreetMap/OSRM." +version: 1.2.0 +author: Mibayy +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [maps, geocoding, places, routing, distance, directions, nearby, location, openstreetmap, nominatim, overpass, osrm] + category: productivity + requires_toolsets: [terminal] + supersedes: [find-nearby] +--- + +# Maps Skill + +Location intelligence using free, open data sources. 8 commands, 44 POI +categories, zero dependencies (Python stdlib only), no API key required. + +Data sources: OpenStreetMap/Nominatim, Overpass API, OSRM, TimeAPI.io. + +This skill supersedes the old `find-nearby` skill — all of find-nearby's +functionality is covered by the `nearby` command below, with the same +`--near ""` shortcut and multi-category support. + +## When to Use + +- User sends a Telegram location pin (latitude/longitude in the message) → `nearby` +- User wants coordinates for a place name → `search` +- User has coordinates and wants the address → `reverse` +- User asks for nearby restaurants, hospitals, pharmacies, hotels, etc. → `nearby` +- User wants driving/walking/cycling distance or travel time → `distance` +- User wants turn-by-turn directions between two places → `directions` +- User wants timezone information for a location → `timezone` +- User wants to search for POIs within a geographic area → `area` + `bbox` + +## Prerequisites + +Python 3.8+ (stdlib only — no pip installs needed). + +Script path: `~/.hermes/skills/maps/scripts/maps_client.py` + +## Commands + +```bash +MAPS=~/.hermes/skills/maps/scripts/maps_client.py +``` + +### search — Geocode a place name + +```bash +python3 $MAPS search "Eiffel Tower" +python3 $MAPS search "1600 Pennsylvania Ave, Washington DC" +``` + +Returns: lat, lon, display name, type, bounding box, importance score. + +### reverse — Coordinates to address + +```bash +python3 $MAPS reverse 48.8584 2.2945 +``` + +Returns: full address breakdown (street, city, state, country, postcode). + +### nearby — Find places by category + +```bash +# By coordinates (from a Telegram location pin, for example) +python3 $MAPS nearby 48.8584 2.2945 restaurant --limit 10 +python3 $MAPS nearby 40.7128 -74.0060 hospital --radius 2000 + +# By address / city / zip / landmark — --near auto-geocodes +python3 $MAPS nearby --near "Times Square, New York" --category cafe +python3 $MAPS nearby --near "90210" --category pharmacy + +# Multiple categories merged into one query +python3 $MAPS nearby --near "downtown austin" --category restaurant --category bar --limit 10 +``` + +46 categories: restaurant, cafe, bar, hospital, pharmacy, hotel, guest_house, +camp_site, supermarket, atm, gas_station, parking, museum, park, school, +university, bank, police, fire_station, library, airport, train_station, +bus_stop, church, mosque, synagogue, dentist, doctor, cinema, theatre, gym, +swimming_pool, post_office, convenience_store, bakery, bookshop, laundry, +car_wash, car_rental, bicycle_rental, taxi, veterinary, zoo, playground, +stadium, nightclub. + +Each result includes: `name`, `address`, `lat`/`lon`, `distance_m`, +`maps_url` (clickable Google Maps link), `directions_url` (Google Maps +directions from the search point), and promoted tags when available — +`cuisine`, `hours` (opening_hours), `phone`, `website`. + +### distance — Travel distance and time + +```bash +python3 $MAPS distance "Paris" --to "Lyon" +python3 $MAPS distance "New York" --to "Boston" --mode driving +python3 $MAPS distance "Big Ben" --to "Tower Bridge" --mode walking +``` + +Modes: driving (default), walking, cycling. Returns road distance, duration, +and straight-line distance for comparison. + +### directions — Turn-by-turn navigation + +```bash +python3 $MAPS directions "Eiffel Tower" --to "Louvre Museum" --mode walking +python3 $MAPS directions "JFK Airport" --to "Times Square" --mode driving +``` + +Returns numbered steps with instruction, distance, duration, road name, and +maneuver type (turn, depart, arrive, etc.). + +### timezone — Timezone for coordinates + +```bash +python3 $MAPS timezone 48.8584 2.2945 +python3 $MAPS timezone 35.6762 139.6503 +``` + +Returns timezone name, UTC offset, and current local time. + +### area — Bounding box and area for a place + +```bash +python3 $MAPS area "Manhattan, New York" +python3 $MAPS area "London" +``` + +Returns bounding box coordinates, width/height in km, and approximate area. +Useful as input for the bbox command. + +### bbox — Search within a bounding box + +```bash +python3 $MAPS bbox 40.75 -74.00 40.77 -73.98 restaurant --limit 20 +``` + +Finds POIs within a geographic rectangle. Use `area` first to get the +bounding box coordinates for a named place. + +## Working With Telegram Location Pins + +When a user sends a location pin, the message contains `latitude:` and +`longitude:` fields. Extract those and pass them straight to `nearby`: + +```bash +# User sent a pin at 36.17, -115.14 and asked "find cafes nearby" +python3 $MAPS nearby 36.17 -115.14 cafe --radius 1500 +``` + +Present results as a numbered list with names, distances, and the +`maps_url` field so the user gets a tap-to-open link in chat. For "open +now?" questions, check the `hours` field; if missing or unclear, verify +with `web_search` since OSM hours are community-maintained and not always +current. + +## Workflow Examples + +**"Find Italian restaurants near the Colosseum":** +1. `nearby --near "Colosseum Rome" --category restaurant --radius 500` + — one command, auto-geocoded + +**"What's near this location pin they sent?":** +1. Extract lat/lon from the Telegram message +2. `nearby LAT LON cafe --radius 1500` + +**"How do I walk from hotel to conference center?":** +1. `directions "Hotel Name" --to "Conference Center" --mode walking` + +**"What restaurants are in downtown Seattle?":** +1. `area "Downtown Seattle"` → get bounding box +2. `bbox S W N E restaurant --limit 30` + +## Pitfalls + +- Nominatim ToS: max 1 req/s (handled automatically by the script) +- `nearby` requires lat/lon OR `--near "
"` — one of the two is needed +- OSRM routing coverage is best for Europe and North America +- Overpass API can be slow during peak hours; the script automatically + falls back between mirrors (overpass-api.de → overpass.kumi.systems) +- `distance` and `directions` use `--to` flag for the destination (not positional) +- If a zip code alone gives ambiguous results globally, include country/state + +## Verification + +```bash +python3 ~/.hermes/skills/maps/scripts/maps_client.py search "Statue of Liberty" +# Should return lat ~40.689, lon ~-74.044 + +python3 ~/.hermes/skills/maps/scripts/maps_client.py nearby --near "Times Square" --category restaurant --limit 3 +# Should return a list of restaurants within ~500m of Times Square +``` diff --git a/maps/scripts/maps_client.py b/maps/scripts/maps_client.py new file mode 100644 index 0000000..9076358 --- /dev/null +++ b/maps/scripts/maps_client.py @@ -0,0 +1,1297 @@ +#!/usr/bin/env python3 +""" +maps_client.py - CLI tool for maps, geocoding, routing, POI search, and more. +Uses only Python stdlib. Data from OpenStreetMap/Nominatim, Overpass API, OSRM, +and TimeAPI.io. + +Commands: + search - Geocode a place name to coordinates + reverse - Reverse geocode coordinates to an address + nearby - Find nearby POIs by category + distance - Road distance and travel time between two places + directions - Turn-by-turn directions between two places + timezone - Timezone info for coordinates + bbox - Find POIs within a bounding box + area - Get bounding box and area info for a named place +""" + +import argparse +import json +import math +import sys +import time +import urllib.error +import urllib.parse +import urllib.request + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +USER_AGENT = "HermesAgent/1.0 (contact: hermes@agent.ai)" +DATA_SOURCE = "OpenStreetMap/Nominatim" + +NOMINATIM_SEARCH = "https://nominatim.openstreetmap.org/search" +NOMINATIM_REVERSE = "https://nominatim.openstreetmap.org/reverse" +# Public Overpass endpoints. We try them in order so a single server +# outage doesn't break the skill — kumi.systems is a well-known mirror. +OVERPASS_URLS = [ + "https://overpass-api.de/api/interpreter", + "https://overpass.kumi.systems/api/interpreter", +] +# Backward-compat alias for any caller that imports OVERPASS_API directly. +OVERPASS_API = OVERPASS_URLS[0] +OSRM_BASE = "https://router.project-osrm.org/route/v1" +TIMEAPI_BASE = "https://timeapi.io/api/timezone/coordinate" + +# Seconds to sleep between Nominatim requests (ToS requirement) +NOMINATIM_RATE_LIMIT = 1.0 + +# Maximum retries for HTTP errors +MAX_RETRIES = 3 +RETRY_DELAY = 2.0 # seconds + +# Category -> (OSM tag key, OSM tag value) +CATEGORY_TAGS = { + # Food & Drink + "restaurant": ("amenity", "restaurant"), + "cafe": ("amenity", "cafe"), + "bar": ("amenity", "bar"), + # bakery is tagged as shop=bakery in the OSM wiki, but some mappers use + # amenity=bakery. Search both so small indie bakeries aren't missed. + "bakery": [("shop", "bakery"), ("amenity", "bakery")], + "convenience_store": ("shop", "convenience"), + # Health + "hospital": ("amenity", "hospital"), + "pharmacy": ("amenity", "pharmacy"), + "dentist": ("amenity", "dentist"), + "doctor": ("amenity", "doctors"), + "veterinary": ("amenity", "veterinary"), + # Accommodation + "hotel": ("tourism", "hotel"), + "guest_house": ("tourism", "guest_house"), + "camp_site": ("tourism", "camp_site"), + # Shopping & Services + "supermarket": ("shop", "supermarket"), + "bookshop": ("shop", "books"), + "laundry": ("shop", "laundry"), + # Finance + "atm": ("amenity", "atm"), + "bank": ("amenity", "bank"), + # Transport + "gas_station": ("amenity", "fuel"), + "parking": ("amenity", "parking"), + "airport": ("aeroway", "aerodrome"), + "train_station": ("railway", "station"), + "bus_stop": ("highway", "bus_stop"), + "taxi": ("amenity", "taxi"), + "car_wash": ("amenity", "car_wash"), + "car_rental": ("amenity", "car_rental"), + "bicycle_rental": ("amenity", "bicycle_rental"), + # Culture & Entertainment + "museum": ("tourism", "museum"), + "cinema": ("amenity", "cinema"), + "theatre": ("amenity", "theatre"), + "nightclub": ("amenity", "nightclub"), + "zoo": ("tourism", "zoo"), + # Education + "school": ("amenity", "school"), + "university": ("amenity", "university"), + "library": ("amenity", "library"), + # Public Services + "police": ("amenity", "police"), + "fire_station": ("amenity", "fire_station"), + "post_office": ("amenity", "post_office"), + # Religion + "church": ("amenity", "place_of_worship"), # refined by religion tag + "mosque": ("amenity", "place_of_worship"), + "synagogue": ("amenity", "place_of_worship"), + # Recreation + "park": ("leisure", "park"), + "gym": ("leisure", "fitness_centre"), + "swimming_pool": ("leisure", "swimming_pool"), + "playground": ("leisure", "playground"), + "stadium": ("leisure", "stadium"), +} + +# Religion-specific overrides for place_of_worship categories +RELIGION_FILTER = { + "church": "christian", + "mosque": "muslim", + "synagogue": "jewish", +} + +VALID_CATEGORIES = sorted(CATEGORY_TAGS.keys()) + + +def _tags_for(category): + """Return the CATEGORY_TAGS entry as a list of (key, value) pairs. + + Most categories map to a single (tag_key, tag_val) tuple, but some + (e.g. ``bakery``) are tagged under more than one OSM key and are + represented as a list of tuples. Normalise both forms to a list. + """ + entry = CATEGORY_TAGS[category] + if isinstance(entry, list): + return list(entry) + return [entry] + +OSRM_PROFILES = { + "driving": "driving", + "walking": "foot", + "cycling": "bike", +} + +# --------------------------------------------------------------------------- +# Output helpers +# --------------------------------------------------------------------------- + +def print_json(data): + """Print data as pretty-printed JSON to stdout.""" + print(json.dumps(data, indent=2, ensure_ascii=False)) + + +def error_exit(message, code=1): + """Print an error result as JSON and exit.""" + print_json({"error": message, "status": "error"}) + sys.exit(code) + + +# --------------------------------------------------------------------------- +# HTTP helpers +# --------------------------------------------------------------------------- + +def http_get(url, params=None, retries=MAX_RETRIES, silent=False): + """ + Perform an HTTP GET request, returning parsed JSON. + Adds the required User-Agent header. Retries on transient errors. + If silent=True, raises RuntimeError instead of calling error_exit. + """ + if params: + url = url + "?" + urllib.parse.urlencode(params) + + req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) + + last_error = None + for attempt in range(1, retries + 1): + try: + with urllib.request.urlopen(req, timeout=15) as resp: + raw = resp.read().decode("utf-8") + return json.loads(raw) + except urllib.error.HTTPError as exc: + last_error = f"HTTP {exc.code}: {exc.reason} for {url}" + if exc.code in {429, 503, 502, 504}: + time.sleep(RETRY_DELAY * attempt) + else: + if silent: + raise RuntimeError(last_error) + error_exit(last_error) + except urllib.error.URLError as exc: + last_error = f"URL error: {exc.reason}" + time.sleep(RETRY_DELAY * attempt) + except json.JSONDecodeError as exc: + last_error = f"JSON parse error: {exc}" + time.sleep(RETRY_DELAY * attempt) + + msg = f"Request failed after {retries} attempts. Last error: {last_error}" + if silent: + raise RuntimeError(msg) + error_exit(msg) + + +def http_get_text(url, params=None, retries=MAX_RETRIES, silent=False): + """ + Like http_get but returns raw text instead of parsed JSON. + Useful for APIs that may return non-JSON responses. + """ + if params: + url = url + "?" + urllib.parse.urlencode(params) + + req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT}) + + last_error = None + for attempt in range(1, retries + 1): + try: + with urllib.request.urlopen(req, timeout=15) as resp: + return resp.read().decode("utf-8") + except urllib.error.HTTPError as exc: + last_error = f"HTTP {exc.code}: {exc.reason} for {url}" + if exc.code in {429, 503, 502, 504}: + time.sleep(RETRY_DELAY * attempt) + else: + if silent: + raise RuntimeError(last_error) + error_exit(last_error) + except urllib.error.URLError as exc: + last_error = f"URL error: {exc.reason}" + time.sleep(RETRY_DELAY * attempt) + + msg = f"Request failed after {retries} attempts. Last error: {last_error}" + if silent: + raise RuntimeError(msg) + error_exit(msg) + + +def http_post(url, data_str, retries=MAX_RETRIES): + """ + Perform an HTTP POST with a plain-text body (for Overpass QL). + Returns parsed JSON. + """ + encoded = data_str.encode("utf-8") + req = urllib.request.Request( + url, + data=encoded, + headers={ + "User-Agent": USER_AGENT, + "Content-Type": "application/x-www-form-urlencoded", + }, + ) + + last_error = None + for attempt in range(1, retries + 1): + try: + with urllib.request.urlopen(req, timeout=30) as resp: + raw = resp.read().decode("utf-8") + return json.loads(raw) + except urllib.error.HTTPError as exc: + last_error = f"HTTP {exc.code}: {exc.reason}" + if exc.code in {429, 503, 502, 504}: + time.sleep(RETRY_DELAY * attempt) + else: + error_exit(last_error) + except urllib.error.URLError as exc: + last_error = f"URL error: {exc.reason}" + time.sleep(RETRY_DELAY * attempt) + except json.JSONDecodeError as exc: + last_error = f"JSON parse error: {exc}" + time.sleep(RETRY_DELAY * attempt) + + error_exit(f"POST failed after {retries} attempts. Last error: {last_error}") + + +def overpass_query(query): + """POST an Overpass QL query, trying each URL in OVERPASS_URLS in turn. + + A single public Overpass mirror can be rate-limited or down; trying the + next mirror before giving up turns a flaky outage into a retry. Returns + parsed JSON. Falls through to error_exit if every mirror fails. + """ + post_data = "data=" + urllib.parse.quote(query) + last_error = None + for url in OVERPASS_URLS: + try: + return http_post(url, post_data, retries=1) + except SystemExit: + # error_exit inside http_post — keep trying the next mirror. + last_error = f"mirror {url} exhausted retries" + continue + except Exception as exc: + last_error = f"{url}: {exc}" + continue + error_exit( + f"All Overpass mirrors failed. Last error: {last_error or 'unknown'}" + ) + + +# --------------------------------------------------------------------------- +# Geo math +# --------------------------------------------------------------------------- + +def haversine_m(lat1, lon1, lat2, lon2): + """Return distance in metres between two lat/lon points (Haversine).""" + R = 6_371_000 # Earth mean radius in metres + phi1 = math.radians(lat1) + phi2 = math.radians(lat2) + dphi = math.radians(lat2 - lat1) + dlam = math.radians(lon2 - lon1) + a = (math.sin(dphi / 2) ** 2 + + math.cos(phi1) * math.cos(phi2) * math.sin(dlam / 2) ** 2) + return 2 * R * math.atan2(math.sqrt(a), math.sqrt(1 - a)) + + +# --------------------------------------------------------------------------- +# Nominatim helpers +# --------------------------------------------------------------------------- + +def nominatim_search(query, limit=5): + """Geocode a free-text query. Returns list of result dicts.""" + params = { + "q": query, + "format": "json", + "limit": limit, + "addressdetails": 1, + } + time.sleep(NOMINATIM_RATE_LIMIT) + return http_get(NOMINATIM_SEARCH, params=params) + + +def nominatim_reverse(lat, lon): + """Reverse geocode lat/lon. Returns a single result dict.""" + params = { + "lat": lat, + "lon": lon, + "format": "json", + "addressdetails": 1, + } + time.sleep(NOMINATIM_RATE_LIMIT) + return http_get(NOMINATIM_REVERSE, params=params) + + +def geocode_single(query): + """ + Geocode a query and return (lat, lon, display_name). + Exits with error if nothing found. + """ + results = nominatim_search(query, limit=1) + if not results: + error_exit(f"Could not geocode: {query}") + r = results[0] + return float(r["lat"]), float(r["lon"]), r.get("display_name", query) + + +# --------------------------------------------------------------------------- +# Overpass helpers +# --------------------------------------------------------------------------- + +def build_overpass_nearby(tag_key, tag_val, lat, lon, radius, limit, + religion=None, tag_pairs=None): + """Build an Overpass QL query for nearby POIs around a point. + + If ``tag_pairs`` is provided, the query unions across every + ``(key, value)`` pair (used for categories like ``bakery`` that are + tagged under more than one OSM key). Otherwise falls back to the + single ``tag_key``/``tag_val`` pair for back-compat. + """ + pairs = tag_pairs if tag_pairs else [(tag_key, tag_val)] + religion_filter = "" + if religion: + religion_filter = f'["religion"="{religion}"]' + body_lines = [] + for k, v in pairs: + body_lines.append( + f' node["{k}"="{v}"]{religion_filter}' + f'(around:{radius},{lat},{lon});' + ) + body_lines.append( + f' way["{k}"="{v}"]{religion_filter}' + f'(around:{radius},{lat},{lon});' + ) + body = "\n".join(body_lines) + return ( + f'[out:json][timeout:25];\n' + f'(\n' + f'{body}\n' + f');\n' + f'out center {limit};\n' + ) + + +def build_overpass_bbox(tag_key, tag_val, south, west, north, east, limit, + religion=None, tag_pairs=None): + """Build an Overpass QL query for POIs within a bounding box. + + See ``build_overpass_nearby`` for ``tag_pairs`` semantics. + """ + pairs = tag_pairs if tag_pairs else [(tag_key, tag_val)] + religion_filter = "" + if religion: + religion_filter = f'["religion"="{religion}"]' + body_lines = [] + for k, v in pairs: + body_lines.append( + f' node["{k}"="{v}"]{religion_filter}' + f'({south},{west},{north},{east});' + ) + body_lines.append( + f' way["{k}"="{v}"]{religion_filter}' + f'({south},{west},{north},{east});' + ) + body = "\n".join(body_lines) + return ( + f'[out:json][timeout:25];\n' + f'(\n' + f'{body}\n' + f');\n' + f'out center {limit};\n' + ) + + +def parse_overpass_elements(elements, ref_lat=None, ref_lon=None): + """ + Parse Overpass elements into a clean list of POI dicts. + If ref_lat/ref_lon are provided, computes distance and sorts by it. + """ + places = [] + for el in elements: + # Ways have a "center" sub-dict; nodes have lat/lon directly + if el["type"] == "way": + center = el.get("center", {}) + el_lat = center.get("lat") + el_lon = center.get("lon") + else: + el_lat = el.get("lat") + el_lon = el.get("lon") + + if el_lat is None or el_lon is None: + continue + + tags = el.get("tags", {}) + name = tags.get("name") or tags.get("name:en") or "" + + # Build a short address from available tags + addr_parts = [] + for part_key in ("addr:housenumber", "addr:street", "addr:city"): + val = tags.get(part_key) + if val: + addr_parts.append(val) + address_str = ", ".join(addr_parts) if addr_parts else "" + + place = { + "name": name, + "address": address_str, + "lat": el_lat, + "lon": el_lon, + "osm_type": el.get("type", ""), + "osm_id": el.get("id", ""), + # Clickable Google Maps link so the agent can render a tap-to-open + # URL in chat without composing one downstream. + "maps_url": f"https://www.google.com/maps/search/?api=1&query={el_lat},{el_lon}", + "tags": { + k: v for k, v in tags.items() + if k not in {"name", "name:en", + "addr:housenumber", "addr:street", "addr:city"} + }, + } + + # Promote commonly-useful tags to top-level fields so agents can + # reference them without digging into the raw ``tags`` dict. + for src_key, dst_key in ( + ("cuisine", "cuisine"), + ("opening_hours", "hours"), + ("phone", "phone"), + ("website", "website"), + ): + val = tags.get(src_key) + if val: + place[dst_key] = val + + if ref_lat is not None and ref_lon is not None: + dist_m = haversine_m(ref_lat, ref_lon, el_lat, el_lon) + place["distance_m"] = round(dist_m, 1) + # With a reference point we can also hand back a directions URL. + place["directions_url"] = ( + f"https://www.google.com/maps/dir/?api=1" + f"&origin={ref_lat},{ref_lon}" + f"&destination={el_lat},{el_lon}" + ) + + places.append(place) + + # Sort by distance if available + if places and "distance_m" in places[0]: + places.sort(key=lambda p: p["distance_m"]) + + return places + + +# --------------------------------------------------------------------------- +# Command: search +# --------------------------------------------------------------------------- + +def cmd_search(args): + """Geocode a place name and return top results.""" + query = " ".join(args.query) + raw = nominatim_search(query, limit=5) + + if not raw: + print_json({ + "query": query, + "results": [], + "count": 0, + "data_source": DATA_SOURCE, + }) + return + + results = [] + for item in raw: + bb = item.get("boundingbox", []) + results.append({ + "name": item.get("name") or item.get("display_name", ""), + "display_name": item.get("display_name", ""), + "lat": float(item["lat"]), + "lon": float(item["lon"]), + "type": item.get("type", ""), + "category": item.get("category", ""), + "osm_type": item.get("osm_type", ""), + "osm_id": item.get("osm_id", ""), + "bounding_box": { + "min_lat": float(bb[0]) if len(bb) > 0 else None, + "max_lat": float(bb[1]) if len(bb) > 1 else None, + "min_lon": float(bb[2]) if len(bb) > 2 else None, + "max_lon": float(bb[3]) if len(bb) > 3 else None, + }, + "importance": item.get("importance"), + }) + + print_json({ + "query": query, + "results": results, + "count": len(results), + "data_source": DATA_SOURCE, + }) + + +# --------------------------------------------------------------------------- +# Command: reverse +# --------------------------------------------------------------------------- + +def cmd_reverse(args): + """Reverse geocode coordinates to a human-readable address.""" + try: + lat = float(args.lat) + lon = float(args.lon) + except ValueError: + error_exit("LAT and LON must be numeric values.") + + if not (-90 <= lat <= 90): + error_exit("Latitude must be between -90 and 90.") + if not (-180 <= lon <= 180): + error_exit("Longitude must be between -180 and 180.") + + data = nominatim_reverse(lat, lon) + + if "error" in data: + error_exit(f"Reverse geocode failed: {data['error']}") + + address = data.get("address", {}) + + print_json({ + "lat": lat, + "lon": lon, + "display_name": data.get("display_name", ""), + "address": { + "house_number": address.get("house_number", ""), + "road": address.get("road", ""), + "neighbourhood": address.get("neighbourhood", ""), + "suburb": address.get("suburb", ""), + "city": (address.get("city") + or address.get("town") + or address.get("village", "")), + "county": address.get("county", ""), + "state": address.get("state", ""), + "postcode": address.get("postcode", ""), + "country": address.get("country", ""), + "country_code": address.get("country_code", ""), + }, + "osm_type": data.get("osm_type", ""), + "osm_id": data.get("osm_id", ""), + "data_source": DATA_SOURCE, + }) + + +# --------------------------------------------------------------------------- +# Command: nearby +# --------------------------------------------------------------------------- + +def cmd_nearby(args): + """Find nearby POIs using the Overpass API. + + Accepts either explicit coordinates (``lat``/``lon``) or a free-form + address via ``--near`` (auto-geocoded through Nominatim). Supports + multiple categories in one call — results are merged, deduplicated + by ``osm_type+osm_id``, sorted by distance. + """ + # Resolve the center point. --near takes precedence if provided so the + # agent can ask "cafes near Times Square" in one command without having + # to geocode first. + if getattr(args, "near", None): + near_query = " ".join(args.near).strip() if isinstance(args.near, list) else str(args.near).strip() + if not near_query: + error_exit("--near must be a non-empty address or place name.") + lat, lon, _ = geocode_single(near_query) + else: + try: + lat = float(args.lat) + lon = float(args.lon) + except (TypeError, ValueError): + error_exit("Provide numeric LAT and LON, or use --near \"
\".") + + # Categories: support both legacy single positional ``category`` and the + # new repeatable ``--category`` flag. Users can ask for multiple place + # types in one query. + categories = [] + if getattr(args, "category_list", None): + categories.extend(args.category_list) + if getattr(args, "category", None): + categories.append(args.category) + # Deduplicate, preserve order, lower-case. + categories = list(dict.fromkeys(c.lower() for c in categories if c)) + if not categories: + error_exit("Provide at least one category (positional or --category).") + unknown = [c for c in categories if c not in CATEGORY_TAGS] + if unknown: + error_exit( + f"Unknown categor{'ies' if len(unknown) > 1 else 'y'} " + f"{', '.join(repr(c) for c in unknown)}. " + f"Valid categories: {', '.join(VALID_CATEGORIES)}" + ) + + radius = int(args.radius) + limit = int(args.limit) + if radius <= 0: + error_exit("Radius must be a positive integer (metres).") + if limit <= 0: + error_exit("Limit must be a positive integer.") + + # Query each category against the Overpass fallback chain, merge results, + # dedupe by OSM identity so POIs tagged under multiple categories don't + # appear twice. + merged = {} + for category in categories: + tag_pairs = _tags_for(category) + religion = RELIGION_FILTER.get(category) + query = build_overpass_nearby(None, None, lat, lon, radius, limit, + religion=religion, tag_pairs=tag_pairs) + raw = overpass_query(query) + elements = raw.get("elements", []) + for place in parse_overpass_elements(elements, ref_lat=lat, ref_lon=lon): + place["category"] = category + key = (place.get("osm_type", ""), place.get("osm_id", "")) + # Prefer the entry that actually has a distance_m attached (first + # pass through the ref_lat/ref_lon branch), then first-seen wins. + if key not in merged: + merged[key] = place + + # Sort merged by distance when we have ref lat/lon, then cap at ``limit``. + places = sorted( + merged.values(), + key=lambda p: p.get("distance_m", float("inf")), + )[:limit] + + print_json({ + "center_lat": lat, + "center_lon": lon, + "categories": categories, + "radius_m": radius, + "count": len(places), + "results": places, + "data_source": DATA_SOURCE, + }) + + +# --------------------------------------------------------------------------- +# Command: distance +# --------------------------------------------------------------------------- + +def cmd_distance(args): + """Calculate road distance and travel time between two places.""" + origin_query = " ".join(args.origin) + destination_query = " ".join(args.to) + mode = args.mode.lower() + + if mode not in OSRM_PROFILES: + error_exit(f"Invalid mode '{mode}'. Choose from: {', '.join(OSRM_PROFILES)}") + + # Geocode origin and destination + o_lat, o_lon, o_name = geocode_single(origin_query) + d_lat, d_lon, d_name = geocode_single(destination_query) + + profile = OSRM_PROFILES[mode] + url = ( + f"{OSRM_BASE}/{profile}/" + f"{o_lon},{o_lat};{d_lon},{d_lat}" + f"?overview=false&steps=false" + ) + + osrm_data = http_get(url) + + if osrm_data.get("code") != "Ok": + error_exit( + f"OSRM routing failed: " + f"{osrm_data.get('message', osrm_data.get('code', 'unknown error'))}" + ) + + routes = osrm_data.get("routes", []) + if not routes: + error_exit("No route found between the two locations.") + + route = routes[0] + distance_m = route.get("distance", 0) + duration_s = route.get("duration", 0) + distance_km = round(distance_m / 1000, 3) + duration_min = round(duration_s / 60, 2) + + # Straight-line distance for reference + straight_m = haversine_m(o_lat, o_lon, d_lat, d_lon) + + print_json({ + "origin": { + "query": origin_query, + "display_name": o_name, + "lat": o_lat, + "lon": o_lon, + }, + "destination": { + "query": destination_query, + "display_name": d_name, + "lat": d_lat, + "lon": d_lon, + }, + "mode": mode, + "distance_km": distance_km, + "distance_m": round(distance_m, 1), + "duration_minutes": duration_min, + "duration_seconds": round(duration_s, 1), + "straight_line_km": round(straight_m / 1000, 3), + "data_source": DATA_SOURCE, + }) + + +# --------------------------------------------------------------------------- +# Command: directions +# --------------------------------------------------------------------------- + +def _format_duration(seconds): + """Format seconds into a human-readable string.""" + if seconds < 60: + return f"{round(seconds)}s" + minutes = seconds / 60 + if minutes < 60: + return f"{round(minutes, 1)} min" + hours = int(minutes // 60) + remaining = round(minutes % 60) + return f"{hours}h {remaining}min" + + +def _format_distance(metres): + """Format metres into a human-readable string.""" + if metres < 1000: + return f"{round(metres)} m" + return f"{round(metres / 1000, 2)} km" + + +def cmd_directions(args): + """Get turn-by-turn directions between two places via OSRM.""" + origin_query = " ".join(args.origin) + destination_query = " ".join(args.to) + mode = args.mode.lower() + + if mode not in OSRM_PROFILES: + error_exit(f"Invalid mode '{mode}'. Choose from: {', '.join(OSRM_PROFILES)}") + + # Geocode origin and destination + o_lat, o_lon, o_name = geocode_single(origin_query) + d_lat, d_lon, d_name = geocode_single(destination_query) + + profile = OSRM_PROFILES[mode] + url = ( + f"{OSRM_BASE}/{profile}/" + f"{o_lon},{o_lat};{d_lon},{d_lat}" + f"?overview=false&steps=true" + ) + + osrm_data = http_get(url) + + if osrm_data.get("code") != "Ok": + error_exit( + f"OSRM routing failed: " + f"{osrm_data.get('message', osrm_data.get('code', 'unknown error'))}" + ) + + routes = osrm_data.get("routes", []) + if not routes: + error_exit("No route found between the two locations.") + + route = routes[0] + distance_m = route.get("distance", 0) + duration_s = route.get("duration", 0) + + # Extract steps from all legs + steps = [] + step_num = 0 + for leg in route.get("legs", []): + for step in leg.get("steps", []): + maneuver = step.get("maneuver", {}) + step_dist = step.get("distance", 0) + step_dur = step.get("duration", 0) + step_name = step.get("name", "") + modifier = maneuver.get("modifier", "") + m_type = maneuver.get("type", "") + + # Build instruction text + if m_type == "depart": + instruction = f"Depart on {step_name}" if step_name else "Depart" + elif m_type == "arrive": + instruction = "Arrive at destination" + elif m_type == "turn": + instruction = f"Turn {modifier} onto {step_name}" if step_name else f"Turn {modifier}" + elif m_type == "new name": + instruction = f"Continue onto {step_name}" if step_name else "Continue" + elif m_type == "merge": + instruction = f"Merge {modifier} onto {step_name}" if step_name else f"Merge {modifier}" + elif m_type == "fork": + instruction = f"Take the {modifier} fork onto {step_name}" if step_name else f"Take the {modifier} fork" + elif m_type == "roundabout": + instruction = f"Enter roundabout, exit onto {step_name}" if step_name else "Enter roundabout" + elif m_type == "rotary": + instruction = f"Enter rotary, exit onto {step_name}" if step_name else "Enter rotary" + elif m_type == "end of road": + instruction = f"At end of road, turn {modifier} onto {step_name}" if step_name else f"At end of road, turn {modifier}" + elif m_type == "continue": + instruction = f"Continue {modifier} on {step_name}" if step_name else f"Continue {modifier}" + elif m_type == "on ramp": + instruction = f"Take ramp onto {step_name}" if step_name else "Take ramp" + elif m_type == "off ramp": + instruction = f"Take exit onto {step_name}" if step_name else "Take exit" + else: + instruction = f"{m_type} {modifier} {step_name}".strip() + + step_num += 1 + steps.append({ + "step": step_num, + "instruction": instruction, + "distance": _format_distance(step_dist), + "distance_m": round(step_dist, 1), + "duration": _format_duration(step_dur), + "duration_s": round(step_dur, 1), + "road_name": step_name, + "maneuver": m_type, + }) + + print_json({ + "origin": { + "query": origin_query, + "display_name": o_name, + "lat": o_lat, + "lon": o_lon, + }, + "destination": { + "query": destination_query, + "display_name": d_name, + "lat": d_lat, + "lon": d_lon, + }, + "mode": mode, + "total_distance": _format_distance(distance_m), + "total_distance_m": round(distance_m, 1), + "total_duration": _format_duration(duration_s), + "total_duration_s": round(duration_s, 1), + "steps": steps, + "step_count": len(steps), + "data_source": DATA_SOURCE, + }) + + +# --------------------------------------------------------------------------- +# Command: timezone +# --------------------------------------------------------------------------- + +def cmd_timezone(args): + """ + Get timezone information for a lat/lon coordinate. + + Strategy: + 1. Try TimeAPI.io (free, no key, supports coordinate-based lookup). + 2. Fallback: derive UTC offset approximation from longitude. + """ + try: + lat = float(args.lat) + lon = float(args.lon) + except ValueError: + error_exit("LAT and LON must be numeric values.") + + if not (-90 <= lat <= 90): + error_exit("Latitude must be between -90 and 90.") + if not (-180 <= lon <= 180): + error_exit("Longitude must be between -180 and 180.") + + timezone_str = None + timezone_src = None + current_time = None + utc_offset = None + + # --- Strategy 1: TimeAPI.io coordinate lookup --- + try: + params = {"latitude": lat, "longitude": lon} + tz_data = http_get(TIMEAPI_BASE, params=params, silent=True) + if isinstance(tz_data, dict): + timezone_str = tz_data.get("timeZone") + current_time = tz_data.get("currentLocalTime") + # Build utc_offset from currentUtcOffset if available + offset_info = tz_data.get("currentUtcOffset", {}) + if isinstance(offset_info, dict): + oh = offset_info.get("hours", 0) + om = abs(offset_info.get("minutes", 0)) + os_ = offset_info.get("seconds", 0) + sign = "+" if oh >= 0 else "-" + utc_offset = f"{sign}{abs(oh):02d}:{om:02d}" + if os_: + utc_offset = f"{utc_offset}:{os_:02d}" + elif tz_data.get("standardUtcOffset"): + offset_info2 = tz_data["standardUtcOffset"] + if isinstance(offset_info2, dict): + oh = offset_info2.get("hours", 0) + om = abs(offset_info2.get("minutes", 0)) + os_ = offset_info2.get("seconds", 0) + sign = "+" if oh >= 0 else "-" + utc_offset = f"{sign}{abs(oh):02d}:{om:02d}" + if os_: + utc_offset = f"{utc_offset}:{os_:02d}" + timezone_src = "timeapi.io" + except (RuntimeError, KeyError, TypeError): + pass # API may be down; continue to fallback + + # --- Strategy 2: longitude-based UTC offset approximation --- + if not timezone_str: + approx_offset_h = round(lon / 15) + if approx_offset_h >= 0: + utc_offset = f"+{approx_offset_h:02d}:00" + else: + utc_offset = f"-{abs(approx_offset_h):02d}:00" + timezone_str = f"UTC{utc_offset}" + timezone_src = "longitude approximation (longitude/15)" + + print_json({ + "lat": lat, + "lon": lon, + "timezone": timezone_str, + "utc_offset": utc_offset, + "current_time": current_time, + "source": timezone_src, + "data_source": DATA_SOURCE, + }) + + +# --------------------------------------------------------------------------- +# Command: bbox +# --------------------------------------------------------------------------- + +def cmd_bbox(args): + """Find POIs within a bounding box using the Overpass API.""" + try: + lat1 = float(args.lat1) + lon1 = float(args.lon1) + lat2 = float(args.lat2) + lon2 = float(args.lon2) + except ValueError: + error_exit("All coordinate arguments must be numeric values.") + + # Normalize: south/west < north/east + south = min(lat1, lat2) + north = max(lat1, lat2) + west = min(lon1, lon2) + east = max(lon1, lon2) + + category = args.category.lower() + if category not in CATEGORY_TAGS: + error_exit( + f"Unknown category '{category}'. " + f"Valid categories: {', '.join(VALID_CATEGORIES)}" + ) + + limit = int(args.limit) + if limit <= 0: + error_exit("Limit must be a positive integer.") + + tag_pairs = _tags_for(category) + religion = RELIGION_FILTER.get(category) + query = build_overpass_bbox(None, None, south, west, north, east, + limit, religion=religion, tag_pairs=tag_pairs) + + raw = overpass_query(query) + + elements = raw.get("elements", []) + + # Use center of bbox as reference for distance sorting + center_lat = (south + north) / 2 + center_lon = (west + east) / 2 + places = parse_overpass_elements(elements, ref_lat=center_lat, + ref_lon=center_lon) + + for p in places: + p["category"] = category + + print_json({ + "bounding_box": { + "south": south, + "west": west, + "north": north, + "east": east, + }, + "category": category, + "count": len(places), + "results": places, + "data_source": DATA_SOURCE, + }) + + +# --------------------------------------------------------------------------- +# Command: area +# --------------------------------------------------------------------------- + +def cmd_area(args): + """Get bounding box and area info for a named place.""" + query = " ".join(args.place) + raw = nominatim_search(query, limit=1) + + if not raw: + error_exit(f"Could not find place: {query}") + + item = raw[0] + bb = item.get("boundingbox", []) + + if len(bb) < 4: + error_exit(f"No bounding box data available for: {query}") + + min_lat = float(bb[0]) + max_lat = float(bb[1]) + min_lon = float(bb[2]) + max_lon = float(bb[3]) + + # Approximate area in km² using the bounding box + # Width in km at the average latitude + avg_lat = (min_lat + max_lat) / 2 + height_km = haversine_m(min_lat, min_lon, max_lat, min_lon) / 1000 + width_km = haversine_m(avg_lat, min_lon, avg_lat, max_lon) / 1000 + approx_area_km2 = round(height_km * width_km, 3) + + print_json({ + "query": query, + "display_name": item.get("display_name", ""), + "lat": float(item["lat"]), + "lon": float(item["lon"]), + "type": item.get("type", ""), + "category": item.get("category", ""), + "bounding_box": { + "south": min_lat, + "north": max_lat, + "west": min_lon, + "east": max_lon, + }, + "dimensions": { + "width_km": round(width_km, 3), + "height_km": round(height_km, 3), + }, + "approx_area_km2": approx_area_km2, + "osm_type": item.get("osm_type", ""), + "osm_id": item.get("osm_id", ""), + "data_source": DATA_SOURCE, + }) + + +# --------------------------------------------------------------------------- +# CLI setup +# --------------------------------------------------------------------------- + +def build_parser(): + parser = argparse.ArgumentParser( + prog="maps_client.py", + description=( + "CLI maps tool: geocoding, reverse geocoding, POI search, " + "routing, directions, timezone, and area lookup. " + "Powered by OpenStreetMap, OSRM, Overpass, and TimeAPI.io. " + "No API keys required." + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=( + "Examples:\n" + " maps_client.py search Times Square\n" + " maps_client.py reverse 40.758 -73.985\n" + " maps_client.py nearby 40.758 -73.985 restaurant --radius 800\n" + " maps_client.py distance New York --to Los Angeles --mode driving\n" + " maps_client.py directions Paris --to Berlin --mode driving\n" + " maps_client.py timezone 48.8566 2.3522\n" + " maps_client.py bbox 40.70 -74.02 40.78 -73.95 restaurant\n" + " maps_client.py area Manhattan" + ), + ) + sub = parser.add_subparsers(dest="command", required=True, + metavar="COMMAND") + + # -- search -- + p_search = sub.add_parser( + "search", + help="Geocode a place name to coordinates.", + description="Search for a place by name and return coordinates and details.", + ) + p_search.add_argument( + "query", nargs="+", + help="Place name or address to search.", + ) + + # -- reverse -- + p_reverse = sub.add_parser( + "reverse", + help="Reverse geocode coordinates to an address.", + description="Convert latitude/longitude coordinates to a human-readable address.", + ) + p_reverse.add_argument("lat", help="Latitude (decimal degrees).") + p_reverse.add_argument("lon", help="Longitude (decimal degrees).") + + # -- nearby -- + p_nearby = sub.add_parser( + "nearby", + help="Find nearby places of a given category.", + description=( + "Find points of interest near a location using the Overpass API.\n" + "Provide either LAT/LON, or use --near \"
\" to auto-geocode.\n" + "Categories can be specified positionally OR repeated via --category\n" + "to merge multiple types in one query (e.g. --category bar --category cafe).\n" + f"Categories: {', '.join(VALID_CATEGORIES)}" + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p_nearby.add_argument( + "lat", nargs="?", default=None, + help="Center latitude (decimal degrees). Omit if using --near.", + ) + p_nearby.add_argument( + "lon", nargs="?", default=None, + help="Center longitude (decimal degrees). Omit if using --near.", + ) + p_nearby.add_argument( + "category", nargs="?", default=None, + help="POI category (use --help for full list). Omit if using --category flags.", + ) + p_nearby.add_argument( + "--near", nargs="+", metavar="PLACE", + help="Address, city, or landmark to search around (geocoded via Nominatim).", + ) + p_nearby.add_argument( + "--category", action="append", dest="category_list", default=[], + metavar="CAT", + help="POI category (repeatable — adds a type to the search).", + ) + p_nearby.add_argument( + "--radius", "-r", + default=500, type=int, metavar="METRES", + help="Search radius in metres (default: 500).", + ) + p_nearby.add_argument( + "--limit", "-n", + default=10, type=int, metavar="N", + help="Maximum number of results (default: 10).", + ) + + # -- distance -- + p_dist = sub.add_parser( + "distance", + help="Calculate road distance and travel time.", + description=( + "Calculate road distance and estimated travel time between two places.\n" + "Example: maps_client.py distance New York --to Los Angeles" + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p_dist.add_argument( + "origin", nargs="+", + help="Origin address or place name.", + ) + p_dist.add_argument( + "--to", nargs="+", required=True, metavar="DEST", + help="Destination address or place name (required).", + ) + p_dist.add_argument( + "--mode", "-m", + default="driving", + choices=list(OSRM_PROFILES.keys()), + help="Travel mode (default: driving).", + ) + + # -- directions -- + p_dir = sub.add_parser( + "directions", + help="Get turn-by-turn directions between two places.", + description=( + "Get step-by-step navigation directions between two places.\n" + "Example: maps_client.py directions Paris --to Berlin --mode driving" + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p_dir.add_argument( + "origin", nargs="+", + help="Origin address or place name.", + ) + p_dir.add_argument( + "--to", nargs="+", required=True, metavar="DEST", + help="Destination address or place name (required).", + ) + p_dir.add_argument( + "--mode", "-m", + default="driving", + choices=list(OSRM_PROFILES.keys()), + help="Travel mode (default: driving).", + ) + + # -- timezone -- + p_tz = sub.add_parser( + "timezone", + help="Get timezone information for coordinates.", + description="Look up timezone and current local time for a lat/lon coordinate.", + ) + p_tz.add_argument("lat", help="Latitude (decimal degrees).") + p_tz.add_argument("lon", help="Longitude (decimal degrees).") + + # -- bbox -- + p_bbox = sub.add_parser( + "bbox", + help="Find POIs within a bounding box.", + description=( + "Search for points of interest within a geographic bounding box.\n" + "Tip: use the 'area' command to find bounding boxes for named places.\n" + f"Categories: {', '.join(VALID_CATEGORIES)}" + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p_bbox.add_argument("lat1", help="First corner latitude.") + p_bbox.add_argument("lon1", help="First corner longitude.") + p_bbox.add_argument("lat2", help="Second corner latitude.") + p_bbox.add_argument("lon2", help="Second corner longitude.") + p_bbox.add_argument("category", help="POI category to search for.") + p_bbox.add_argument( + "--limit", "-n", + default=20, type=int, metavar="N", + help="Maximum number of results (default: 20).", + ) + + # -- area -- + p_area = sub.add_parser( + "area", + help="Get bounding box and area info for a named place.", + description=( + "Look up a place by name and return its bounding box, dimensions, " + "and approximate area. Useful as input to the 'bbox' command." + ), + ) + p_area.add_argument( + "place", nargs="+", + help="Place name to look up (e.g., 'Manhattan' or 'downtown Seattle').", + ) + + return parser + + +def main(): + parser = build_parser() + args = parser.parse_args() + + dispatch = { + "search": cmd_search, + "reverse": cmd_reverse, + "nearby": cmd_nearby, + "distance": cmd_distance, + "directions": cmd_directions, + "timezone": cmd_timezone, + "bbox": cmd_bbox, + "area": cmd_area, + } + + handler = dispatch.get(args.command) + if handler is None: + error_exit(f"Unknown command: {args.command}") + + handler(args) + + +if __name__ == "__main__": + main() diff --git a/nano-pdf/SKILL.md b/nano-pdf/SKILL.md new file mode 100644 index 0000000..a8022c8 --- /dev/null +++ b/nano-pdf/SKILL.md @@ -0,0 +1,53 @@ +--- +name: nano-pdf +description: "Edit text in existing PDFs via natural-language prompts." +version: 1.0.0 +author: community +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [PDF, Documents, Editing, NLP, Productivity] + homepage: https://pypi.org/project/nano-pdf/ + related_skills: [pdf, ocr-and-documents] +--- + +# nano-pdf + +Edit PDFs using natural-language instructions. Point it at a page and describe what to change. For structural PDF work (merge, split, forms, watermarks, creation), see the `pdf` skill; for text extraction from scans, see `ocr-and-documents`. + +## Prerequisites + +```bash +# Install with uv (recommended — already available in Hermes) +uv pip install nano-pdf + +# Or with pip +pip install nano-pdf +``` + +## Usage + +```bash +nano-pdf edit "" +``` + +## Examples + +```bash +# Change a title on page 1 +nano-pdf edit deck.pdf 1 "Change the title to 'Q3 Results' and fix the typo in the subtitle" + +# Update a date on a specific page +nano-pdf edit report.pdf 3 "Update the date from January to February 2026" + +# Fix content +nano-pdf edit contract.pdf 2 "Change the client name from 'Acme Corp' to 'Acme Industries'" +``` + +## Notes + +- Page numbers may be 0-based or 1-based depending on version — if the edit hits the wrong page, retry with ±1 +- Always verify the output PDF after editing (use `read_file` to check file size, or open it) +- The tool uses an LLM under the hood — requires an API key (check `nano-pdf --help` for config) +- Works well for text changes; complex layout modifications may need a different approach diff --git a/notion/SKILL.md b/notion/SKILL.md new file mode 100644 index 0000000..22010c6 --- /dev/null +++ b/notion/SKILL.md @@ -0,0 +1,448 @@ +--- +name: notion +description: "Notion API + ntn CLI: pages, databases, markdown, Workers." +version: 2.0.0 +author: community +license: MIT +platforms: [linux, macos, windows] +prerequisites: + env_vars: [NOTION_API_KEY] +metadata: + hermes: + tags: [Notion, Productivity, Notes, Database, API, CLI, Workers] + homepage: https://developers.notion.com +--- + +# Notion + +Talk to Notion two ways. Same integration token works for both — pick by what's available. + +◆ **`ntn` CLI** — Notion's official CLI. Shorter syntax, one-line file uploads, required for Workers. macOS + Linux only as of May 2026 (Windows support "coming soon"). **Default when installed.** +◆ **HTTP + curl** — works everywhere including Windows. **Default fallback** when `ntn` isn't installed. + +## Setup + +### 1. Get an integration token (required for both paths) + +1. Create an integration at https://notion.so/my-integrations +2. Copy the API key (starts with `ntn_` or `secret_`) +3. Store in `${HERMES_HOME:-~/.hermes}/.env`: + ``` + NOTION_API_KEY=ntn_your_key_here + ``` +4. **Share target pages/databases with the integration** in Notion: page menu `...` → `Connect to` → your integration name. Without this, the API returns 404 for that page even though it exists. + +### 2. Install `ntn` (preferred path on macOS / Linux) + +```bash +# Recommended +curl -fsSL https://ntn.dev | bash + +# Or via npm (needs Node 22+, npm 10+) +npm install --global ntn + +ntn --version # verify +``` + +**Skip `ntn login` — use the integration token instead.** This works headlessly, no browser needed: +```bash +export NOTION_API_TOKEN=$NOTION_API_KEY # ntn reads NOTION_API_TOKEN +export NOTION_KEYRING=0 # don't try to use the OS keychain +``` + +Add those exports to your shell profile (or to `${HERMES_HOME:-~/.hermes}/.env`) so every session inherits them. + +### 3. Choose path at runtime + +```bash +if command -v ntn >/dev/null 2>&1; then + # use ntn +else + # fall back to curl +fi +``` + +Windows users: skip step 2 entirely until native `ntn` ships — Path B works fine. If you want CLI ergonomics now, install `ntn` inside WSL2. + +## API Basics + +`Notion-Version: 2025-09-03` is required on all HTTP requests. `ntn` handles this for you. In this version, what users call "databases" are called **data sources** in the API. + +## Path A — `ntn` CLI (preferred, macOS / Linux) + +### Raw API calls (shorthand for curl) +```bash +ntn api v1/users # GET +ntn api v1/pages parent[page_id]=abc123 \ # POST with inline body + properties[title][0][text][content]="Notes" +ntn api v1/pages/abc123 -X PATCH archived:=true # PATCH; := is non-string (bool/num/null) +``` + +Syntax notes: +- `key=value` — string fields +- `key[nested]=value` — nested object fields +- `key:=value` — typed assignment (booleans, numbers, null, arrays) + +### Search +```bash +ntn api v1/search query="page title" +``` + +### Read page metadata +```bash +ntn api v1/pages/{page_id} +``` + +### Read page as Markdown (agent-friendly) +```bash +ntn api v1/pages/{page_id}/markdown +``` + +### Read page content as blocks +```bash +ntn api v1/blocks/{page_id}/children +``` + +### Create page from Markdown +```bash +ntn api v1/pages \ + parent[page_id]=xxx \ + properties[title][0][text][content]="Notes from meeting" \ + markdown="# Agenda + +- Q3 roadmap +- Hiring" +``` + +### Patch a page with Markdown +```bash +ntn api v1/pages/{page_id}/markdown -X PATCH \ + markdown="## Update + +Shipped the prototype." +``` + +### Query a database (data source) +```bash +ntn api v1/data_sources/{data_source_id}/query -X POST \ + filter[property]=Status filter[select][equals]=Active +``` + +For complex queries with `sorts`, multiple filter clauses, or compound logic, pipe JSON in: +```bash +echo '{"filter": {"property": "Status", "select": {"equals": "Active"}}, "sorts": [{"property": "Date", "direction": "descending"}]}' | \ + ntn api v1/data_sources/{data_source_id}/query -X POST --json - +``` + +### File uploads (one-liner — biggest CLI win) +```bash +ntn files create < photo.png +ntn files create --external-url https://example.com/photo.png +ntn files list +``` + +Compare to the 3-step HTTP flow (create upload → PUT bytes → reference). + +### Useful env vars +| Var | Effect | +|---|---| +| `NOTION_API_TOKEN` | Auth token (overrides keychain) — set this to your integration token | +| `NOTION_KEYRING=0` | File-based creds at `~/.config/notion/auth.json` instead of OS keychain | +| `NOTION_WORKSPACE_ID` | Skip the workspace picker prompt | + +## Path B — HTTP + curl (cross-platform, default on Windows) + +All requests share this pattern: + +```bash +curl -s -X GET "https://api.notion.com/v1/..." \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" +``` + +On Windows the `curl` shipped with Windows 10+ works as-is. PowerShell users can also use `Invoke-RestMethod`. + +### Search +```bash +curl -s -X POST "https://api.notion.com/v1/search" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{"query": "page title"}' +``` + +### Read page metadata +```bash +curl -s "https://api.notion.com/v1/pages/{page_id}" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" +``` + +### Read page as Markdown (agent-friendly) + +Easier to feed to a model than block JSON. + +```bash +curl -s "https://api.notion.com/v1/pages/{page_id}/markdown" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" +``` + +### Read page content as blocks (when you need structure) +```bash +curl -s "https://api.notion.com/v1/blocks/{page_id}/children" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" +``` + +### Create page from Markdown + +`POST /v1/pages` accepts a `markdown` body param. + +```bash +curl -s -X POST "https://api.notion.com/v1/pages" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{ + "parent": {"page_id": "xxx"}, + "properties": {"title": [{"text": {"content": "Notes from meeting"}}]}, + "markdown": "# Agenda\n\n- Q3 roadmap\n- Hiring\n\n## Decisions\n- Ship MVP Friday" + }' +``` + +### Patch a page with Markdown +```bash +curl -s -X PATCH "https://api.notion.com/v1/pages/{page_id}/markdown" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{"markdown": "## Update\n\nShipped the prototype."}' +``` + +### Create page in a database (typed properties) +```bash +curl -s -X POST "https://api.notion.com/v1/pages" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{ + "parent": {"database_id": "xxx"}, + "properties": { + "Name": {"title": [{"text": {"content": "New Item"}}]}, + "Status": {"select": {"name": "Todo"}} + } + }' +``` + +### Query a database (data source) +```bash +curl -s -X POST "https://api.notion.com/v1/data_sources/{data_source_id}/query" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{ + "filter": {"property": "Status", "select": {"equals": "Active"}}, + "sorts": [{"property": "Date", "direction": "descending"}] + }' +``` + +### Create a database +```bash +curl -s -X POST "https://api.notion.com/v1/data_sources" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{ + "parent": {"page_id": "xxx"}, + "title": [{"text": {"content": "My Database"}}], + "properties": { + "Name": {"title": {}}, + "Status": {"select": {"options": [{"name": "Todo"}, {"name": "Done"}]}}, + "Date": {"date": {}} + } + }' +``` + +### Update page properties +```bash +curl -s -X PATCH "https://api.notion.com/v1/pages/{page_id}" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{"properties": {"Status": {"select": {"name": "Done"}}}}' +``` + +### Append blocks to a page +```bash +curl -s -X PATCH "https://api.notion.com/v1/blocks/{page_id}/children" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{ + "children": [ + {"object": "block", "type": "paragraph", "paragraph": {"rich_text": [{"text": {"content": "Hello from Hermes!"}}]}} + ] + }' +``` + +### File uploads (3-step flow) +```bash +# 1. Create upload +curl -s -X POST "https://api.notion.com/v1/file_uploads" \ + -H "Authorization: Bearer $NOTION_API_KEY" \ + -H "Notion-Version: 2025-09-03" \ + -H "Content-Type: application/json" \ + -d '{"filename": "photo.png", "content_type": "image/png"}' + +# 2. PUT bytes to the upload_url returned above +curl -s -X PUT "{upload_url}" --data-binary @photo.png + +# 3. Reference {file_upload_id} in a page/block payload +``` + +## Property Types + +Common property formats for database items: + +- **Title:** `{"title": [{"text": {"content": "..."}}]}` +- **Rich text:** `{"rich_text": [{"text": {"content": "..."}}]}` +- **Select:** `{"select": {"name": "Option"}}` +- **Multi-select:** `{"multi_select": [{"name": "A"}, {"name": "B"}]}` +- **Date:** `{"date": {"start": "2026-01-15", "end": "2026-01-16"}}` +- **Checkbox:** `{"checkbox": true}` +- **Number:** `{"number": 42}` +- **URL:** `{"url": "https://..."}` +- **Email:** `{"email": "user@example.com"}` +- **Relation:** `{"relation": [{"id": "page_id"}]}` + +## API Version 2025-09-03 — Databases vs Data Sources + +- **Databases became data sources.** Use `/data_sources/` endpoints for queries and retrieval. +- **Two IDs per database:** `database_id` and `data_source_id`. + - `database_id` when creating pages: `parent: {"database_id": "..."}` + - `data_source_id` when querying: `POST /v1/data_sources/{id}/query` +- Search returns databases as `"object": "data_source"` with the `data_source_id` field. + +## Notion Workers (advanced, requires `ntn`) + +Workers are TypeScript programs Notion hosts for you. One worker can expose any combination of: +- **Syncs** — pull data from external APIs into a Notion database on a schedule (default 30 min). +- **Tools** — appear as callable tools inside Notion's Custom Agents. +- **Webhooks** — receive HTTP events from external services (GitHub, Stripe, etc.) and act in Notion. + +**Plan / platform gating:** +- CLI works on all plans. **Deploying Workers requires Business or Enterprise.** +- `ntn` is macOS/Linux only as of May 2026. Windows users need WSL2 or to wait for native support. +- Free through August 11, 2026; metered on Notion credits after. + +### Minimal Worker + +```bash +ntn workers new my-worker # scaffold +cd my-worker +# Edit src/index.ts +ntn workers deploy --name my-worker +``` + +`src/index.ts`: +```typescript +import { Worker } from "@notionhq/workers"; + +const worker = new Worker(); +export default worker; + +worker.tool("greet", { + title: "Greet a User", + description: "Returns a friendly greeting", + inputSchema: { type: "object", properties: { name: { type: "string" } }, required: ["name"] }, + execute: async ({ name }) => `Hello, ${name}!`, +}); +``` + +### Webhook capability + +```typescript +worker.webhook("onGithubPush", { + title: "GitHub Push Handler", + execute: async (events, { notion }) => { + for (const event of events) { + // event.body, event.rawBody (for signature verification), event.headers + console.log("got delivery", event.deliveryId); + } + }, +}); +``` + +After deploy: `ntn workers webhooks list` shows the URL Notion generates. Treat that URL as a secret — anyone with it can POST events unless you add signature verification. + +### Worker lifecycle commands + +```bash +ntn workers deploy +ntn workers list +ntn workers exec -d '{"name": "world"}' +ntn workers sync trigger # run a sync now +ntn workers sync pause +ntn workers env set GITHUB_WEBHOOK_SECRET=... +ntn workers runs list # recent invocations +ntn workers runs logs +ntn workers webhooks list +``` + +When asked to build a Worker, scaffold with `ntn workers new`, write the code in `src/index.ts`, set any secrets with `ntn workers env set`, and deploy. Notion's docs at https://developers.notion.com/workers cover the full API surface. + +## Notion-Flavored Markdown (used by `/markdown` endpoints) + +Standard CommonMark plus XML-like tags for Notion-specific blocks. Use **tabs** for indentation. + +**Blocks beyond CommonMark:** +``` + + Ship the MVP by **Friday**. + + +
+Toggle title + Children indented one tab +
+ + + Left side + Right side + + + +``` + +**Inline:** +- Mentions: ``, `Title`, `` +- Underline: `text` +- Color: `text` or block-level `{color="blue"}` on the first line +- Math: inline `$x^2$`, block `$$ ... $$` +- Citations: `[^https://example.com]` + +**Colors:** `gray brown orange yellow green blue purple pink red`, plus `*_bg` variants for backgrounds. + +Headings 5/6 collapse to H4. Multiple `>` lines render as separate quote blocks — use `
` inside a single `>` for multi-line quotes. + +## Choosing the Right Path + +| Task | mac / Linux | Windows | +|---|---|---| +| Read/write pages, search, query databases | `ntn api ...` | curl | +| Read a page for an agent to summarize | `ntn api v1/pages/{id}/markdown` | curl `/markdown` endpoint | +| Upload a file | `ntn files create < file` | 3-step HTTP flow | +| One-off API exploration | `ntn api ...` | curl | +| Build a sync / webhook / agent tool hosted by Notion | `ntn workers ...` | WSL2 + `ntn workers ...` | + +## Notes + +- Page/database IDs are UUIDs (with or without dashes — both accepted). +- Rate limit: ~3 requests/second average. The CLI doesn't bypass this. +- The API cannot set database **view** filters — that's UI-only. +- Use `"is_inline": true` when creating data sources to embed them in a page. +- Always pass `-s` to curl to suppress progress bars (cleaner agent output). +- Pipe JSON through `jq` when reading: `... | jq '.results[0].properties'`. +- Notion also ships an MCP server now (`Notion MCP`, ~91% more token-efficient on DB ops than the previous version) — wire it via Hermes' MCP support if you want streaming Notion access from inside a session, but the paths above are enough for most one-shot tasks. diff --git a/notion/references/block-types.md b/notion/references/block-types.md new file mode 100644 index 0000000..943b6a4 --- /dev/null +++ b/notion/references/block-types.md @@ -0,0 +1,112 @@ +# Notion Block Types + +Reference for creating and reading all common Notion block types via the API. + +## Creating blocks + +Use `PATCH /v1/blocks/{page_id}/children` with a `children` array. Each block follows this structure: + +```json +{"object": "block", "type": "", "": { ... }} +``` + +### Paragraph + +```json +{"type": "paragraph", "paragraph": {"rich_text": [{"text": {"content": "Hello world"}}]}} +``` + +### Headings + +```json +{"type": "heading_1", "heading_1": {"rich_text": [{"text": {"content": "Title"}}]}} +{"type": "heading_2", "heading_2": {"rich_text": [{"text": {"content": "Section"}}]}} +{"type": "heading_3", "heading_3": {"rich_text": [{"text": {"content": "Subsection"}}]}} +``` + +### Bulleted list + +```json +{"type": "bulleted_list_item", "bulleted_list_item": {"rich_text": [{"text": {"content": "Item"}}]}} +``` + +### Numbered list + +```json +{"type": "numbered_list_item", "numbered_list_item": {"rich_text": [{"text": {"content": "Step 1"}}]}} +``` + +### To-do / checkbox + +```json +{"type": "to_do", "to_do": {"rich_text": [{"text": {"content": "Task"}}], "checked": false}} +``` + +### Quote + +```json +{"type": "quote", "quote": {"rich_text": [{"text": {"content": "Something wise"}}]}} +``` + +### Callout + +```json +{"type": "callout", "callout": {"rich_text": [{"text": {"content": "Important note"}}], "icon": {"emoji": "💡"}}} +``` + +### Code + +```json +{"type": "code", "code": {"rich_text": [{"text": {"content": "print('hello')"}}], "language": "python"}} +``` + +### Toggle + +```json +{"type": "toggle", "toggle": {"rich_text": [{"text": {"content": "Click to expand"}}]}} +``` + +### Divider + +```json +{"type": "divider", "divider": {}} +``` + +### Bookmark + +```json +{"type": "bookmark", "bookmark": {"url": "https://example.com"}} +``` + +### Image (external URL) + +```json +{"type": "image", "image": {"type": "external", "external": {"url": "https://example.com/photo.png"}}} +``` + +## Reading blocks + +When reading blocks from `GET /v1/blocks/{page_id}/children`, each block has a `type` field. Extract readable text like this: + +| Type | Text location | Extra fields | +|------|--------------|--------------| +| `paragraph` | `.paragraph.rich_text` | — | +| `heading_1/2/3` | `.heading_N.rich_text` | — | +| `bulleted_list_item` | `.bulleted_list_item.rich_text` | — | +| `numbered_list_item` | `.numbered_list_item.rich_text` | — | +| `to_do` | `.to_do.rich_text` | `.to_do.checked` (bool) | +| `toggle` | `.toggle.rich_text` | has children | +| `code` | `.code.rich_text` | `.code.language` | +| `quote` | `.quote.rich_text` | — | +| `callout` | `.callout.rich_text` | `.callout.icon.emoji` | +| `divider` | — | — | +| `image` | `.image.caption` | `.image.file.url` or `.image.external.url` | +| `bookmark` | `.bookmark.caption` | `.bookmark.url` | +| `child_page` | — | `.child_page.title` | +| `child_database` | — | `.child_database.title` | + +Rich text arrays contain objects with `.plain_text` — concatenate them for readable output. + +--- + +*Contributed by [@dogiladeveloper](https://github.com/dogiladeveloper)* diff --git a/ocr-and-documents/DESCRIPTION.md b/ocr-and-documents/DESCRIPTION.md new file mode 100644 index 0000000..b74c8a0 --- /dev/null +++ b/ocr-and-documents/DESCRIPTION.md @@ -0,0 +1,3 @@ +--- +description: Skills for extracting text from PDFs, scanned documents, images, and other file formats using OCR and document parsing tools. +--- diff --git a/ocr-and-documents/SKILL.md b/ocr-and-documents/SKILL.md new file mode 100644 index 0000000..7f6e7bf --- /dev/null +++ b/ocr-and-documents/SKILL.md @@ -0,0 +1,173 @@ +--- +name: ocr-and-documents +description: "Extract text from PDFs/scans (pymupdf, marker-pdf)." +version: 2.3.0 +author: Hermes Agent +license: MIT +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [PDF, Documents, Research, Arxiv, Text-Extraction, OCR] + related_skills: [pdf, docx, powerpoint] +--- + +# PDF & Document Extraction + +For DOCX: see the `docx` skill (create/edit) or use `python-docx` for structured reads. +For PPTX: see the `powerpoint` skill (full create/read/edit support). +For PDF manipulation (merge, split, forms, watermarks, creation): see the `pdf` skill. +This skill covers **text extraction from PDFs and scanned documents**. + +## Step 1: Remote URL Available? + +If the document has a URL, **always try `web_extract` first**: + +``` +web_extract(urls=["https://arxiv.org/pdf/2402.03300"]) +web_extract(urls=["https://example.com/report.pdf"]) +``` + +This handles PDF-to-markdown conversion via Firecrawl with no local dependencies. + +Only use local extraction when: the file is local, web_extract fails, or you need batch processing. + +## Step 2: Choose Local Extractor + +| Feature | pymupdf (~25MB) | marker-pdf (~3-5GB) | +|---------|-----------------|---------------------| +| **Text-based PDF** | ✅ | ✅ | +| **Scanned PDF (OCR)** | ❌ | ✅ (90+ languages) | +| **Tables** | ✅ (basic) | ✅ (high accuracy) | +| **Equations / LaTeX** | ❌ | ✅ | +| **Code blocks** | ❌ | ✅ | +| **Forms** | ❌ | ✅ | +| **Headers/footers removal** | ❌ | ✅ | +| **Reading order detection** | ❌ | ✅ | +| **Images extraction** | ✅ (embedded) | ✅ (with context) | +| **Images → text (OCR)** | ❌ | ✅ | +| **EPUB** | ✅ | ✅ | +| **Markdown output** | ✅ (via pymupdf4llm) | ✅ (native, higher quality) | +| **Install size** | ~25MB | ~3-5GB (PyTorch + models) | +| **Speed** | Instant | ~1-14s/page (CPU), ~0.2s/page (GPU) | + +**Decision**: Use pymupdf unless you need OCR, equations, forms, or complex layout analysis. + +If the user needs marker capabilities but the system lacks ~5GB free disk: +> "This document needs OCR/advanced extraction (marker-pdf), which requires ~5GB for PyTorch and models. Your system has [X]GB free. Options: free up space, provide a URL so I can use web_extract, or I can try pymupdf which works for text-based PDFs but not scanned documents or equations." + +--- + +## pymupdf (lightweight) + +```bash +pip install pymupdf pymupdf4llm +``` + +**Via helper script**: +```bash +python scripts/extract_pymupdf.py document.pdf # Plain text +python scripts/extract_pymupdf.py document.pdf --markdown # Markdown +python scripts/extract_pymupdf.py document.pdf --tables # Tables +python scripts/extract_pymupdf.py document.pdf --images out/ # Extract images +python scripts/extract_pymupdf.py document.pdf --metadata # Title, author, pages +python scripts/extract_pymupdf.py document.pdf --pages 0-4 # Specific pages +``` + +**Inline**: +```bash +python3 -c " +import pymupdf +doc = pymupdf.open('document.pdf') +for page in doc: + print(page.get_text()) +" +``` + +--- + +## marker-pdf (high-quality OCR) + +```bash +# Check disk space first +python scripts/extract_marker.py --check + +pip install marker-pdf +``` + +**Via helper script**: +```bash +python scripts/extract_marker.py document.pdf # Markdown +python scripts/extract_marker.py document.pdf --json # JSON with metadata +python scripts/extract_marker.py document.pdf --output_dir out/ # Save images +python scripts/extract_marker.py scanned.pdf # Scanned PDF (OCR) +python scripts/extract_marker.py document.pdf --use_llm # LLM-boosted accuracy +``` + +**CLI** (installed with marker-pdf): +```bash +marker_single document.pdf --output_dir ./output +marker /path/to/folder --workers 4 # Batch +``` + +--- + +## Arxiv Papers + +``` +# Abstract only (fast) +web_extract(urls=["https://arxiv.org/abs/2402.03300"]) + +# Full paper +web_extract(urls=["https://arxiv.org/pdf/2402.03300"]) + +# Search +web_search(query="arxiv GRPO reinforcement learning 2026") +``` + +## Split, Merge & Search + +pymupdf handles these natively — use `execute_code` or inline Python: + +```python +# Split: extract pages 1-5 to a new PDF +import pymupdf +doc = pymupdf.open("report.pdf") +new = pymupdf.open() +for i in range(5): + new.insert_pdf(doc, from_page=i, to_page=i) +new.save("pages_1-5.pdf") +``` + +```python +# Merge multiple PDFs +import pymupdf +result = pymupdf.open() +for path in ["a.pdf", "b.pdf", "c.pdf"]: + result.insert_pdf(pymupdf.open(path)) +result.save("merged.pdf") +``` + +```python +# Search for text across all pages +import pymupdf +doc = pymupdf.open("report.pdf") +for i, page in enumerate(doc): + results = page.search_for("revenue") + if results: + print(f"Page {i+1}: {len(results)} match(es)") + print(page.get_text("text")) +``` + +No extra dependencies needed — pymupdf covers split, merge, search, and text extraction in one package. + +--- + +## Notes + +- `web_extract` is always first choice for URLs +- pymupdf is the safe default — instant, no models, works everywhere +- marker-pdf is for OCR, scanned docs, equations, complex layouts — install only when needed +- Both helper scripts accept `--help` for full usage +- marker-pdf downloads ~2.5GB of models to `~/.cache/huggingface/` on first use +- For Word docs: `pip install python-docx` (better than OCR — parses actual structure) +- For PowerPoint: see the `powerpoint` skill (uses python-pptx) diff --git a/ocr-and-documents/scripts/extract_marker.py b/ocr-and-documents/scripts/extract_marker.py new file mode 100644 index 0000000..d48fd10 --- /dev/null +++ b/ocr-and-documents/scripts/extract_marker.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +"""Extract text from documents using marker-pdf. High-quality OCR + layout analysis. + +Requires ~3-5GB disk (PyTorch + models downloaded on first use). +Supports: PDF, DOCX, PPTX, XLSX, HTML, EPUB, images. + +Usage: + python extract_marker.py document.pdf + python extract_marker.py document.pdf --output_dir ./output + python extract_marker.py presentation.pptx + python extract_marker.py spreadsheet.xlsx + python extract_marker.py scanned_doc.pdf # OCR works here + python extract_marker.py document.pdf --json # Structured output + python extract_marker.py document.pdf --use_llm # LLM-boosted accuracy +""" +import sys +import os + +def convert(path, output_dir=None, output_format="markdown", use_llm=False): + from marker.converters.pdf import PdfConverter + from marker.models import create_model_dict + from marker.config.parser import ConfigParser + + config_dict = {} + if use_llm: + config_dict["use_llm"] = True + + config_parser = ConfigParser(config_dict) + models = create_model_dict() + converter = PdfConverter(config=config_parser.generate_config_dict(), artifact_dict=models) + rendered = converter(path) + + if output_format == "json": + import json + print(json.dumps({ + "markdown": rendered.markdown, + "metadata": rendered.metadata if hasattr(rendered, "metadata") else {}, + }, indent=2, ensure_ascii=False)) + else: + print(rendered.markdown) + + # Save images if output_dir specified + if output_dir and hasattr(rendered, "images") and rendered.images: + from pathlib import Path + Path(output_dir).mkdir(parents=True, exist_ok=True) + for name, img_data in rendered.images.items(): + img_path = os.path.join(output_dir, name) + with open(img_path, "wb") as f: + f.write(img_data) + print(f"\nSaved {len(rendered.images)} image(s) to {output_dir}/", file=sys.stderr) + + +def check_requirements(): + """Check disk space before installing.""" + import shutil + free_gb = shutil.disk_usage("/").free / (1024**3) + if free_gb < 5: + print(f"⚠️ Only {free_gb:.1f}GB free. marker-pdf needs ~5GB for PyTorch + models.") + print("Use pymupdf instead (scripts/extract_pymupdf.py) or free up disk space.") + sys.exit(1) + print(f"✓ {free_gb:.1f}GB free — sufficient for marker-pdf") + + +if __name__ == "__main__": + args = sys.argv[1:] + if not args or args[0] in {"-h", "--help"}: + print(__doc__) + sys.exit(0) + + if args[0] == "--check": + check_requirements() + sys.exit(0) + + path = args[0] + output_dir = None + output_format = "markdown" + use_llm = False + + if "--output_dir" in args: + idx = args.index("--output_dir") + output_dir = args[idx + 1] + if "--json" in args: + output_format = "json" + if "--use_llm" in args: + use_llm = True + + convert(path, output_dir=output_dir, output_format=output_format, use_llm=use_llm) diff --git a/ocr-and-documents/scripts/extract_pymupdf.py b/ocr-and-documents/scripts/extract_pymupdf.py new file mode 100644 index 0000000..50cb8ee --- /dev/null +++ b/ocr-and-documents/scripts/extract_pymupdf.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Extract text from documents using pymupdf. Lightweight (~25MB), no models. + +Usage: + python extract_pymupdf.py document.pdf + python extract_pymupdf.py document.pdf --markdown + python extract_pymupdf.py document.pdf --pages 0-4 + python extract_pymupdf.py document.pdf --images output_dir/ + python extract_pymupdf.py document.pdf --tables + python extract_pymupdf.py document.pdf --metadata +""" +import sys +import json + +def extract_text(path, pages=None): + import pymupdf + doc = pymupdf.open(path) + page_range = range(len(doc)) if pages is None else pages + for i in page_range: + if i < len(doc): + print(f"\n--- Page {i+1}/{len(doc)} ---\n") + print(doc[i].get_text()) + +def extract_markdown(path, pages=None): + import pymupdf4llm + md = pymupdf4llm.to_markdown(path, pages=pages) + print(md) + +def extract_tables(path): + import pymupdf + doc = pymupdf.open(path) + for i, page in enumerate(doc): + tables = page.find_tables() + for j, table in enumerate(tables.tables): + print(f"\n--- Page {i+1}, Table {j+1} ---\n") + df = table.to_pandas() + print(df.to_markdown(index=False)) + +def extract_images(path, output_dir): + import pymupdf + from pathlib import Path + Path(output_dir).mkdir(parents=True, exist_ok=True) + doc = pymupdf.open(path) + count = 0 + for i, page in enumerate(doc): + for img_idx, img in enumerate(page.get_images(full=True)): + xref = img[0] + pix = pymupdf.Pixmap(doc, xref) + if pix.n >= 5: + pix = pymupdf.Pixmap(pymupdf.csRGB, pix) + out_path = f"{output_dir}/page{i+1}_img{img_idx+1}.png" + pix.save(out_path) + count += 1 + print(f"Extracted {count} images to {output_dir}/") + +def show_metadata(path): + import pymupdf + doc = pymupdf.open(path) + print(json.dumps({ + "pages": len(doc), + "title": doc.metadata.get("title", ""), + "author": doc.metadata.get("author", ""), + "subject": doc.metadata.get("subject", ""), + "creator": doc.metadata.get("creator", ""), + "producer": doc.metadata.get("producer", ""), + "format": doc.metadata.get("format", ""), + }, indent=2)) + +if __name__ == "__main__": + args = sys.argv[1:] + if not args or args[0] in {"-h", "--help"}: + print(__doc__) + sys.exit(0) + + path = args[0] + pages = None + + if "--pages" in args: + idx = args.index("--pages") + p = args[idx + 1] + if "-" in p: + start, end = p.split("-") + pages = list(range(int(start), int(end) + 1)) + else: + pages = [int(p)] + + if "--metadata" in args: + show_metadata(path) + elif "--tables" in args: + extract_tables(path) + elif "--images" in args: + idx = args.index("--images") + output_dir = args[idx + 1] if idx + 1 < len(args) else "./images" + extract_images(path, output_dir) + elif "--markdown" in args: + extract_markdown(path, pages=pages) + else: + extract_text(path, pages=pages) diff --git a/pdf/LICENSE.txt b/pdf/LICENSE.txt new file mode 100644 index 0000000..c55ab42 --- /dev/null +++ b/pdf/LICENSE.txt @@ -0,0 +1,30 @@ +© 2025 Anthropic, PBC. All rights reserved. + +LICENSE: Use of these materials (including all code, prompts, assets, files, +and other components of this Skill) is governed by your agreement with +Anthropic regarding use of Anthropic's services. If no separate agreement +exists, use is governed by Anthropic's Consumer Terms of Service or +Commercial Terms of Service, as applicable: +https://www.anthropic.com/legal/consumer-terms +https://www.anthropic.com/legal/commercial-terms +Your applicable agreement is referred to as the "Agreement." "Services" are +as defined in the Agreement. + +ADDITIONAL RESTRICTIONS: Notwithstanding anything in the Agreement to the +contrary, users may not: + +- Extract these materials from the Services or retain copies of these + materials outside the Services +- Reproduce or copy these materials, except for temporary copies created + automatically during authorized use of the Services +- Create derivative works based on these materials +- Distribute, sublicense, or transfer these materials to any third party +- Make, offer to sell, sell, or import any inventions embodied in these + materials +- Reverse engineer, decompile, or disassemble these materials + +The receipt, viewing, or possession of these materials does not convey or +imply any license or right beyond those expressly granted above. + +Anthropic retains all right, title, and interest in these materials, +including all copyrights, patents, and other intellectual property rights. diff --git a/pdf/SKILL.md b/pdf/SKILL.md new file mode 100644 index 0000000..23d9730 --- /dev/null +++ b/pdf/SKILL.md @@ -0,0 +1,174 @@ +--- +name: pdf +description: "Create, merge, split, fill, and secure PDF files." +version: 1.0.0 +author: Anthropic (adapted by Nous Research) +license: Proprietary. LICENSE.txt has complete terms +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [PDF, Documents, Forms, Office, Productivity] + category: productivity + related_skills: [ocr-and-documents, nano-pdf, docx, xlsx] +--- + +# PDF Skill + +Create, combine, split, transform, and secure PDF files — merging, page manipulation, form filling, watermarks, encryption, and text/table extraction. For heavy text extraction from scanned documents prefer the `ocr-and-documents` skill; for natural-language edits to existing PDF text prefer `nano-pdf`. + +## When to Use + +Use this skill whenever the user wants to do anything with PDF files: reading or extracting text/tables, combining or merging multiple PDFs, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting, extracting images, or OCR on scanned PDFs. If the user mentions a .pdf file or asks to produce one, use this skill. + +## Prerequisites + +```bash +pip install pypdf pdfplumber reportlab +which pdftotext || sudo apt install -y poppler-utils # pdftotext, pdftoppm, pdfimages +which qpdf || sudo apt install -y qpdf # CLI merge/split/decrypt +``` + +macOS: `brew install poppler qpdf`. OCR extras: `pip install pytesseract pdf2image` + `sudo apt install -y tesseract-ocr`. + +> Script paths below are relative to this skill's directory. Form filling has its own workflow — read [forms.md](forms.md) and follow it. Advanced library usage (pypdfium2, pdf-lib) and troubleshooting: [reference.md](reference.md). + +## Quick Reference + +| Task | Best Tool | Command/Code | +|------|-----------|--------------| +| Merge PDFs | pypdf | `writer.add_page(page)` per page | +| Split PDFs | pypdf | One page per file | +| Extract text | pdfplumber | `page.extract_text()` | +| Extract tables | pdfplumber | `page.extract_tables()` | +| Create PDFs | reportlab | Canvas or Platypus | +| Command-line merge/split | qpdf | `qpdf --empty --pages ...` | +| OCR scanned PDFs | pytesseract | Convert to images first (or use `ocr-and-documents`) | +| Fill PDF forms | see [forms.md](forms.md) | `scripts/fill_fillable_fields.py` etc. | +| Edit existing text | `nano-pdf` skill | `nano-pdf edit file.pdf ""` | + +## Common operations + +### Merge / split / rotate (pypdf) + +```python +from pypdf import PdfReader, PdfWriter + +# Merge +writer = PdfWriter() +for pdf_file in ["doc1.pdf", "doc2.pdf"]: + for page in PdfReader(pdf_file).pages: + writer.add_page(page) +with open("merged.pdf", "wb") as f: + writer.write(f) + +# Split: one file per page +reader = PdfReader("input.pdf") +for i, page in enumerate(reader.pages): + w = PdfWriter(); w.add_page(page) + with open(f"page_{i+1}.pdf", "wb") as f: + w.write(f) + +# Rotate +page = reader.pages[0] +page.rotate(90) # clockwise +``` + +### Extract text and tables (pdfplumber) + +```python +import pdfplumber, pandas as pd + +with pdfplumber.open("document.pdf") as pdf: + text = "\n".join(page.extract_text() or "" for page in pdf.pages) + tables = [pd.DataFrame(t[1:], columns=t[0]) + for page in pdf.pages + for t in page.extract_tables() if t] +``` + +### Create PDFs (reportlab) + +```python +from reportlab.lib.pagesizes import letter +from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, PageBreak +from reportlab.lib.styles import getSampleStyleSheet + +doc = SimpleDocTemplate("report.pdf", pagesize=letter) +styles = getSampleStyleSheet() +story = [Paragraph("Report Title", styles["Title"]), Spacer(1, 12), + Paragraph("Body text...", styles["Normal"]), PageBreak(), + Paragraph("Page 2", styles["Heading1"])] +doc.build(story) +``` + +**Subscripts/superscripts:** never use Unicode sub/superscript characters (₀₁₂, ⁰¹²) — the built-in fonts lack the glyphs and render solid black boxes. Use ``/`` markup inside `Paragraph` objects: `Paragraph("H2O", styles['Normal'])`. For canvas-drawn text, adjust font size and position manually. + +### Command-line tools + +```bash +pdftotext -layout input.pdf output.txt # text, layout preserved +pdftotext -f 1 -l 5 input.pdf output.txt # pages 1-5 +qpdf --empty --pages file1.pdf file2.pdf -- merged.pdf # merge +qpdf input.pdf --pages . 1-5 -- pages1-5.pdf # split range +qpdf input.pdf output.pdf --rotate=+90:1 # rotate page 1 +qpdf --password=pw --decrypt encrypted.pdf decrypted.pdf # remove password +pdfimages -j input.pdf img # extract images +``` + +### Watermark + +```python +from pypdf import PdfReader, PdfWriter + +watermark = PdfReader("watermark.pdf").pages[0] +reader, writer = PdfReader("document.pdf"), PdfWriter() +for page in reader.pages: + page.merge_page(watermark) + writer.add_page(page) +with open("watermarked.pdf", "wb") as f: + writer.write(f) +``` + +### Password protection + +```python +writer.encrypt("userpassword", "ownerpassword") +``` + +### OCR scanned PDFs + +```python +import pytesseract +from pdf2image import convert_from_path + +pages = convert_from_path("scanned.pdf") +text = "\n\n".join(pytesseract.image_to_string(img) for img in pages) +``` + +For batch/structured extraction from scans, the `ocr-and-documents` skill (pymupdf, marker-pdf) is the better path. + +## Form filling + +Read [forms.md](forms.md) first — it distinguishes fillable (AcroForm) PDFs from flat scanned forms and walks through the helper scripts: + +- `scripts/check_fillable_fields.py` — does the PDF have AcroForm fields? +- `scripts/extract_form_field_info.py` / `scripts/extract_form_structure.py` — enumerate fields +- `scripts/fill_fillable_fields.py` — fill AcroForm fields +- `scripts/fill_pdf_form_with_annotations.py` — overlay text on flat forms +- `scripts/check_bounding_boxes.py`, `scripts/create_validation_image.py` — verify placement visually + +## Pitfalls + +- `page.extract_text()` returns `None` on image-only pages — guard with `or ""` and fall back to OCR. +- pypdf preserves encryption flags: reading an encrypted PDF requires `PdfReader(path, password=...)` before pages are accessible. +- reportlab coordinates are bottom-left origin, points (1/72″) — not top-left. +- When filling flat forms by annotation overlay, always render a validation image and check the placement before delivering. + +## Verification + +1. Open the output with `PdfReader` and assert the expected page count. +2. Re-extract text from the output (`pdftotext` or pdfplumber) and confirm the content you added is present. +3. For anything visual (watermarks, filled forms, created reports): `pdftoppm -jpeg -r 100 output.pdf page` and inspect the images with `vision_analyze`. + +## Related skills + +`ocr-and-documents` (scanned-document text extraction), `nano-pdf` (NL text edits in place), `docx` (Word), `xlsx` (spreadsheets), `powerpoint` (decks). diff --git a/pdf/forms.md b/pdf/forms.md new file mode 100644 index 0000000..6e7e1e0 --- /dev/null +++ b/pdf/forms.md @@ -0,0 +1,294 @@ +**CRITICAL: You MUST complete these steps in order. Do not skip ahead to writing code.** + +If you need to fill out a PDF form, first check to see if the PDF has fillable form fields. Run this script from this file's directory: + `python scripts/check_fillable_fields `, and depending on the result go to either the "Fillable fields" or "Non-fillable fields" and follow those instructions. + +# Fillable fields +If the PDF has fillable form fields: +- Run this script from this file's directory: `python scripts/extract_form_field_info.py `. It will create a JSON file with a list of fields in this format: +``` +[ + { + "field_id": (unique ID for the field), + "page": (page number, 1-based), + "rect": ([left, bottom, right, top] bounding box in PDF coordinates, y=0 is the bottom of the page), + "type": ("text", "checkbox", "radio_group", or "choice"), + }, + // Checkboxes have "checked_value" and "unchecked_value" properties: + { + "field_id": (unique ID for the field), + "page": (page number, 1-based), + "type": "checkbox", + "checked_value": (Set the field to this value to check the checkbox), + "unchecked_value": (Set the field to this value to uncheck the checkbox), + }, + // Radio groups have a "radio_options" list with the possible choices. + { + "field_id": (unique ID for the field), + "page": (page number, 1-based), + "type": "radio_group", + "radio_options": [ + { + "value": (set the field to this value to select this radio option), + "rect": (bounding box for the radio button for this option) + }, + // Other radio options + ] + }, + // Multiple choice fields have a "choice_options" list with the possible choices: + { + "field_id": (unique ID for the field), + "page": (page number, 1-based), + "type": "choice", + "choice_options": [ + { + "value": (set the field to this value to select this option), + "text": (display text of the option) + }, + // Other choice options + ], + } +] +``` +- Convert the PDF to PNGs (one image for each page) with this script (run from this file's directory): +`python scripts/convert_pdf_to_images.py ` +Then analyze the images to determine the purpose of each form field (make sure to convert the bounding box PDF coordinates to image coordinates). +- Create a `field_values.json` file in this format with the values to be entered for each field: +``` +[ + { + "field_id": "last_name", // Must match the field_id from `extract_form_field_info.py` + "description": "The user's last name", + "page": 1, // Must match the "page" value in field_info.json + "value": "Simpson" + }, + { + "field_id": "Checkbox12", + "description": "Checkbox to be checked if the user is 18 or over", + "page": 1, + "value": "/On" // If this is a checkbox, use its "checked_value" value to check it. If it's a radio button group, use one of the "value" values in "radio_options". + }, + // more fields +] +``` +- Run the `fill_fillable_fields.py` script from this file's directory to create a filled-in PDF: +`python scripts/fill_fillable_fields.py ` +This script will verify that the field IDs and values you provide are valid; if it prints error messages, correct the appropriate fields and try again. + +# Non-fillable fields +If the PDF doesn't have fillable form fields, you'll add text annotations. First try to extract coordinates from the PDF structure (more accurate), then fall back to visual estimation if needed. + +## Step 1: Try Structure Extraction First + +Run this script to extract text labels, lines, and checkboxes with their exact PDF coordinates: +`python scripts/extract_form_structure.py form_structure.json` + +This creates a JSON file containing: +- **labels**: Every text element with exact coordinates (x0, top, x1, bottom in PDF points) +- **lines**: Horizontal lines that define row boundaries +- **checkboxes**: Small square rectangles that are checkboxes (with center coordinates) +- **row_boundaries**: Row top/bottom positions calculated from horizontal lines + +**Check the results**: If `form_structure.json` has meaningful labels (text elements that correspond to form fields), use **Approach A: Structure-Based Coordinates**. If the PDF is scanned/image-based and has few or no labels, use **Approach B: Visual Estimation**. + +--- + +## Approach A: Structure-Based Coordinates (Preferred) + +Use this when `extract_form_structure.py` found text labels in the PDF. + +### A.1: Analyze the Structure + +Read form_structure.json and identify: + +1. **Label groups**: Adjacent text elements that form a single label (e.g., "Last" + "Name") +2. **Row structure**: Labels with similar `top` values are in the same row +3. **Field columns**: Entry areas start after label ends (x0 = label.x1 + gap) +4. **Checkboxes**: Use the checkbox coordinates directly from the structure + +**Coordinate system**: PDF coordinates where y=0 is at TOP of page, y increases downward. + +### A.2: Check for Missing Elements + +The structure extraction may not detect all form elements. Common cases: +- **Circular checkboxes**: Only square rectangles are detected as checkboxes +- **Complex graphics**: Decorative elements or non-standard form controls +- **Faded or light-colored elements**: May not be extracted + +If you see form fields in the PDF images that aren't in form_structure.json, you'll need to use **visual analysis** for those specific fields (see "Hybrid Approach" below). + +### A.3: Create fields.json with PDF Coordinates + +For each field, calculate entry coordinates from the extracted structure: + +**Text fields:** +- entry x0 = label x1 + 5 (small gap after label) +- entry x1 = next label's x0, or row boundary +- entry top = same as label top +- entry bottom = row boundary line below, or label bottom + row_height + +**Checkboxes:** +- Use the checkbox rectangle coordinates directly from form_structure.json +- entry_bounding_box = [checkbox.x0, checkbox.top, checkbox.x1, checkbox.bottom] + +Create fields.json using `pdf_width` and `pdf_height` (signals PDF coordinates): +```json +{ + "pages": [ + {"page_number": 1, "pdf_width": 612, "pdf_height": 792} + ], + "form_fields": [ + { + "page_number": 1, + "description": "Last name entry field", + "field_label": "Last Name", + "label_bounding_box": [43, 63, 87, 73], + "entry_bounding_box": [92, 63, 260, 79], + "entry_text": {"text": "Smith", "font_size": 10} + }, + { + "page_number": 1, + "description": "US Citizen Yes checkbox", + "field_label": "Yes", + "label_bounding_box": [260, 200, 280, 210], + "entry_bounding_box": [285, 197, 292, 205], + "entry_text": {"text": "X"} + } + ] +} +``` + +**Important**: Use `pdf_width`/`pdf_height` and coordinates directly from form_structure.json. + +### A.4: Validate Bounding Boxes + +Before filling, check your bounding boxes for errors: +`python scripts/check_bounding_boxes.py fields.json` + +This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling. + +--- + +## Approach B: Visual Estimation (Fallback) + +Use this when the PDF is scanned/image-based and structure extraction found no usable text labels (e.g., all text shows as "(cid:X)" patterns). + +### B.1: Convert PDF to Images + +`python scripts/convert_pdf_to_images.py ` + +### B.2: Initial Field Identification + +Examine each page image to identify form sections and get **rough estimates** of field locations: +- Form field labels and their approximate positions +- Entry areas (lines, boxes, or blank spaces for text input) +- Checkboxes and their approximate locations + +For each field, note approximate pixel coordinates (they don't need to be precise yet). + +### B.3: Zoom Refinement (CRITICAL for accuracy) + +For each field, crop a region around the estimated position to refine coordinates precisely. + +**Create a zoomed crop using ImageMagick:** +```bash +magick -crop x++ +repage +``` + +Where: +- `, ` = top-left corner of crop region (use your rough estimate minus padding) +- `, ` = size of crop region (field area plus ~50px padding on each side) + +**Example:** To refine a "Name" field estimated around (100, 150): +```bash +magick images_dir/page_1.png -crop 300x80+50+120 +repage crops/name_field.png +``` + +(Note: if the `magick` command isn't available, try `convert` with the same arguments). + +**Examine the cropped image** to determine precise coordinates: +1. Identify the exact pixel where the entry area begins (after the label) +2. Identify where the entry area ends (before next field or edge) +3. Identify the top and bottom of the entry line/box + +**Convert crop coordinates back to full image coordinates:** +- full_x = crop_x + crop_offset_x +- full_y = crop_y + crop_offset_y + +Example: If the crop started at (50, 120) and the entry box starts at (52, 18) within the crop: +- entry_x0 = 52 + 50 = 102 +- entry_top = 18 + 120 = 138 + +**Repeat for each field**, grouping nearby fields into single crops when possible. + +### B.4: Create fields.json with Refined Coordinates + +Create fields.json using `image_width` and `image_height` (signals image coordinates): +```json +{ + "pages": [ + {"page_number": 1, "image_width": 1700, "image_height": 2200} + ], + "form_fields": [ + { + "page_number": 1, + "description": "Last name entry field", + "field_label": "Last Name", + "label_bounding_box": [120, 175, 242, 198], + "entry_bounding_box": [255, 175, 720, 218], + "entry_text": {"text": "Smith", "font_size": 10} + } + ] +} +``` + +**Important**: Use `image_width`/`image_height` and the refined pixel coordinates from the zoom analysis. + +### B.5: Validate Bounding Boxes + +Before filling, check your bounding boxes for errors: +`python scripts/check_bounding_boxes.py fields.json` + +This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling. + +--- + +## Hybrid Approach: Structure + Visual + +Use this when structure extraction works for most fields but misses some elements (e.g., circular checkboxes, unusual form controls). + +1. **Use Approach A** for fields that were detected in form_structure.json +2. **Convert PDF to images** for visual analysis of missing fields +3. **Use zoom refinement** (from Approach B) for the missing fields +4. **Combine coordinates**: For fields from structure extraction, use `pdf_width`/`pdf_height`. For visually-estimated fields, you must convert image coordinates to PDF coordinates: + - pdf_x = image_x * (pdf_width / image_width) + - pdf_y = image_y * (pdf_height / image_height) +5. **Use a single coordinate system** in fields.json - convert all to PDF coordinates with `pdf_width`/`pdf_height` + +--- + +## Step 2: Validate Before Filling + +**Always validate bounding boxes before filling:** +`python scripts/check_bounding_boxes.py fields.json` + +This checks for: +- Intersecting bounding boxes (which would cause overlapping text) +- Entry boxes that are too small for the specified font size + +Fix any reported errors in fields.json before proceeding. + +## Step 3: Fill the Form + +The fill script auto-detects the coordinate system and handles conversion: +`python scripts/fill_pdf_form_with_annotations.py fields.json ` + +## Step 4: Verify Output + +Convert the filled PDF to images and verify text placement: +`python scripts/convert_pdf_to_images.py ` + +If text is mispositioned: +- **Approach A**: Check that you're using PDF coordinates from form_structure.json with `pdf_width`/`pdf_height` +- **Approach B**: Check that image dimensions match and coordinates are accurate pixels +- **Hybrid**: Ensure coordinate conversions are correct for visually-estimated fields diff --git a/pdf/reference.md b/pdf/reference.md new file mode 100644 index 0000000..41400bf --- /dev/null +++ b/pdf/reference.md @@ -0,0 +1,612 @@ +# PDF Processing Advanced Reference + +This document contains advanced PDF processing features, detailed examples, and additional libraries not covered in the main skill instructions. + +## pypdfium2 Library (Apache/BSD License) + +### Overview +pypdfium2 is a Python binding for PDFium (Chromium's PDF library). It's excellent for fast PDF rendering, image generation, and serves as a PyMuPDF replacement. + +### Render PDF to Images +```python +import pypdfium2 as pdfium +from PIL import Image + +# Load PDF +pdf = pdfium.PdfDocument("document.pdf") + +# Render page to image +page = pdf[0] # First page +bitmap = page.render( + scale=2.0, # Higher resolution + rotation=0 # No rotation +) + +# Convert to PIL Image +img = bitmap.to_pil() +img.save("page_1.png", "PNG") + +# Process multiple pages +for i, page in enumerate(pdf): + bitmap = page.render(scale=1.5) + img = bitmap.to_pil() + img.save(f"page_{i+1}.jpg", "JPEG", quality=90) +``` + +### Extract Text with pypdfium2 +```python +import pypdfium2 as pdfium + +pdf = pdfium.PdfDocument("document.pdf") +for i, page in enumerate(pdf): + text = page.get_text() + print(f"Page {i+1} text length: {len(text)} chars") +``` + +## JavaScript Libraries + +### pdf-lib (MIT License) + +pdf-lib is a powerful JavaScript library for creating and modifying PDF documents in any JavaScript environment. + +#### Load and Manipulate Existing PDF +```javascript +import { PDFDocument } from 'pdf-lib'; +import fs from 'fs'; + +async function manipulatePDF() { + // Load existing PDF + const existingPdfBytes = fs.readFileSync('input.pdf'); + const pdfDoc = await PDFDocument.load(existingPdfBytes); + + // Get page count + const pageCount = pdfDoc.getPageCount(); + console.log(`Document has ${pageCount} pages`); + + // Add new page + const newPage = pdfDoc.addPage([600, 400]); + newPage.drawText('Added by pdf-lib', { + x: 100, + y: 300, + size: 16 + }); + + // Save modified PDF + const pdfBytes = await pdfDoc.save(); + fs.writeFileSync('modified.pdf', pdfBytes); +} +``` + +#### Create Complex PDFs from Scratch +```javascript +import { PDFDocument, rgb, StandardFonts } from 'pdf-lib'; +import fs from 'fs'; + +async function createPDF() { + const pdfDoc = await PDFDocument.create(); + + // Add fonts + const helveticaFont = await pdfDoc.embedFont(StandardFonts.Helvetica); + const helveticaBold = await pdfDoc.embedFont(StandardFonts.HelveticaBold); + + // Add page + const page = pdfDoc.addPage([595, 842]); // A4 size + const { width, height } = page.getSize(); + + // Add text with styling + page.drawText('Invoice #12345', { + x: 50, + y: height - 50, + size: 18, + font: helveticaBold, + color: rgb(0.2, 0.2, 0.8) + }); + + // Add rectangle (header background) + page.drawRectangle({ + x: 40, + y: height - 100, + width: width - 80, + height: 30, + color: rgb(0.9, 0.9, 0.9) + }); + + // Add table-like content + const items = [ + ['Item', 'Qty', 'Price', 'Total'], + ['Widget', '2', '$50', '$100'], + ['Gadget', '1', '$75', '$75'] + ]; + + let yPos = height - 150; + items.forEach(row => { + let xPos = 50; + row.forEach(cell => { + page.drawText(cell, { + x: xPos, + y: yPos, + size: 12, + font: helveticaFont + }); + xPos += 120; + }); + yPos -= 25; + }); + + const pdfBytes = await pdfDoc.save(); + fs.writeFileSync('created.pdf', pdfBytes); +} +``` + +#### Advanced Merge and Split Operations +```javascript +import { PDFDocument } from 'pdf-lib'; +import fs from 'fs'; + +async function mergePDFs() { + // Create new document + const mergedPdf = await PDFDocument.create(); + + // Load source PDFs + const pdf1Bytes = fs.readFileSync('doc1.pdf'); + const pdf2Bytes = fs.readFileSync('doc2.pdf'); + + const pdf1 = await PDFDocument.load(pdf1Bytes); + const pdf2 = await PDFDocument.load(pdf2Bytes); + + // Copy pages from first PDF + const pdf1Pages = await mergedPdf.copyPages(pdf1, pdf1.getPageIndices()); + pdf1Pages.forEach(page => mergedPdf.addPage(page)); + + // Copy specific pages from second PDF (pages 0, 2, 4) + const pdf2Pages = await mergedPdf.copyPages(pdf2, [0, 2, 4]); + pdf2Pages.forEach(page => mergedPdf.addPage(page)); + + const mergedPdfBytes = await mergedPdf.save(); + fs.writeFileSync('merged.pdf', mergedPdfBytes); +} +``` + +### pdfjs-dist (Apache License) + +PDF.js is Mozilla's JavaScript library for rendering PDFs in the browser. + +#### Basic PDF Loading and Rendering +```javascript +import * as pdfjsLib from 'pdfjs-dist'; + +// Configure worker (important for performance) +pdfjsLib.GlobalWorkerOptions.workerSrc = './pdf.worker.js'; + +async function renderPDF() { + // Load PDF + const loadingTask = pdfjsLib.getDocument('document.pdf'); + const pdf = await loadingTask.promise; + + console.log(`Loaded PDF with ${pdf.numPages} pages`); + + // Get first page + const page = await pdf.getPage(1); + const viewport = page.getViewport({ scale: 1.5 }); + + // Render to canvas + const canvas = document.createElement('canvas'); + const context = canvas.getContext('2d'); + canvas.height = viewport.height; + canvas.width = viewport.width; + + const renderContext = { + canvasContext: context, + viewport: viewport + }; + + await page.render(renderContext).promise; + document.body.appendChild(canvas); +} +``` + +#### Extract Text with Coordinates +```javascript +import * as pdfjsLib from 'pdfjs-dist'; + +async function extractText() { + const loadingTask = pdfjsLib.getDocument('document.pdf'); + const pdf = await loadingTask.promise; + + let fullText = ''; + + // Extract text from all pages + for (let i = 1; i <= pdf.numPages; i++) { + const page = await pdf.getPage(i); + const textContent = await page.getTextContent(); + + const pageText = textContent.items + .map(item => item.str) + .join(' '); + + fullText += `\n--- Page ${i} ---\n${pageText}`; + + // Get text with coordinates for advanced processing + const textWithCoords = textContent.items.map(item => ({ + text: item.str, + x: item.transform[4], + y: item.transform[5], + width: item.width, + height: item.height + })); + } + + console.log(fullText); + return fullText; +} +``` + +#### Extract Annotations and Forms +```javascript +import * as pdfjsLib from 'pdfjs-dist'; + +async function extractAnnotations() { + const loadingTask = pdfjsLib.getDocument('annotated.pdf'); + const pdf = await loadingTask.promise; + + for (let i = 1; i <= pdf.numPages; i++) { + const page = await pdf.getPage(i); + const annotations = await page.getAnnotations(); + + annotations.forEach(annotation => { + console.log(`Annotation type: ${annotation.subtype}`); + console.log(`Content: ${annotation.contents}`); + console.log(`Coordinates: ${JSON.stringify(annotation.rect)}`); + }); + } +} +``` + +## Advanced Command-Line Operations + +### poppler-utils Advanced Features + +#### Extract Text with Bounding Box Coordinates +```bash +# Extract text with bounding box coordinates (essential for structured data) +pdftotext -bbox-layout document.pdf output.xml + +# The XML output contains precise coordinates for each text element +``` + +#### Advanced Image Conversion +```bash +# Convert to PNG images with specific resolution +pdftoppm -png -r 300 document.pdf output_prefix + +# Convert specific page range with high resolution +pdftoppm -png -r 600 -f 1 -l 3 document.pdf high_res_pages + +# Convert to JPEG with quality setting +pdftoppm -jpeg -jpegopt quality=85 -r 200 document.pdf jpeg_output +``` + +#### Extract Embedded Images +```bash +# Extract all embedded images with metadata +pdfimages -j -p document.pdf page_images + +# List image info without extracting +pdfimages -list document.pdf + +# Extract images in their original format +pdfimages -all document.pdf images/img +``` + +### qpdf Advanced Features + +#### Complex Page Manipulation +```bash +# Split PDF into groups of pages +qpdf --split-pages=3 input.pdf output_group_%02d.pdf + +# Extract specific pages with complex ranges +qpdf input.pdf --pages input.pdf 1,3-5,8,10-end -- extracted.pdf + +# Merge specific pages from multiple PDFs +qpdf --empty --pages doc1.pdf 1-3 doc2.pdf 5-7 doc3.pdf 2,4 -- combined.pdf +``` + +#### PDF Optimization and Repair +```bash +# Optimize PDF for web (linearize for streaming) +qpdf --linearize input.pdf optimized.pdf + +# Remove unused objects and compress +qpdf --optimize-level=all input.pdf compressed.pdf + +# Attempt to repair corrupted PDF structure +qpdf --check input.pdf +qpdf --fix-qdf damaged.pdf repaired.pdf + +# Show detailed PDF structure for debugging +qpdf --show-all-pages input.pdf > structure.txt +``` + +#### Advanced Encryption +```bash +# Add password protection with specific permissions +qpdf --encrypt user_pass owner_pass 256 --print=none --modify=none -- input.pdf encrypted.pdf + +# Check encryption status +qpdf --show-encryption encrypted.pdf + +# Remove password protection (requires password) +qpdf --password=secret123 --decrypt encrypted.pdf decrypted.pdf +``` + +## Advanced Python Techniques + +### pdfplumber Advanced Features + +#### Extract Text with Precise Coordinates +```python +import pdfplumber + +with pdfplumber.open("document.pdf") as pdf: + page = pdf.pages[0] + + # Extract all text with coordinates + chars = page.chars + for char in chars[:10]: # First 10 characters + print(f"Char: '{char['text']}' at x:{char['x0']:.1f} y:{char['y0']:.1f}") + + # Extract text by bounding box (left, top, right, bottom) + bbox_text = page.within_bbox((100, 100, 400, 200)).extract_text() +``` + +#### Advanced Table Extraction with Custom Settings +```python +import pdfplumber +import pandas as pd + +with pdfplumber.open("complex_table.pdf") as pdf: + page = pdf.pages[0] + + # Extract tables with custom settings for complex layouts + table_settings = { + "vertical_strategy": "lines", + "horizontal_strategy": "lines", + "snap_tolerance": 3, + "intersection_tolerance": 15 + } + tables = page.extract_tables(table_settings) + + # Visual debugging for table extraction + img = page.to_image(resolution=150) + img.save("debug_layout.png") +``` + +### reportlab Advanced Features + +#### Create Professional Reports with Tables +```python +from reportlab.platypus import SimpleDocTemplate, Table, TableStyle, Paragraph +from reportlab.lib.styles import getSampleStyleSheet +from reportlab.lib import colors + +# Sample data +data = [ + ['Product', 'Q1', 'Q2', 'Q3', 'Q4'], + ['Widgets', '120', '135', '142', '158'], + ['Gadgets', '85', '92', '98', '105'] +] + +# Create PDF with table +doc = SimpleDocTemplate("report.pdf") +elements = [] + +# Add title +styles = getSampleStyleSheet() +title = Paragraph("Quarterly Sales Report", styles['Title']) +elements.append(title) + +# Add table with advanced styling +table = Table(data) +table.setStyle(TableStyle([ + ('BACKGROUND', (0, 0), (-1, 0), colors.grey), + ('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke), + ('ALIGN', (0, 0), (-1, -1), 'CENTER'), + ('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'), + ('FONTSIZE', (0, 0), (-1, 0), 14), + ('BOTTOMPADDING', (0, 0), (-1, 0), 12), + ('BACKGROUND', (0, 1), (-1, -1), colors.beige), + ('GRID', (0, 0), (-1, -1), 1, colors.black) +])) +elements.append(table) + +doc.build(elements) +``` + +## Complex Workflows + +### Extract Figures/Images from PDF + +#### Method 1: Using pdfimages (fastest) +```bash +# Extract all images with original quality +pdfimages -all document.pdf images/img +``` + +#### Method 2: Using pypdfium2 + Image Processing +```python +import pypdfium2 as pdfium +from PIL import Image +import numpy as np + +def extract_figures(pdf_path, output_dir): + pdf = pdfium.PdfDocument(pdf_path) + + for page_num, page in enumerate(pdf): + # Render high-resolution page + bitmap = page.render(scale=3.0) + img = bitmap.to_pil() + + # Convert to numpy for processing + img_array = np.array(img) + + # Simple figure detection (non-white regions) + mask = np.any(img_array != [255, 255, 255], axis=2) + + # Find contours and extract bounding boxes + # (This is simplified - real implementation would need more sophisticated detection) + + # Save detected figures + # ... implementation depends on specific needs +``` + +### Batch PDF Processing with Error Handling +```python +import os +import glob +from pypdf import PdfReader, PdfWriter +import logging + +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +def batch_process_pdfs(input_dir, operation='merge'): + pdf_files = glob.glob(os.path.join(input_dir, "*.pdf")) + + if operation == 'merge': + writer = PdfWriter() + for pdf_file in pdf_files: + try: + reader = PdfReader(pdf_file) + for page in reader.pages: + writer.add_page(page) + logger.info(f"Processed: {pdf_file}") + except Exception as e: + logger.error(f"Failed to process {pdf_file}: {e}") + continue + + with open("batch_merged.pdf", "wb") as output: + writer.write(output) + + elif operation == 'extract_text': + for pdf_file in pdf_files: + try: + reader = PdfReader(pdf_file) + text = "" + for page in reader.pages: + text += page.extract_text() + + output_file = pdf_file.replace('.pdf', '.txt') + with open(output_file, 'w', encoding='utf-8') as f: + f.write(text) + logger.info(f"Extracted text from: {pdf_file}") + + except Exception as e: + logger.error(f"Failed to extract text from {pdf_file}: {e}") + continue +``` + +### Advanced PDF Cropping +```python +from pypdf import PdfWriter, PdfReader + +reader = PdfReader("input.pdf") +writer = PdfWriter() + +# Crop page (left, bottom, right, top in points) +page = reader.pages[0] +page.mediabox.left = 50 +page.mediabox.bottom = 50 +page.mediabox.right = 550 +page.mediabox.top = 750 + +writer.add_page(page) +with open("cropped.pdf", "wb") as output: + writer.write(output) +``` + +## Performance Optimization Tips + +### 1. For Large PDFs +- Use streaming approaches instead of loading entire PDF in memory +- Use `qpdf --split-pages` for splitting large files +- Process pages individually with pypdfium2 + +### 2. For Text Extraction +- `pdftotext -bbox-layout` is fastest for plain text extraction +- Use pdfplumber for structured data and tables +- Avoid `pypdf.extract_text()` for very large documents + +### 3. For Image Extraction +- `pdfimages` is much faster than rendering pages +- Use low resolution for previews, high resolution for final output + +### 4. For Form Filling +- pdf-lib maintains form structure better than most alternatives +- Pre-validate form fields before processing + +### 5. Memory Management +```python +# Process PDFs in chunks +def process_large_pdf(pdf_path, chunk_size=10): + reader = PdfReader(pdf_path) + total_pages = len(reader.pages) + + for start_idx in range(0, total_pages, chunk_size): + end_idx = min(start_idx + chunk_size, total_pages) + writer = PdfWriter() + + for i in range(start_idx, end_idx): + writer.add_page(reader.pages[i]) + + # Process chunk + with open(f"chunk_{start_idx//chunk_size}.pdf", "wb") as output: + writer.write(output) +``` + +## Troubleshooting Common Issues + +### Encrypted PDFs +```python +# Handle password-protected PDFs +from pypdf import PdfReader + +try: + reader = PdfReader("encrypted.pdf") + if reader.is_encrypted: + reader.decrypt("password") +except Exception as e: + print(f"Failed to decrypt: {e}") +``` + +### Corrupted PDFs +```bash +# Use qpdf to repair +qpdf --check corrupted.pdf +qpdf --replace-input corrupted.pdf +``` + +### Text Extraction Issues +```python +# Fallback to OCR for scanned PDFs +import pytesseract +from pdf2image import convert_from_path + +def extract_text_with_ocr(pdf_path): + images = convert_from_path(pdf_path) + text = "" + for i, image in enumerate(images): + text += pytesseract.image_to_string(image) + return text +``` + +## License Information + +- **pypdf**: BSD License +- **pdfplumber**: MIT License +- **pypdfium2**: Apache/BSD License +- **reportlab**: BSD License +- **poppler-utils**: GPL-2 License +- **qpdf**: Apache License +- **pdf-lib**: MIT License +- **pdfjs-dist**: Apache License \ No newline at end of file diff --git a/pdf/scripts/check_bounding_boxes.py b/pdf/scripts/check_bounding_boxes.py new file mode 100644 index 0000000..b9c6ea7 --- /dev/null +++ b/pdf/scripts/check_bounding_boxes.py @@ -0,0 +1,65 @@ +from dataclasses import dataclass +import json +import sys + + + + +@dataclass +class RectAndField: + rect: list[float] + rect_type: str + field: dict + + +def get_bounding_box_messages(fields_json_stream) -> list[str]: + messages = [] + fields = json.load(fields_json_stream) + messages.append(f"Read {len(fields['form_fields'])} fields") + + def rects_intersect(r1, r2): + disjoint_horizontal = r1[0] >= r2[2] or r1[2] <= r2[0] + disjoint_vertical = r1[1] >= r2[3] or r1[3] <= r2[1] + return not (disjoint_horizontal or disjoint_vertical) + + rects_and_fields = [] + for f in fields["form_fields"]: + rects_and_fields.append(RectAndField(f["label_bounding_box"], "label", f)) + rects_and_fields.append(RectAndField(f["entry_bounding_box"], "entry", f)) + + has_error = False + for i, ri in enumerate(rects_and_fields): + for j in range(i + 1, len(rects_and_fields)): + rj = rects_and_fields[j] + if ri.field["page_number"] == rj.field["page_number"] and rects_intersect(ri.rect, rj.rect): + has_error = True + if ri.field is rj.field: + messages.append(f"FAILURE: intersection between label and entry bounding boxes for `{ri.field['description']}` ({ri.rect}, {rj.rect})") + else: + messages.append(f"FAILURE: intersection between {ri.rect_type} bounding box for `{ri.field['description']}` ({ri.rect}) and {rj.rect_type} bounding box for `{rj.field['description']}` ({rj.rect})") + if len(messages) >= 20: + messages.append("Aborting further checks; fix bounding boxes and try again") + return messages + if ri.rect_type == "entry": + if "entry_text" in ri.field: + font_size = ri.field["entry_text"].get("font_size", 14) + entry_height = ri.rect[3] - ri.rect[1] + if entry_height < font_size: + has_error = True + messages.append(f"FAILURE: entry bounding box height ({entry_height}) for `{ri.field['description']}` is too short for the text content (font size: {font_size}). Increase the box height or decrease the font size.") + if len(messages) >= 20: + messages.append("Aborting further checks; fix bounding boxes and try again") + return messages + + if not has_error: + messages.append("SUCCESS: All bounding boxes are valid") + return messages + +if __name__ == "__main__": + if len(sys.argv) != 2: + print("Usage: check_bounding_boxes.py [fields.json]") + sys.exit(1) + with open(sys.argv[1], encoding="utf-8") as f: + messages = get_bounding_box_messages(f) + for msg in messages: + print(msg) diff --git a/pdf/scripts/check_fillable_fields.py b/pdf/scripts/check_fillable_fields.py new file mode 100644 index 0000000..36dfb95 --- /dev/null +++ b/pdf/scripts/check_fillable_fields.py @@ -0,0 +1,11 @@ +import sys +from pypdf import PdfReader + + + + +reader = PdfReader(sys.argv[1]) +if (reader.get_fields()): + print("This PDF has fillable form fields") +else: + print("This PDF does not have fillable form fields; you will need to visually determine where to enter data") diff --git a/pdf/scripts/convert_pdf_to_images.py b/pdf/scripts/convert_pdf_to_images.py new file mode 100644 index 0000000..7939cef --- /dev/null +++ b/pdf/scripts/convert_pdf_to_images.py @@ -0,0 +1,33 @@ +import os +import sys + +from pdf2image import convert_from_path + + + + +def convert(pdf_path, output_dir, max_dim=1000): + images = convert_from_path(pdf_path, dpi=200) + + for i, image in enumerate(images): + width, height = image.size + if width > max_dim or height > max_dim: + scale_factor = min(max_dim / width, max_dim / height) + new_width = int(width * scale_factor) + new_height = int(height * scale_factor) + image = image.resize((new_width, new_height)) + + image_path = os.path.join(output_dir, f"page_{i+1}.png") + image.save(image_path) + print(f"Saved page {i+1} as {image_path} (size: {image.size})") + + print(f"Converted {len(images)} pages to PNG images") + + +if __name__ == "__main__": + if len(sys.argv) != 3: + print("Usage: convert_pdf_to_images.py [input pdf] [output directory]") + sys.exit(1) + pdf_path = sys.argv[1] + output_directory = sys.argv[2] + convert(pdf_path, output_directory) diff --git a/pdf/scripts/create_validation_image.py b/pdf/scripts/create_validation_image.py new file mode 100644 index 0000000..ce98a85 --- /dev/null +++ b/pdf/scripts/create_validation_image.py @@ -0,0 +1,37 @@ +import json +import sys + +from PIL import Image, ImageDraw + + + + +def create_validation_image(page_number, fields_json_path, input_path, output_path): + with open(fields_json_path, 'r', encoding='utf-8') as f: + data = json.load(f) + + img = Image.open(input_path) + draw = ImageDraw.Draw(img) + num_boxes = 0 + + for field in data["form_fields"]: + if field["page_number"] == page_number: + entry_box = field['entry_bounding_box'] + label_box = field['label_bounding_box'] + draw.rectangle(entry_box, outline='red', width=2) + draw.rectangle(label_box, outline='blue', width=2) + num_boxes += 2 + + img.save(output_path) + print(f"Created validation image at {output_path} with {num_boxes} bounding boxes") + + +if __name__ == "__main__": + if len(sys.argv) != 5: + print("Usage: create_validation_image.py [page number] [fields.json file] [input image path] [output image path]") + sys.exit(1) + page_number = int(sys.argv[1]) + fields_json_path = sys.argv[2] + input_image_path = sys.argv[3] + output_image_path = sys.argv[4] + create_validation_image(page_number, fields_json_path, input_image_path, output_image_path) diff --git a/pdf/scripts/extract_form_field_info.py b/pdf/scripts/extract_form_field_info.py new file mode 100644 index 0000000..64cd470 --- /dev/null +++ b/pdf/scripts/extract_form_field_info.py @@ -0,0 +1,122 @@ +import json +import sys + +from pypdf import PdfReader + + + + +def get_full_annotation_field_id(annotation): + components = [] + while annotation: + field_name = annotation.get('/T') + if field_name: + components.append(field_name) + annotation = annotation.get('/Parent') + return ".".join(reversed(components)) if components else None + + +def make_field_dict(field, field_id): + field_dict = {"field_id": field_id} + ft = field.get('/FT') + if ft == "/Tx": + field_dict["type"] = "text" + elif ft == "/Btn": + field_dict["type"] = "checkbox" + states = field.get("/_States_", []) + if len(states) == 2: + if "/Off" in states: + field_dict["checked_value"] = states[0] if states[0] != "/Off" else states[1] + field_dict["unchecked_value"] = "/Off" + else: + print(f"Unexpected state values for checkbox `${field_id}`. Its checked and unchecked values may not be correct; if you're trying to check it, visually verify the results.") + field_dict["checked_value"] = states[0] + field_dict["unchecked_value"] = states[1] + elif ft == "/Ch": + field_dict["type"] = "choice" + states = field.get("/_States_", []) + field_dict["choice_options"] = [{ + "value": state[0], + "text": state[1], + } for state in states] + else: + field_dict["type"] = f"unknown ({ft})" + return field_dict + + +def get_field_info(reader: PdfReader): + fields = reader.get_fields() + + field_info_by_id = {} + possible_radio_names = set() + + for field_id, field in fields.items(): + if field.get("/Kids"): + if field.get("/FT") == "/Btn": + possible_radio_names.add(field_id) + continue + field_info_by_id[field_id] = make_field_dict(field, field_id) + + + radio_fields_by_id = {} + + for page_index, page in enumerate(reader.pages): + annotations = page.get('/Annots', []) + for ann in annotations: + field_id = get_full_annotation_field_id(ann) + if field_id in field_info_by_id: + field_info_by_id[field_id]["page"] = page_index + 1 + field_info_by_id[field_id]["rect"] = ann.get('/Rect') + elif field_id in possible_radio_names: + try: + on_values = [v for v in ann["/AP"]["/N"] if v != "/Off"] + except KeyError: + continue + if len(on_values) == 1: + rect = ann.get("/Rect") + if field_id not in radio_fields_by_id: + radio_fields_by_id[field_id] = { + "field_id": field_id, + "type": "radio_group", + "page": page_index + 1, + "radio_options": [], + } + radio_fields_by_id[field_id]["radio_options"].append({ + "value": on_values[0], + "rect": rect, + }) + + fields_with_location = [] + for field_info in field_info_by_id.values(): + if "page" in field_info: + fields_with_location.append(field_info) + else: + print(f"Unable to determine location for field id: {field_info.get('field_id')}, ignoring") + + def sort_key(f): + if "radio_options" in f: + rect = f["radio_options"][0]["rect"] or [0, 0, 0, 0] + else: + rect = f.get("rect") or [0, 0, 0, 0] + adjusted_position = [-rect[1], rect[0]] + return [f.get("page"), adjusted_position] + + sorted_fields = fields_with_location + list(radio_fields_by_id.values()) + sorted_fields.sort(key=sort_key) + + return sorted_fields + + +def write_field_info(pdf_path: str, json_output_path: str): + reader = PdfReader(pdf_path) + field_info = get_field_info(reader) + with open(json_output_path, "w") as f: + json.dump(field_info, f, indent=2) + print(f"Wrote {len(field_info)} fields to {json_output_path}") + + +if __name__ == "__main__": + if len(sys.argv) != 3: + print("Usage: extract_form_field_info.py [input pdf] [output json]") + sys.exit(1) + write_field_info(sys.argv[1], sys.argv[2]) diff --git a/pdf/scripts/extract_form_structure.py b/pdf/scripts/extract_form_structure.py new file mode 100755 index 0000000..f219e7d --- /dev/null +++ b/pdf/scripts/extract_form_structure.py @@ -0,0 +1,115 @@ +""" +Extract form structure from a non-fillable PDF. + +This script analyzes the PDF to find: +- Text labels with their exact coordinates +- Horizontal lines (row boundaries) +- Checkboxes (small rectangles) + +Output: A JSON file with the form structure that can be used to generate +accurate field coordinates for filling. + +Usage: python extract_form_structure.py +""" + +import json +import sys +import pdfplumber + + +def extract_form_structure(pdf_path): + structure = { + "pages": [], + "labels": [], + "lines": [], + "checkboxes": [], + "row_boundaries": [] + } + + with pdfplumber.open(pdf_path) as pdf: + for page_num, page in enumerate(pdf.pages, 1): + structure["pages"].append({ + "page_number": page_num, + "width": float(page.width), + "height": float(page.height) + }) + + words = page.extract_words() + for word in words: + structure["labels"].append({ + "page": page_num, + "text": word["text"], + "x0": round(float(word["x0"]), 1), + "top": round(float(word["top"]), 1), + "x1": round(float(word["x1"]), 1), + "bottom": round(float(word["bottom"]), 1) + }) + + for line in page.lines: + if abs(float(line["x1"]) - float(line["x0"])) > page.width * 0.5: + structure["lines"].append({ + "page": page_num, + "y": round(float(line["top"]), 1), + "x0": round(float(line["x0"]), 1), + "x1": round(float(line["x1"]), 1) + }) + + for rect in page.rects: + width = float(rect["x1"]) - float(rect["x0"]) + height = float(rect["bottom"]) - float(rect["top"]) + if 5 <= width <= 15 and 5 <= height <= 15 and abs(width - height) < 2: + structure["checkboxes"].append({ + "page": page_num, + "x0": round(float(rect["x0"]), 1), + "top": round(float(rect["top"]), 1), + "x1": round(float(rect["x1"]), 1), + "bottom": round(float(rect["bottom"]), 1), + "center_x": round((float(rect["x0"]) + float(rect["x1"])) / 2, 1), + "center_y": round((float(rect["top"]) + float(rect["bottom"])) / 2, 1) + }) + + lines_by_page = {} + for line in structure["lines"]: + page = line["page"] + if page not in lines_by_page: + lines_by_page[page] = [] + lines_by_page[page].append(line["y"]) + + for page, y_coords in lines_by_page.items(): + y_coords = sorted(set(y_coords)) + for i in range(len(y_coords) - 1): + structure["row_boundaries"].append({ + "page": page, + "row_top": y_coords[i], + "row_bottom": y_coords[i + 1], + "row_height": round(y_coords[i + 1] - y_coords[i], 1) + }) + + return structure + + +def main(): + if len(sys.argv) != 3: + print("Usage: extract_form_structure.py ") + sys.exit(1) + + pdf_path = sys.argv[1] + output_path = sys.argv[2] + + print(f"Extracting structure from {pdf_path}...") + structure = extract_form_structure(pdf_path) + + with open(output_path, "w") as f: + json.dump(structure, f, indent=2) + + print(f"Found:") + print(f" - {len(structure['pages'])} pages") + print(f" - {len(structure['labels'])} text labels") + print(f" - {len(structure['lines'])} horizontal lines") + print(f" - {len(structure['checkboxes'])} checkboxes") + print(f" - {len(structure['row_boundaries'])} row boundaries") + print(f"Saved to {output_path}") + + +if __name__ == "__main__": + main() diff --git a/pdf/scripts/fill_fillable_fields.py b/pdf/scripts/fill_fillable_fields.py new file mode 100644 index 0000000..7bc4b1c --- /dev/null +++ b/pdf/scripts/fill_fillable_fields.py @@ -0,0 +1,98 @@ +import json +import sys + +from pypdf import PdfReader, PdfWriter + +from extract_form_field_info import get_field_info + + + + +def fill_pdf_fields(input_pdf_path: str, fields_json_path: str, output_pdf_path: str): + with open(fields_json_path, encoding="utf-8") as f: + fields = json.load(f) + fields_by_page = {} + for field in fields: + if "value" in field: + field_id = field["field_id"] + page = field["page"] + if page not in fields_by_page: + fields_by_page[page] = {} + fields_by_page[page][field_id] = field["value"] + + reader = PdfReader(input_pdf_path) + + has_error = False + field_info = get_field_info(reader) + fields_by_ids = {f["field_id"]: f for f in field_info} + for field in fields: + existing_field = fields_by_ids.get(field["field_id"]) + if not existing_field: + has_error = True + print(f"ERROR: `{field['field_id']}` is not a valid field ID") + elif field["page"] != existing_field["page"]: + has_error = True + print(f"ERROR: Incorrect page number for `{field['field_id']}` (got {field['page']}, expected {existing_field['page']})") + else: + if "value" in field: + err = validation_error_for_field_value(existing_field, field["value"]) + if err: + print(err) + has_error = True + if has_error: + sys.exit(1) + + writer = PdfWriter(clone_from=reader) + for page, field_values in fields_by_page.items(): + writer.update_page_form_field_values(writer.pages[page - 1], field_values, auto_regenerate=False) + + writer.set_need_appearances_writer(True) + + with open(output_pdf_path, "wb") as f: + writer.write(f) + + +def validation_error_for_field_value(field_info, field_value): + field_type = field_info["type"] + field_id = field_info["field_id"] + if field_type == "checkbox": + checked_val = field_info["checked_value"] + unchecked_val = field_info["unchecked_value"] + if field_value != checked_val and field_value != unchecked_val: + return f'ERROR: Invalid value "{field_value}" for checkbox field "{field_id}". The checked value is "{checked_val}" and the unchecked value is "{unchecked_val}"' + elif field_type == "radio_group": + option_values = [opt["value"] for opt in field_info["radio_options"]] + if field_value not in option_values: + return f'ERROR: Invalid value "{field_value}" for radio group field "{field_id}". Valid values are: {option_values}' + elif field_type == "choice": + choice_values = [opt["value"] for opt in field_info["choice_options"]] + if field_value not in choice_values: + return f'ERROR: Invalid value "{field_value}" for choice field "{field_id}". Valid values are: {choice_values}' + return None + + +def monkeypatch_pydpf_method(): + from pypdf.generic import DictionaryObject + from pypdf.constants import FieldDictionaryAttributes + + original_get_inherited = DictionaryObject.get_inherited + + def patched_get_inherited(self, key: str, default = None): + result = original_get_inherited(self, key, default) + if key == FieldDictionaryAttributes.Opt: + if isinstance(result, list) and all(isinstance(v, list) and len(v) == 2 for v in result): + result = [r[0] for r in result] + return result + + DictionaryObject.get_inherited = patched_get_inherited + + +if __name__ == "__main__": + if len(sys.argv) != 4: + print("Usage: fill_fillable_fields.py [input pdf] [field_values.json] [output pdf]") + sys.exit(1) + monkeypatch_pydpf_method() + input_pdf = sys.argv[1] + fields_json = sys.argv[2] + output_pdf = sys.argv[3] + fill_pdf_fields(input_pdf, fields_json, output_pdf) diff --git a/pdf/scripts/fill_pdf_form_with_annotations.py b/pdf/scripts/fill_pdf_form_with_annotations.py new file mode 100644 index 0000000..204e940 --- /dev/null +++ b/pdf/scripts/fill_pdf_form_with_annotations.py @@ -0,0 +1,107 @@ +import json +import sys + +from pypdf import PdfReader, PdfWriter +from pypdf.annotations import FreeText + + + + +def transform_from_image_coords(bbox, image_width, image_height, pdf_width, pdf_height): + x_scale = pdf_width / image_width + y_scale = pdf_height / image_height + + left = bbox[0] * x_scale + right = bbox[2] * x_scale + + top = pdf_height - (bbox[1] * y_scale) + bottom = pdf_height - (bbox[3] * y_scale) + + return left, bottom, right, top + + +def transform_from_pdf_coords(bbox, pdf_height): + left = bbox[0] + right = bbox[2] + + pypdf_top = pdf_height - bbox[1] + pypdf_bottom = pdf_height - bbox[3] + + return left, pypdf_bottom, right, pypdf_top + + +def fill_pdf_form(input_pdf_path, fields_json_path, output_pdf_path): + + with open(fields_json_path, "r", encoding="utf-8") as f: + fields_data = json.load(f) + + reader = PdfReader(input_pdf_path) + writer = PdfWriter() + + writer.append(reader) + + pdf_dimensions = {} + for i, page in enumerate(reader.pages): + mediabox = page.mediabox + pdf_dimensions[i + 1] = [mediabox.width, mediabox.height] + + annotations = [] + for field in fields_data["form_fields"]: + page_num = field["page_number"] + + page_info = next(p for p in fields_data["pages"] if p["page_number"] == page_num) + pdf_width, pdf_height = pdf_dimensions[page_num] + + if "pdf_width" in page_info: + transformed_entry_box = transform_from_pdf_coords( + field["entry_bounding_box"], + float(pdf_height) + ) + else: + image_width = page_info["image_width"] + image_height = page_info["image_height"] + transformed_entry_box = transform_from_image_coords( + field["entry_bounding_box"], + image_width, image_height, + float(pdf_width), float(pdf_height) + ) + + if "entry_text" not in field or "text" not in field["entry_text"]: + continue + entry_text = field["entry_text"] + text = entry_text["text"] + if not text: + continue + + font_name = entry_text.get("font", "Arial") + font_size = str(entry_text.get("font_size", 14)) + "pt" + font_color = entry_text.get("font_color", "000000") + + annotation = FreeText( + text=text, + rect=transformed_entry_box, + font=font_name, + font_size=font_size, + font_color=font_color, + border_color=None, + background_color=None, + ) + annotations.append(annotation) + writer.add_annotation(page_number=page_num - 1, annotation=annotation) + + with open(output_pdf_path, "wb") as output: + writer.write(output) + + print(f"Successfully filled PDF form and saved to {output_pdf_path}") + print(f"Added {len(annotations)} text annotations") + + +if __name__ == "__main__": + if len(sys.argv) != 4: + print("Usage: fill_pdf_form_with_annotations.py [input pdf] [fields.json] [output pdf]") + sys.exit(1) + input_pdf = sys.argv[1] + fields_json = sys.argv[2] + output_pdf = sys.argv[3] + + fill_pdf_form(input_pdf, fields_json, output_pdf) diff --git a/powerpoint/LICENSE.txt b/powerpoint/LICENSE.txt new file mode 100644 index 0000000..c55ab42 --- /dev/null +++ b/powerpoint/LICENSE.txt @@ -0,0 +1,30 @@ +© 2025 Anthropic, PBC. All rights reserved. + +LICENSE: Use of these materials (including all code, prompts, assets, files, +and other components of this Skill) is governed by your agreement with +Anthropic regarding use of Anthropic's services. If no separate agreement +exists, use is governed by Anthropic's Consumer Terms of Service or +Commercial Terms of Service, as applicable: +https://www.anthropic.com/legal/consumer-terms +https://www.anthropic.com/legal/commercial-terms +Your applicable agreement is referred to as the "Agreement." "Services" are +as defined in the Agreement. + +ADDITIONAL RESTRICTIONS: Notwithstanding anything in the Agreement to the +contrary, users may not: + +- Extract these materials from the Services or retain copies of these + materials outside the Services +- Reproduce or copy these materials, except for temporary copies created + automatically during authorized use of the Services +- Create derivative works based on these materials +- Distribute, sublicense, or transfer these materials to any third party +- Make, offer to sell, sell, or import any inventions embodied in these + materials +- Reverse engineer, decompile, or disassemble these materials + +The receipt, viewing, or possession of these materials does not convey or +imply any license or right beyond those expressly granted above. + +Anthropic retains all right, title, and interest in these materials, +including all copyrights, patents, and other intellectual property rights. diff --git a/powerpoint/SKILL.md b/powerpoint/SKILL.md new file mode 100644 index 0000000..0292bc6 --- /dev/null +++ b/powerpoint/SKILL.md @@ -0,0 +1,256 @@ +--- +name: powerpoint +description: "Create, read, edit .pptx decks, slides, notes, templates." +version: 2.0.0 +author: Anthropic (adapted by Nous Research) +license: Proprietary. LICENSE.txt has complete terms +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [PowerPoint, PPTX, Presentations, Office, Productivity] + category: productivity + related_skills: [docx, xlsx, pdf] +--- + +# Powerpoint Skill + +Create, read, and edit PowerPoint decks — from-scratch generation with pptxgenjs, template-based editing via direct XML manipulation, speaker notes, charts, and design QA. A `.pptx` is a ZIP archive of XML files. + +## When to Use + +Use this skill any time a .pptx or .potx file is involved in any way — as input, output, or both: creating slide decks, pitch decks, or presentations; reading or extracting text from any .pptx; editing existing presentations; combining or splitting slide files; working with templates (.potx), layouts, speaker notes, or comments. Trigger whenever the user mentions "deck," "slides," "presentation," or references a .pptx/.potx filename. + +## Prerequisites + +```bash +npm ls pptxgenjs --depth=0 2>/dev/null | grep -q pptxgenjs || npm install pptxgenjs +pip install "markitdown[pptx]" Pillow defusedxml lxml +which soffice || sudo apt install -y libreoffice # rendering/QA +which pdftoppm || sudo apt install -y poppler-utils # PDF → images +``` + +macOS: `brew install libreoffice poppler`. Icons in generated decks additionally use `react-icons react react-dom sharp` (npm). + +## Quick Reference + +| Task | Approach | +|---|---| +| **Create** a new deck | Write a `pptxgenjs` script — see gotchas below | +| **Edit** an existing deck, or build from a template | unzip → edit `ppt/slides/slideN.xml` → zip | +| **Read** content | `markitdown deck.pptx` (one block per slide under `` markers); visual grid: `python scripts/thumbnail.py deck.pptx` | + +## Scripts + +Paths are relative to this skill's directory. Everything else is plain Python, `node`, or shell. + +| Script | What it does | +|---|---| +| `scripts/thumbnail.py deck.pptx [prefix]` | Labeled grid of every slide, for picking template layouts. `.pptx` only. Pass `prefix` — it defaults to `thumbnails`, which overwrites the grids of any other deck done in the same directory | +| `scripts/add_slide.py unpacked/ slide2.xml [--after slideN.xml]` | Duplicate a slide (or a `slideLayoutN.xml`) with all the package bookkeeping. Also takes a `.pptx` directly with `-o out.pptx` | +| `scripts/clean.py unpacked/` | Delete slides, media, and rels no longer referenced. Run **after** `` is final | +| `scripts/office/validate.py deck.pptx [--original src.pptx]` | Schema, relationship, content-type, chart and slide checks; each failure names its fix. Pass `--original` for any template-derived deck — it baselines the schema checks against the template, so the template's own XSD errors don't read as yours | +| `scripts/office/soffice.py --headless --convert-to pdf deck.pptx` | LibreOffice wrapper — bare `soffice` hangs in sandboxed environments | + +## Creating with pptxgenjs — gotchas + +Write the script and `require('pptxgenjs')`. The model knows the API; these are the footguns: + +- **Set `pres.layout` before adding slides.** The default canvas is `LAYOUT_16x9` = **10" × 5.625"**, not 13.3" wide. Coordinates past the edge are written, not clamped — the shape just isn't on the slide. (`LAYOUT_WIDE` is 13.3" × 7.5".) +- **Hex colors: never `#`, never 8 digits.** `color: "FF0000"`. Both `"#FF0000"` and alpha baked into the hex (`"00000020"`) **corrupt the file**. For translucency: `transparency: 0-100` on fills and images, `opacity: 0.0-1.0` on shadows — each is silently ignored on the other. +- **pptxgenjs mutates option objects in place** (converts values to EMU on first use). Never share one `shadow`/options object across two `add*` calls — build a fresh object each time. +- **Shadow `offset` must be ≥ 0** — a negative offset corrupts the file. To cast a shadow upward, use `angle: 270` with a positive offset. +- **`letterSpacing` is silently ignored** — the real option is `charSpacing`. +- **Lists:** `bullet: true` on each item, never a literal `•` (renders double bullets). Set `breakLine: true` on every array item except the last. Space bulleted paragraphs with `paraSpaceAfter`, not `lineSpacing` (huge gaps). +- **One `new pptxgen()` per output file** — never reuse an instance. +- **`rectRadius` only works on `ROUNDED_RECTANGLE`**, not `RECTANGLE`. +- **Gradient fills aren't supported** — use a gradient image as the background instead. +- **Text boxes have built-in internal padding** — set `margin: 0` whenever text must align with a shape, line, or icon at the same x. +- **Speaker notes go in `slide.addNotes("...")`** (plain text, once per slide), never in a text box on the slide. +- **Keep charts native.** Use `addChart()` for everything PowerPoint can chart (pass an array of `{type, data, options}` for combos). For PowerPoint-native features the library doesn't expose (trendlines, error bars), compute the extra series yourself or post-process the generated OOXML — do not fall back to a rendered image. Only chart types PowerPoint has no native form for (Sankey, network, chord) go in as images. +- **Default charts render bare** — no title, no data labels, dated palette. Set `showTitle` + `title`, `showValue: true` + `dataLabelPosition`, `chartColors: [...]` from your palette, and quiet the frame (`catAxisLabelColor`/`valAxisLabelColor`, `valGridLine: { color, size }`, `catGridLine: { style: "none" }`, `showLegend: false` for a single series). +- **On a stacked bar or column chart, `dataLabelPosition` must be `ctr`, `inEnd`, or `inBase`.** `outEnd` **corrupts the file**. +- **A combo series using `secondaryValAxis`/`secondaryCatAxis` needs both `valAxes` and `catAxes` on the chart options, two entries each.** Without them pptxgenjs writes axis *ids* it never declares, and PowerPoint **discards that chart** and reports the file as corrupt. Supplying only `valAxes` is not enough. +- **After `writeFile()`, run `python scripts/office/validate.py deck.pptx`.** It reports the two chart faults above and the slide-XML defects PowerPoint refuses, and names the fix for each. Fix them in your generator, not by hand-editing the packed XML. +- **Never reorder the children of ``.** pptxgenjs writes `` right after `` and points both masters at one theme part. PowerPoint reads that happily — move the element and the same deck becomes unopenable. +- **Icons:** render `react-icons` to SVG (`ReactDOMServer.renderToStaticMarkup`), rasterize with `sharp` at ≥256px, and insert via `addImage({ data: "image/png;base64," + buf.toString("base64") })` — the `image/png;base64,` prefix is required. + +## Editing existing decks and templates + +Pick layouts first: `python scripts/thumbnail.py template.pptx template-thumbs` writes a labeled grid of every slide and prints the file(s) it created — `template-thumbs.jpg`, split into `template-thumbs-N.jpg` past 12 slides. **Always pass that second argument, named after the deck.** It defaults to `thumbnails`, so two decks thumbnailed in one directory silently overwrite each other's grids (template analysis only — visual QA needs the full-resolution renders from [Converting to Images](#converting-to-images); it only accepts `.pptx`, so copy a `.potx` to a `.pptx` name first). Use it with `markitdown` to map each content section onto a template slide, and vary the layouts — don't put every section on the same title-and-bullets slide. + +```bash +python3 -c "import sys,zipfile; zipfile.ZipFile(sys.argv[1]).extractall('unpacked')" deck.pptx +python scripts/add_slide.py unpacked/ slide2.xml --after slide2.xml # duplicate a slide (or slideLayoutN.xml); prints the new slide's path +# reorder / delete slides = edit in ppt/presentation.xml +python scripts/clean.py unpacked/ # after deletions: removes orphaned slides, media, rels +# edit slide content in ppt/slides/slideN.xml +(cd unpacked && rm -f ../out.pptx && zip -Xr ../out.pptx .) # zip from INSIDE the dir; rm first or deleted parts survive +python scripts/office/validate.py out.pptx --original deck.pptx +``` + +- **Do all structural work — add, delete, reorder — before editing any slide's content.** `add_slide.py` copies a slide file verbatim, so duplicating after you edit clones the edited content; and `clean.py` deletes any slide missing from ``, including one you just wrote. +- **Never copy a slide file by hand** — `add_slide.py` does every registration a new slide needs and reports what it made. It also works directly on a file: `add_slide.py deck.pptx slide2.xml -o out.pptx` — **pass `-o`, or it rewrites the input deck in place.** A duplicated slide still *references* its source's chart/SmartArt/embedded-object parts rather than cloning them, so editing one slide's chart changes the other's. +- **If you use `python-pptx`**, three things it won't do: duplicate a slide (its only entry point is `add_slide(layout)`), preserve formatting through `text_frame.text = "..."` (that collapses the paragraph to a single unstyled run — assign `run.text` instead), or read the SVG/EMF most template art uses (`add_picture` raises `UnidentifiedImageError`). +- Legacy `.ppt` must be converted first: `python scripts/office/soffice.py --headless --convert-to pptx file.ppt`. `.potx` templates unpack and pack identically — keep the `.potx` extension on the output. +- To reuse a template icon or image, duplicate a slide or layout that already contains it. + +When filling in a template: + +- If you script an XML transform, parse with `defusedxml.minidom` — round-tripping OOXML through `xml.etree.ElementTree` rewrites namespace prefixes and corrupts the deck. +- **Template slots ≠ source items.** If the template shows 4 team members and you have 3, delete the 4th member's entire group (image + text boxes), not just its text — then check for orphaned visuals in QA. +- One `` per list item — never concatenate items into a single paragraph. Copy the sibling `` to preserve spacing, and put `b="1"` on the `` of titles, section headers, and inline labels (`Status:`, `Owner:`). +- Let bullets inherit from the layout; only add ``, `` (numbered), or `` to override — never a literal `•` in the text. +- Text with leading or trailing spaces needs `xml:space="preserve"` on its ``. + +## Design Ideas + +**Don't create boring slides.** Plain bullets on a white background won't impress anyone. Consider ideas from this list for each slide. + +### Before Starting + +- **Pick a bold, content-informed color palette**: The palette should feel designed for THIS topic. If swapping your colors into a completely different presentation would still "work," you haven't made specific enough choices. +- **Dominance over equality**: One color should dominate (60-70% visual weight), with 1-2 supporting tones and one sharp accent. Never give all colors equal weight. +- **Dark/light contrast**: Dark backgrounds for title + conclusion slides, light for content ("sandwich" structure). Or commit to dark throughout for a premium feel. +- **Commit to a visual motif**: Pick ONE distinctive element and repeat it — rounded image frames, icons in colored circles. Carry it across every slide. **Do not use a color bar or accent stripe as your motif** (see Avoid list). + +### Color Palettes + +Choose colors that match your topic — don't default to generic blue. Use these palettes as inspiration: + +| Theme | Primary | Secondary | Accent | +|-------|---------|-----------|--------| +| **Midnight Executive** | `1E2761` (navy) | `CADCFC` (ice blue) | `FFFFFF` (white) | +| **Forest & Moss** | `2C5F2D` (forest) | `97BC62` (moss) | `F5F5F5` (cream) | +| **Coral Energy** | `F96167` (coral) | `F9E795` (gold) | `2F3C7E` (navy) | +| **Warm Terracotta** | `B85042` (terracotta) | `E7E8D1` (sand) | `A7BEAE` (sage) | +| **Ocean Gradient** | `065A82` (deep blue) | `1C7293` (teal) | `21295C` (midnight) | +| **Charcoal Minimal** | `36454F` (charcoal) | `F2F2F2` (off-white) | `212121` (black) | +| **Teal Trust** | `028090` (teal) | `00A896` (seafoam) | `02C39A` (mint) | +| **Berry & Cream** | `6D2E46` (berry) | `A26769` (dusty rose) | `ECE2D0` (cream) | +| **Sage Calm** | `84B59F` (sage) | `69A297` (eucalyptus) | `50808E` (slate) | +| **Cherry Bold** | `990011` (cherry) | `FCF6F5` (off-white) | `2F3C7E` (navy) | + +### For Each Slide + +**Every slide needs a visual element** — image, chart, icon, or shape. Text-only slides are forgettable. + +**Layout options:** +- Two-column (text left, illustration on right) +- Icon + text rows (icon in colored circle, bold header, description below) +- 2x2 or 2x3 grid (image on one side, grid of content blocks on other) +- Half-bleed image (full left or right side) with content overlay + +**Data display:** +- Large stat callouts (big numbers 60-72pt with small labels below) +- Comparison columns (before/after, pros/cons, side-by-side options) +- Timeline or process flow (numbered steps, arrows) + +**Visual polish:** +- Icons in small colored circles next to section headers +- Italic accent text for key stats or taglines + +### Typography + +**Font names you write into the .pptx are rendered by the user's PowerPoint, not by this environment.** Your visual QA renders via LibreOffice, which substitutes fonts it doesn't have — and for some fonts the substitute has different widths, so your QA preview can show text overflow (or fit) that the real deck won't have. To keep your QA trustworthy: + +- **Safe fonts** (render true-to-width in QA *and* ship with Office): **Arial, Calibri, Cambria, Times New Roman, Courier New, Bookman Old Style, Century Schoolbook**. Use these for body text and anything where fit matters. +- **Headers with personality at zero QA risk**: pair a safe-list serif header (Cambria, Bookman Old Style, Century Schoolbook) with a safe-list sans body (Calibri or Arial). +- **If the user asks for a font outside the safe list** (e.g. Georgia or Trebuchet MS): use it where the user asked, but size those containers with extra slack (~10%) and don't trust QA text-fit on those elements. +- **QA-unreliable fonts** (substitute has different widths — overflow checks can be wrong): Georgia, Trebuchet MS, Impact, Arial Black, Garamond, Consolas, Palatino Linotype. Calibri Light substitution varies by environment; treat as QA-unreliable. +- **Never default to Aptos** — Office's post-2023 default has no metric-compatible substitute here *and* is missing from older Office installs, so it's unreliable on both ends. + +| Element | Size | +|---------|------| +| Slide title | 36-44pt bold | +| Section header | 20-24pt bold | +| Body text | 14-16pt | +| Captions | 10-12pt muted | + +### Spacing + +- 0.5" minimum margins +- 0.3-0.5" between content blocks +- Leave breathing room—don't fill every inch + +### Avoid (Common Mistakes) + +- **Don't repeat the same layout** — vary columns, cards, and callouts across slides +- **Don't center body text** — left-align paragraphs and lists; center only titles +- **Don't skimp on size contrast** — titles need 36pt+ to stand out from 14-16pt body +- **Don't default to blue** — pick colors that reflect the specific topic +- **Don't mix spacing randomly** — choose 0.3" or 0.5" gaps and use consistently +- **Don't style one slide and leave the rest plain** — commit fully or keep it simple throughout +- **Don't create text-only slides** — add images, icons, charts, or visual elements; avoid plain title + bullets +- **Don't forget text box padding** — when aligning lines or shapes with text edges, set `margin: 0` on the text box or offset the shape to account for padding +- **Don't use low-contrast elements** — icons AND text need strong contrast against the background +- **NEVER use accent lines under titles** — these are a hallmark of AI-generated slides; use whitespace or background color instead +- **NEVER add decorative color bars or accent stripes** — this includes: header/footer bars spanning the slide width, vertical sidebar stripes down one edge of the slide, thin accent stripes along one edge of a card or content block, and "single-side borders" on rectangles. These read as AI-generated filler. If you want to set a card apart, use a subtle background tint, a drop shadow, or an icon — not an edge stripe. +- **Don't default to cream/beige backgrounds** — when no background is specified, use white (`FFFFFF`) or the user's brand palette; avoid warm-neutral defaults like `F5F5DC`, `FAF0E6`, `FAEBD7`, `FFF8E1` +- **Don't ship text that overflows its shape** — if text doesn't fit, reduce font size, split across slides, or enlarge the container; never leave content cut off or spilling past bounds + +## QA (Required) + +Your first render usually has a few real issues — overlaps, overflow, misalignment. Find and fix those, re-render only the slides you changed, and stop. + +### Content QA + +```bash +markitdown output.pptx +``` + +Check for missing content, typos, wrong order. + +**When using templates, check for leftover placeholder text:** + +```bash +markitdown output.pptx | grep -iE "\bx{3,}\b|lorem|ipsum|\bTODO|\[insert|this.*(page|slide).*layout" +``` + +If grep returns results, fix them before declaring success. + +### File QA (required) + +```bash +python scripts/office/validate.py output.pptx # built from scratch +python scripts/office/validate.py output.pptx --original src.pptx # built from a template +``` + +**If the deck came from a template, always pass `--original`.** A template may itself contain parts the XSD rejects, so a bare run can report failures you never caused — and a genuine regression can hide among them. `--original` baselines the schema and slide checks against the template. The structural checks — relationships, content types, charts — ignore `--original` and report template-inherited problems either way, so read those on their own merits. + +pptxgenjs emits chart XML PowerPoint refuses to open, and every other tool accepts: python-pptx opens those decks, LibreOffice renders them, the XSD passes them. Every failure names its fix. Fix it in the generator and rebuild. + +### Visual QA + +Convert the slides to images (see [Converting to Images](#converting-to-images)) and inspect every one with `vision_analyze`. After staring at the generating code you tend to see what you expect rather than what rendered, so look at the images fresh (a `delegate_task` subagent works well for this). User-visible defects to look for: + +- **Text overflow or text cut off at a box or slide boundary — check this first.** It is the most common defect and always user-visible. (For a font the previewer renders unreliably per Typography, the preview is approximate: trust the ~10% slack you left, not its apparent fit.) +- Overlapping elements (text through shapes, lines through words, stacked elements) +- Source citations or footers colliding with content above +- Elements too close (< 0.3" gaps) or cards/sections nearly touching +- Uneven gaps (large empty area in one place, cramped in another) +- Insufficient margin from slide edges (< 0.5") +- Columns or similar elements not aligned consistently +- Low-contrast text (e.g., light gray text on cream-colored background) +- Template decoration mispositioned after text replacement — e.g., a title underline positioned for one line, but the replaced title wrapped to two +- Low-contrast icons (e.g., dark icons on dark backgrounds without a contrasting circle) +- Text boxes too narrow causing excessive wrapping +- Leftover placeholder content + +## Converting to Images + +Convert presentations to individual slide images for visual inspection: + +```bash +python scripts/office/soffice.py --headless --convert-to pdf output.pptx +rm -f slide-*.jpg +pdftoppm -jpeg -r 150 output.pdf slide +ls -1 "$PWD"/slide-*.jpg +``` + +**Pass the absolute paths printed above directly to `vision_analyze`.** The `rm` clears stale images from prior runs. `pdftoppm` zero-pads based on page count: `slide-1.jpg` for decks under 10 pages, `slide-01.jpg` for 10-99, `slide-001.jpg` for 100+. + +**After fixes, rerun all four commands above** — the PDF must be regenerated from the edited `.pptx` before `pdftoppm` can reflect your changes. + +## Related skills + +`docx` (Word documents), `xlsx` (spreadsheets), `pdf` (PDF work), optional `pptx-author` (finance-grade model-backed decks). diff --git a/powerpoint/scripts/__init__.py b/powerpoint/scripts/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/powerpoint/scripts/add_slide.py b/powerpoint/scripts/add_slide.py new file mode 100644 index 0000000..f013ea9 --- /dev/null +++ b/powerpoint/scripts/add_slide.py @@ -0,0 +1,367 @@ +"""Add a slide to a PPTX: duplicate an existing slide or instantiate a layout. + +Does all of the package bookkeeping, so the deck stays valid: + - writes the new ppt/slides/slideN.xml (and its .rels, minus any + notesSlide reference, so the source's speaker notes aren't shared) + - registers it in [Content_Types].xml + - adds a slide relationship with a fresh rId to presentation.xml.rels + - inserts with a fresh id into + — at the end, or after --after SLIDE + +Works on an unpacked directory (during an editing session) or directly on a +.pptx/.potx file (extracted to a temp dir, then rezipped atomically; the +temp dir is discarded, so unpack the output if you still need to edit the +new slide's content). + +Usage: + python add_slide.py unpacked/ slide2.xml # duplicate slide2 + python add_slide.py unpacked/ slideLayout3.xml # new slide from a layout + python add_slide.py unpacked/ slide2.xml --after slide2.xml + python add_slide.py deck.pptx slide2.xml # rewrite deck.pptx in place + python add_slide.py deck.pptx slide2.xml -o out.pptx + +A duplicated slide still holds the source's content: edit ppt/slides/slideN.xml +(printed on success) to change it. To list layouts: ls /ppt/slideLayouts/ +""" + +import argparse +import re +import shutil +import sys +from typing import NoReturn +import tempfile +import zipfile +from pathlib import Path + +from office.helpers import rezip, safe_extract + +MINIMAL_SLIDE_XML = ''' + + + + + + + + + + + + + + + + + + + + + +''' + +SHARED_PART_TYPES = ("chart", "diagramData", "oleObject", "package") + +NOTES_SLIDE_TYPE_RE = re.compile(r"""Type=["'][^"']*/relationships/notesSlide["']""") +RELATIONSHIP_RE = re.compile(r"]*?(?:/>|>.*?)", re.DOTALL) + +SLIDE_ID_MIN = 256 +SLIDE_ID_MAX = 2147483647 + + +def _die(msg: str) -> NoReturn: + print(f"Error: {msg}", file=sys.stderr) + sys.exit(1) + + +def get_next_slide_number(slides_dir: Path) -> int: + existing = [int(m.group(1)) for f in slides_dir.glob("slide*.xml") + if (m := re.match(r"slide(\d+)\.xml", f.name))] + return max(existing) + 1 if existing else 1 + + +def parse_source(source: str) -> tuple[str, str | None]: + if source.startswith("slideLayout") and source.endswith(".xml"): + return ("layout", source) + + return ("slide", None) + + +def create_slide_from_layout(unpacked_dir: Path, layout_file: str, after: str | None = None) -> str: + slides_dir = unpacked_dir / "ppt" / "slides" + rels_dir = slides_dir / "_rels" + layout_path = unpacked_dir / "ppt" / "slideLayouts" / layout_file + + if not layout_path.exists(): + _die(f"{layout_path} not found") + + next_num = get_next_slide_number(slides_dir) + dest = f"slide{next_num}.xml" + after_rid = _precheck_registration(unpacked_dir, after, dest) + slides_dir.mkdir(parents=True, exist_ok=True) + + (slides_dir / dest).write_text(MINIMAL_SLIDE_XML, encoding="utf-8") + + rels_dir.mkdir(exist_ok=True) + rels_xml = f''' + + +''' + (rels_dir / f"{dest}.rels").write_text(rels_xml, encoding="utf-8") + + _register_slide(unpacked_dir, dest, layout_file, after_rid) + return dest + + +def duplicate_slide(unpacked_dir: Path, source: str, after: str | None = None) -> str: + slides_dir = unpacked_dir / "ppt" / "slides" + rels_dir = slides_dir / "_rels" + source_slide = slides_dir / source + + if not source_slide.exists(): + _die(f"{source_slide} not found") + + next_num = get_next_slide_number(slides_dir) + dest = f"slide{next_num}.xml" + after_rid = _precheck_registration(unpacked_dir, after, dest) + + shutil.copy2(source_slide, slides_dir / dest) + + source_rels = rels_dir / f"{source}.rels" + shared_parts: list[str] = [] + if source_rels.exists(): + dest_rels = rels_dir / f"{dest}.rels" + shutil.copy2(source_rels, dest_rels) + rels_content = dest_rels.read_text(encoding="utf-8") + rels_content = RELATIONSHIP_RE.sub( + lambda m: "" if NOTES_SLIDE_TYPE_RE.search(m.group(0)) else m.group(0), + rels_content, + ) + dest_rels.write_text(rels_content, encoding="utf-8") + shared_parts = sorted({ + t for t in re.findall(r'Type="[^"]*/relationships/(\w+)"', rels_content) + if t in SHARED_PART_TYPES + }) + + _register_slide(unpacked_dir, dest, source, after_rid) + if shared_parts: + print( + f"Note: {dest} shares its {', '.join(shared_parts)} part(s) with {source} " + f"(they are referenced, not copied) — editing those parts changes both slides" + ) + return dest + + +def _precheck_registration(unpacked_dir: Path, after: str | None, dest: str) -> str | None: + pres_path = unpacked_dir / "ppt" / "presentation.xml" + if not pres_path.exists(): + _die(f"{pres_path} not found — is this an unpacked PPTX?") + xml = pres_path.read_text(encoding="utf-8") + + has_slot = ( + "" in xml + or re.search(r"", xml) + or "" in xml + ) + if not has_slot: + _die("presentation.xml has no (or to anchor a new one)") + + stale = [] + content_types = unpacked_dir / "[Content_Types].xml" + if content_types.exists() and f'PartName="/ppt/slides/{dest}"' in content_types.read_text(encoding="utf-8"): + stale.append("[Content_Types].xml") + pres_rels = unpacked_dir / "ppt" / "_rels" / "presentation.xml.rels" + if pres_rels.exists() and _find_slide_relationship( + pres_rels.read_text(encoding="utf-8"), dest + ): + stale.append("presentation.xml.rels") + if stale: + _die( + f"{dest} is still registered in {' and '.join(stale)} but absent from ppt/slides/ — " + f"run clean.py first" + ) + + if not after: + return None + after_rid = _rid_for_slide(unpacked_dir, after) + if not re.search(rf']*r:id="{re.escape(after_rid)}"[^>]*>', xml): + _die(f"{after} ({after_rid}) is not listed in ") + return after_rid + + +def _register_slide(unpacked_dir: Path, dest: str, source_desc: str, after_rid: str | None) -> None: + _add_to_content_types(unpacked_dir, dest) + rid = _add_to_presentation_rels(unpacked_dir, dest) + slide_id = _get_next_slide_id(unpacked_dir) + pos, total = _insert_into_sld_id_lst(unpacked_dir, slide_id, rid, after_rid) + + print(f"Created ppt/slides/{dest} from {source_desc}") + print( + f'Inserted into ' + f"at position {pos} of {total}" + ) + + +def _add_to_content_types(unpacked_dir: Path, dest: str) -> None: + content_types_path = unpacked_dir / "[Content_Types].xml" + content_types = content_types_path.read_text(encoding="utf-8") + + new_override = f'' + + if f'PartName="/ppt/slides/{dest}"' not in content_types: + content_types = content_types.replace("", f" {new_override}\n") + content_types_path.write_text(content_types, encoding="utf-8") + + +def _add_to_presentation_rels(unpacked_dir: Path, dest: str) -> str: + pres_rels_path = unpacked_dir / "ppt" / "_rels" / "presentation.xml.rels" + pres_rels = pres_rels_path.read_text(encoding="utf-8") + + existing = _find_slide_relationship(pres_rels, dest) + if existing: + return existing + + pres_xml = (unpacked_dir / "ppt" / "presentation.xml").read_text(encoding="utf-8") + used = {int(n) for n in re.findall(r'\bId="rId(\d+)"', pres_rels)} + used |= {int(n) for n in re.findall(r'\br:id="rId(\d+)"', pres_xml)} + rid = f"rId{max(used) + 1 if used else 1}" + + new_rel = f'' + pres_rels = pres_rels.replace("", f" {new_rel}\n") + pres_rels_path.write_text(pres_rels, encoding="utf-8") + + return rid + + +def _find_slide_relationship(pres_rels: str, slide_name: str) -> str | None: + for m in re.finditer(r"]*>", pres_rels): + element = m.group(0) + if re.search(rf'Target="(?:/ppt/)?slides/{re.escape(slide_name)}"', element): + id_match = re.search(r'\bId="([^"]+)"', element) + if id_match: + return id_match.group(1) + return None + + +def _get_next_slide_id(unpacked_dir: Path) -> int: + pres_content = (unpacked_dir / "ppt" / "presentation.xml").read_text(encoding="utf-8") + used = {int(m) for m in re.findall(r']*\bid="(\d+)"', pres_content)} + + candidate = max((i for i in used if i >= SLIDE_ID_MIN), default=SLIDE_ID_MIN - 1) + 1 + if candidate <= SLIDE_ID_MAX and candidate not in used: + return candidate + for i in range(SLIDE_ID_MIN, SLIDE_ID_MAX + 1): + if i not in used: + return i + _die("no slide id available in [256, 2147483647] — the deck is full") + + +def _insert_into_sld_id_lst( + unpacked_dir: Path, slide_id: int, rid: str, after_rid: str | None = None +) -> tuple[int, int]: + pres_path = unpacked_dir / "ppt" / "presentation.xml" + xml = pres_path.read_text(encoding="utf-8") + entry = f'' + + if f'r:id="{rid}"' in xml: + _die(f"presentation.xml already references {rid}; refusing to add a duplicate") + + if after_rid: + open_tag = re.search(rf']*r:id="{re.escape(after_rid)}"[^>]*>', xml) + if not open_tag: + _die(f"{after_rid} is not listed in ") + end = open_tag.end() + if not open_tag.group(0).endswith("/>"): + close = xml.find("", end) + if close == -1: + _die(f"unclosed for {after_rid} in presentation.xml") + end = close + len("") + xml = xml[:end] + entry + xml[end:] + elif "" in xml: + xml = xml.replace("", f"{entry}", 1) + elif re.search(r"", xml): + xml = re.sub(r"", f"{entry}", xml, count=1) + elif "" in xml: + xml = xml.replace( + "", f"{entry}", 1 + ) + else: + _die("presentation.xml has no (or to anchor a new one)") + + pres_path.write_text(xml, encoding="utf-8") + + lst = re.search(r"(.*)", xml, re.DOTALL) + entries = re.findall(r"]*>", lst.group(1)) if lst else [] + position = next( + (i for i, e in enumerate(entries, 1) if f'r:id="{rid}"' in e), len(entries) + ) + return position, len(entries) + + +def _rid_for_slide(unpacked_dir: Path, slide_name: str) -> str: + pres_rels_path = unpacked_dir / "ppt" / "_rels" / "presentation.xml.rels" + rid = _find_slide_relationship(pres_rels_path.read_text(encoding="utf-8"), slide_name) + if not rid: + _die(f"{slide_name} has no relationship in presentation.xml.rels") + return rid + + +def add_slide(unpacked_dir: Path, source: str, after: str | None = None) -> str: + source_type, layout_file = parse_source(source) + if source_type == "layout" and layout_file is not None: + return create_slide_from_layout(unpacked_dir, layout_file, after) + return duplicate_slide(unpacked_dir, source, after) + + +def add_slide_to_package( + package: Path, source: str, after: str | None = None, output: Path | None = None +) -> str: + out = output or package + with tempfile.TemporaryDirectory() as tmp: + tmp_path = Path(tmp) + with zipfile.ZipFile(package) as zf: + safe_extract(zf, tmp_path) + dest = add_slide(tmp_path, source, after) + rezip(tmp_path, out) + print(f"Wrote {out} — the new slide is ppt/slides/{dest} inside it (unpack to edit its content)") + return dest + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Add a slide to a PPTX: duplicate a slide or instantiate a layout. " + "Registers content types, relationships, and ." + ) + parser.add_argument("target", help="Unpacked PPTX directory OR a .pptx/.potx file") + parser.add_argument( + "source", + help="slideN.xml to duplicate, or slideLayoutN.xml to create from a layout " + "(list layouts with: ls /ppt/slideLayouts/)", + ) + parser.add_argument( + "--after", + metavar="SLIDE", + help="insert after this slide, e.g. slide2.xml (default: append at the end)", + ) + parser.add_argument( + "-o", + "--output", + help="output file (only with a .pptx/.potx target; default: rewrite the input in place)", + ) + args = parser.parse_args() + + target = Path(args.target) + if target.is_dir(): + if args.output: + parser.error("--output is only valid for .pptx/.potx input; a directory is modified in place") + add_slide(target, args.source, args.after) + elif target.is_file() and target.suffix.lower() in (".pptx", ".potx"): + try: + add_slide_to_package(target, args.source, args.after, Path(args.output) if args.output else None) + except (OSError, ValueError, zipfile.BadZipFile) as e: + _die(str(e)) + else: + _die(f"{target} is neither a directory nor a .pptx/.potx file") + + +if __name__ == "__main__": + main() diff --git a/powerpoint/scripts/clean.py b/powerpoint/scripts/clean.py new file mode 100644 index 0000000..551dd23 --- /dev/null +++ b/powerpoint/scripts/clean.py @@ -0,0 +1,309 @@ +"""Remove unreferenced files from an unpacked PPTX directory. + +Usage: python clean.py + +Example: + python clean.py unpacked/ + +This script removes: +- Orphaned slides (not in sldIdLst) and their relationships +- [trash] directory (unreferenced files) +- Orphaned .rels files for deleted resources +- Unreferenced media, embeddings, charts, diagrams, drawings, ink files +- Unreferenced theme files +- Unreferenced notes slides +- Content-Type overrides for deleted files +""" + +import posixpath +import re +import sys +from pathlib import Path + +import defusedxml.minidom + +from office.helpers import SLIDE_REL_TYPE, opc_target, rels_source_part + + +def _slide_rids(pres_rels_path: Path, unpacked_dir: Path) -> dict[str, str]: + source_part = rels_source_part(pres_rels_path, unpacked_dir) + rels_dom = defusedxml.minidom.parse(str(pres_rels_path)) + + rids: dict[str, str] = {} + for rel in rels_dom.getElementsByTagName("Relationship"): + if rel.getAttribute("Type") != SLIDE_REL_TYPE: + continue + part = opc_target( + rel.getAttribute("Target"), source_part, rel.getAttribute("TargetMode") + ) + if part is not None: + rids[rel.getAttribute("Id")] = part + return rids + + +def get_slides_in_sldidlst(unpacked_dir: Path) -> set[str]: + pres_path = unpacked_dir / "ppt" / "presentation.xml" + pres_rels_path = unpacked_dir / "ppt" / "_rels" / "presentation.xml.rels" + + if not pres_path.exists() or not pres_rels_path.exists(): + return set() + + rid_to_slide = _slide_rids(pres_rels_path, unpacked_dir) + + pres_content = pres_path.read_text(encoding="utf-8") + referenced_rids = set(re.findall(r']*r:id="([^"]+)"', pres_content)) + + return { + posixpath.basename(rid_to_slide[rid]) + for rid in referenced_rids + if rid in rid_to_slide + } + + +class RefusedToClean(Exception): + """The package does not look the way a readable package should.""" + + +def remove_orphaned_slides(unpacked_dir: Path) -> list[str]: + slides_dir = unpacked_dir / "ppt" / "slides" + slides_rels_dir = slides_dir / "_rels" + pres_rels_path = unpacked_dir / "ppt" / "_rels" / "presentation.xml.rels" + + if not slides_dir.exists(): + return [] + + referenced_slides = get_slides_in_sldidlst(unpacked_dir) + on_disk = sorted(slides_dir.glob("slide*.xml")) + + if on_disk and not any(s.name in referenced_slides for s in on_disk): + listed = re.findall( + r']*r:id="([^"]+)"', + (unpacked_dir / "ppt" / "presentation.xml").read_text(encoding="utf-8") + if (unpacked_dir / "ppt" / "presentation.xml").exists() + else "", + ) + if listed: + raise RefusedToClean( + f" lists {len(listed)} slide(s) and none of the " + f"{len(on_disk)} slide(s) on disk match any of them. Refusing to " + f"delete them all — this is a parse failure, not an empty deck." + ) + + removed = [] + + for slide_file in on_disk: + if slide_file.name not in referenced_slides: + rel_path = slide_file.relative_to(unpacked_dir) + slide_file.unlink() + removed.append(str(rel_path)) + + rels_file = slides_rels_dir / f"{slide_file.name}.rels" + if rels_file.exists(): + rels_file.unlink() + removed.append(str(rels_file.relative_to(unpacked_dir))) + + if removed and pres_rels_path.exists(): + rels_dom = defusedxml.minidom.parse(str(pres_rels_path)) + source_part = rels_source_part(pres_rels_path, unpacked_dir) + changed = False + + for rel in list(rels_dom.getElementsByTagName("Relationship")): + if rel.getAttribute("Type") != SLIDE_REL_TYPE: + continue + part = opc_target( + rel.getAttribute("Target"), source_part, rel.getAttribute("TargetMode") + ) + if part is None: + continue + if posixpath.basename(part) not in referenced_slides: + if rel.parentNode: + rel.parentNode.removeChild(rel) + changed = True + + if changed: + with open(pres_rels_path, "wb") as f: + f.write(rels_dom.toxml(encoding="utf-8")) + + return removed + + +def remove_trash_directory(unpacked_dir: Path) -> list[str]: + trash_dir = unpacked_dir / "[trash]" + removed = [] + + if trash_dir.exists() and trash_dir.is_dir(): + for file_path in trash_dir.iterdir(): + if file_path.is_file(): + rel_path = file_path.relative_to(unpacked_dir) + removed.append(str(rel_path)) + file_path.unlink() + trash_dir.rmdir() + + return removed + + +def _referenced_by(rels_files, unpacked_dir: Path) -> set: + referenced = set() + + for rels_file in rels_files: + source_part = rels_source_part(rels_file, unpacked_dir) + dom = defusedxml.minidom.parse(str(rels_file)) + for rel in dom.getElementsByTagName("Relationship"): + part = opc_target( + rel.getAttribute("Target"), source_part, rel.getAttribute("TargetMode") + ) + if part is not None: + referenced.add(Path(part)) + + return referenced + + +def remove_orphaned_rels_files(unpacked_dir: Path) -> list[str]: + resource_dirs = ["charts", "diagrams", "drawings"] + removed = [] + + for dir_name in resource_dirs: + rels_dir = unpacked_dir / "ppt" / dir_name / "_rels" + if not rels_dir.exists(): + continue + + for rels_file in rels_dir.glob("*.rels"): + resource_file = rels_dir.parent / rels_file.name.replace(".rels", "") + if not resource_file.exists(): + rels_file.unlink() + removed.append(str(rels_file.relative_to(unpacked_dir))) + + return removed + + +def get_referenced_files(unpacked_dir: Path) -> set: + return _referenced_by(sorted(unpacked_dir.rglob("*.rels")), unpacked_dir) + + +def remove_orphaned_files(unpacked_dir: Path, referenced: set) -> list[str]: + resource_dirs = ["media", "embeddings", "charts", "diagrams", "tags", "drawings", "ink"] + removed = [] + + for dir_name in resource_dirs: + dir_path = unpacked_dir / "ppt" / dir_name + if not dir_path.exists(): + continue + + for file_path in dir_path.glob("*"): + if not file_path.is_file(): + continue + rel_path = file_path.relative_to(unpacked_dir) + if rel_path not in referenced: + file_path.unlink() + removed.append(str(rel_path)) + + theme_dir = unpacked_dir / "ppt" / "theme" + if theme_dir.exists(): + for file_path in theme_dir.glob("theme*.xml"): + rel_path = file_path.relative_to(unpacked_dir) + if rel_path not in referenced: + file_path.unlink() + removed.append(str(rel_path)) + theme_rels = theme_dir / "_rels" / f"{file_path.name}.rels" + if theme_rels.exists(): + theme_rels.unlink() + removed.append(str(theme_rels.relative_to(unpacked_dir))) + + notes_dir = unpacked_dir / "ppt" / "notesSlides" + if notes_dir.exists(): + for file_path in notes_dir.glob("*.xml"): + if not file_path.is_file(): + continue + rel_path = file_path.relative_to(unpacked_dir) + if rel_path not in referenced: + file_path.unlink() + removed.append(str(rel_path)) + + notes_rels_dir = notes_dir / "_rels" + if notes_rels_dir.exists(): + for file_path in notes_rels_dir.glob("*.rels"): + notes_file = notes_dir / file_path.name.replace(".rels", "") + if not notes_file.exists(): + file_path.unlink() + removed.append(str(file_path.relative_to(unpacked_dir))) + + return removed + + +def update_content_types(unpacked_dir: Path, removed_files: list[str]) -> None: + ct_path = unpacked_dir / "[Content_Types].xml" + if not ct_path.exists(): + return + + dom = defusedxml.minidom.parse(str(ct_path)) + changed = False + + for override in list(dom.getElementsByTagName("Override")): + part_name = override.getAttribute("PartName").lstrip("/") + if part_name in removed_files: + if override.parentNode: + override.parentNode.removeChild(override) + changed = True + + if changed: + with open(ct_path, "wb") as f: + f.write(dom.toxml(encoding="utf-8")) + + +def clean_unused_files(unpacked_dir: Path) -> list[str]: + all_removed = [] + + if list(unpacked_dir.rglob("*.rels")) and not get_referenced_files(unpacked_dir): + raise RefusedToClean( + "no relationship in this package names a part we can resolve. " + "Refusing to treat every file as unreferenced." + ) + + slides_removed = remove_orphaned_slides(unpacked_dir) + all_removed.extend(slides_removed) + + trash_removed = remove_trash_directory(unpacked_dir) + all_removed.extend(trash_removed) + + while True: + removed_rels = remove_orphaned_rels_files(unpacked_dir) + referenced = get_referenced_files(unpacked_dir) + removed_files = remove_orphaned_files(unpacked_dir, referenced) + + total_removed = removed_rels + removed_files + if not total_removed: + break + + all_removed.extend(total_removed) + + if all_removed: + update_content_types(unpacked_dir, all_removed) + + return all_removed + + +if __name__ == "__main__": + if len(sys.argv) != 2: + print("Usage: python clean.py ", file=sys.stderr) + print("Example: python clean.py unpacked/", file=sys.stderr) + sys.exit(1) + + unpacked_dir = Path(sys.argv[1]) + + if not unpacked_dir.exists(): + print(f"Error: {unpacked_dir} not found", file=sys.stderr) + sys.exit(1) + + try: + removed = clean_unused_files(unpacked_dir) + except (RefusedToClean, ValueError) as e: + print(f"Error: {e}", file=sys.stderr) + print("Nothing was deleted.", file=sys.stderr) + sys.exit(1) + + if removed: + print(f"Removed {len(removed)} unreferenced files:") + for f in removed: + print(f" {f}") + else: + print("No unreferenced files found") diff --git a/powerpoint/scripts/office/helpers/__init__.py b/powerpoint/scripts/office/helpers/__init__.py new file mode 100644 index 0000000..d3c5817 --- /dev/null +++ b/powerpoint/scripts/office/helpers/__init__.py @@ -0,0 +1,111 @@ +import os +import posixpath +import re +import stat +import tempfile +import urllib.parse +import zipfile +from pathlib import Path + +OOXML_FAMILY = { + ".docx": "docx", + ".dotx": "docx", + ".pptx": "pptx", + ".potx": "pptx", + ".xlsx": "xlsx", + ".xltx": "xlsx", +} + +_SCHEME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9+.\-]*:") + +SLIDE_REL_TYPE = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide" + + +def opc_target(target: str, source_part: str, target_mode: str = "") -> str | None: + if not target: + return None + if target_mode.lower() == "external": + return None + if _SCHEME_RE.match(target): + return None + + target = urllib.parse.unquote(target) + + if "\\" in target: + raise ValueError(f"relationship target is not a POSIX part name: {target!r}") + + if target.startswith("/"): + joined = target.lstrip("/") + else: + joined = posixpath.join(posixpath.dirname(source_part), target) + + parts: list[str] = [] + for segment in posixpath.normpath(joined).split("/"): + if segment in ("", "."): + continue + if segment == "..": + if not parts: + raise ValueError(f"relationship target escapes the package: {target!r}") + parts.pop() + else: + parts.append(segment) + + if not parts: + raise ValueError(f"relationship target resolves to nothing: {target!r}") + return "/".join(parts) + + +def rels_source_part(rels_file: Path, unpacked_dir: Path) -> str: + owner_dir = rels_file.parent.parent.relative_to(unpacked_dir) + return posixpath.join(owner_dir.as_posix(), rels_file.name[: -len(".rels")]).lstrip("./") + + +def part_text(data: bytes) -> str: + return data.decode("utf-8", "surrogateescape") + + +XML_SPACE = " \t\r\n" + + +def rendered_text(text: str, preserve: bool) -> str: + return text if preserve else text.strip(XML_SPACE) + + +def safe_extract(zf: zipfile.ZipFile, dest: Path) -> None: + dest = dest.resolve() + for m in zf.infolist(): + if stat.S_ISLNK(m.external_attr >> 16): + raise ValueError(f"symlink archive entry not allowed: {m.filename!r}") + target = (dest / m.filename).resolve() + if not target.is_relative_to(dest): + raise ValueError(f"unsafe archive entry: {m.filename!r}") + zf.extract(m, dest) + + +def rezip(src_dir: Path, out_path: Path) -> None: + files = sorted(p for p in src_dir.rglob("*") if p.is_file()) + ct = src_dir / "[Content_Types].xml" + fd, tmp_name = tempfile.mkstemp( + prefix=out_path.name + ".", suffix=".tmp", dir=out_path.parent + ) + tmp_out = Path(tmp_name) + try: + with os.fdopen(fd, "wb") as fh: + with zipfile.ZipFile(fh, "w", zipfile.ZIP_DEFLATED) as zf: + if ct.exists(): + zf.write(ct, ct.relative_to(src_dir), compress_type=zipfile.ZIP_STORED) + for f in files: + if f == ct: + continue + zf.write(f, f.relative_to(src_dir)) + if out_path.exists(): + mode = out_path.stat().st_mode & 0o777 + else: + umask = os.umask(0) + os.umask(umask) + mode = 0o666 & ~umask + os.chmod(tmp_out, mode) + os.replace(tmp_out, out_path) + finally: + if tmp_out.exists(): + tmp_out.unlink() diff --git a/powerpoint/scripts/office/helpers/pptx_chart.py b/powerpoint/scripts/office/helpers/pptx_chart.py new file mode 100644 index 0000000..209cb7c --- /dev/null +++ b/powerpoint/scripts/office/helpers/pptx_chart.py @@ -0,0 +1,170 @@ +"""Find chart XML that PowerPoint refuses but the schema accepts. + +Detection only: for either fault more than one repair is valid, and only the +author knows which was meant. +""" + + +from __future__ import annotations + +import re +from typing import Mapping + +from . import part_text + + +_CHART_PART_RE = re.compile(r"ppt/charts/chart\d+\.xml") + +_GROUPING_RE = re.compile(r"""]*?\bval=["'](\w+)["']""") +_DLBL_POS_RE = re.compile(r"""]*?\bval=["'](\w+)["']""") + +def _strip_ext_lst(text: str) -> str: + out, cursor = [], 0 + for lo, hi in _ext_lst_spans(text): + out.append(text[cursor:lo]) + cursor = hi + out.append(text[cursor:]) + return "".join(out) + +_BAR_GROUP_RE = re.compile(r"]*(?.*?", re.DOTALL) + +STACKED_GROUPINGS = frozenset({"stacked", "percentStacked"}) +ILLEGAL_ON_STACKED = frozenset({"outEnd"}) +LEGAL_ON_STACKED = ("ctr", "inEnd", "inBase") + + +def _check_stacked_label_positions(part: str, xml: str) -> list[str]: + problems: list[str] = [] + for match in _BAR_GROUP_RE.finditer(xml): + block = _strip_ext_lst(match.group(0)) + group = match.group(1) + + grouping = _GROUPING_RE.search(block) + if grouping is None or grouping.group(1) not in STACKED_GROUPINGS: + continue + + bad = [p for p in _DLBL_POS_RE.findall(block) if p in ILLEGAL_ON_STACKED] + for pos in sorted(set(bad)): + problems.append( + f'{part}: {bad.count(pos)} data label(s) use dLblPos="{pos}" on a ' + f"{grouping.group(1)} {group}; PowerPoint allows only " + f"{', '.join(LEGAL_ON_STACKED)} there" + ) + return problems + + + +_ANY_CHART_GROUP_RE = re.compile(r"]*(?.*?", re.DOTALL) + +_AXID_RE = re.compile( + r"""\s*]*?\bval=["'](-?\d+)["']\s*(?:/>|>\s*)""" +) + +_AXIS_DECL_RE = re.compile( + r"""]*(?\s*]*?\bval=["'](-?\d+)["']""" +) + +AXID_LIMIT = { + "barChart": 2, "lineChart": 2, "areaChart": 2, "scatterChart": 2, + "bubbleChart": 2, "radarChart": 2, "stockChart": 2, + "bar3DChart": 3, "line3DChart": 3, "area3DChart": 3, + "surfaceChart": 3, "surface3DChart": 3, +} + +AXID_MINIMUM = { + "barChart": 2, "lineChart": 2, "areaChart": 2, "scatterChart": 2, + "bubbleChart": 2, "radarChart": 2, "stockChart": 2, + "bar3DChart": 2, "area3DChart": 2, "surfaceChart": 2, + "line3DChart": 3, "surface3DChart": 3, +} + + +def _declared_axes(xml: str) -> dict[str, list[str]]: + axes: dict[str, list[str]] = {} + for kind, axid in _AXIS_DECL_RE.findall(xml): + axes.setdefault(kind, []).append(axid) + return axes + + +def _canonical_ids(axes: dict[str, list[str]], limit: int) -> list[str] | None: + category = axes.get("catAx", []) + axes.get("dateAx", []) + value = axes.get("valAx", []) + series = axes.get("serAx", []) + if len(category) != 1 or len(value) != 1 or len(series) > 1: + return None + ids = [category[0], value[0]] + if limit >= 3 and series: + ids.append(series[0]) + return ids + + +def _undeclared_axes(kind: str, block: str, axes: dict[str, list[str]]) -> list[str] | None: + if kind not in AXID_LIMIT: + return None + ids = _AXID_RE.findall(block) + declared = {i for group in axes.values() for i in group} + if len([i for i in ids if i in declared]) >= 2: + return None + return ids + + +def _check_chart_axis_references(part: str, xml: str) -> list[str]: + axes = _declared_axes(xml) + problems: list[str] = [] + declared = {i for group in axes.values() for i in group} + for match in _ANY_CHART_GROUP_RE.finditer(xml): + kind, block = match.group(1), match.group(0) + ids = _undeclared_axes(kind, block, axes) + if ids is None: + continue + if not ids: + problems.append( + f"{part}: declares no this part can resolve; a chart " + f"group needs {AXID_MINIMUM[kind]}, and PowerPoint discards one with fewer" + ) + continue + dead = [i for i in ids if i not in declared] + canonical = _canonical_ids(axes, AXID_LIMIT[kind]) + if canonical is not None and len(canonical) >= AXID_MINIMUM[kind]: + hint = f"Fix: point them at the axes this part declares ({', '.join(canonical)})" + else: + hint = ("Fix: the part declares several axes of a kind -- declare the " + "secondary axes the series expects, or drop them") + detail = (f"of which {', '.join(dead)} name no declared axis" + if dead else f"only {len(ids)} of which this part declares") + problems.append( + f"{part}: references axId {', '.join(ids)}, {detail}, " + f"leaving fewer than two live axes; PowerPoint discards the chart. {hint}" + ) + return problems + + +def _ext_lst_spans(text: str) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + depth = 0 + start = 0 + for match in re.finditer(r"<(/?)c:extLst\b[^>]*?(/?)>", text): + closing, self_closing = match.group(1), match.group(2) + if self_closing: + continue + if closing: + depth -= 1 + if depth == 0: + spans.append((start, match.end())) + else: + if depth == 0: + start = match.start() + depth += 1 + return spans + + +CHART_CHECKS = (_check_stacked_label_positions, _check_chart_axis_references) + + +def find_chart_problems(files: Mapping[str, bytes]) -> list[str]: + problems: list[str] = [] + for part in sorted(n for n in files if _CHART_PART_RE.fullmatch(n)): + xml = part_text(files[part]) + for check in CHART_CHECKS: + problems.extend(check(part, xml)) + return problems diff --git a/powerpoint/scripts/office/helpers/pptx_slide.py b/powerpoint/scripts/office/helpers/pptx_slide.py new file mode 100644 index 0000000..22f9aee --- /dev/null +++ b/powerpoint/scripts/office/helpers/pptx_slide.py @@ -0,0 +1,60 @@ +"""Pick the slide-XML schema errors PowerPoint refuses the file over. + +A denylist over lxml's messages, so an unrecognised error class is a miss rather +than a false alarm. +""" + + +from __future__ import annotations + +import re + +SLIDE_PART_RE = re.compile( + r"ppt/(slides|slideLayouts|slideMasters|notesSlides|notesMasters|handoutMasters)" + r"/[^/]+\.xml" +) + +FATAL_SLIDE_ERRORS: tuple[tuple[re.Pattern[str], str], ...] = ( + ( + re.compile(r"\}tableStyleId': This element is not expected"), + "two in one (the schema allows one)", + ), + ( + re.compile(r"\}srgbClr', attribute 'val'"), + "a colour that is not six hex digits", + ), + ( + re.compile(r"\}txBody': Missing child element"), + "a with no children", + ), + ( + re.compile(r"\}miter', attribute 'lim'"), + 'a line join with lim="NaN"', + ), + ( + re.compile(r"\}uLnTx': This element is not expected"), + " in a position the schema forbids", + ), + ( + re.compile(r"\}overrideClrMapping': This element is not expected"), + " in a position the schema forbids", + ), + ( + re.compile(r"\}nvGrpSpPr': Missing child element"), + "a with no children", + ), +) + + +def is_schema_verdict(error: str) -> bool: + return error.startswith("Element ") + + +def fatal_slide_errors(errors: set[str]) -> list[str]: + out = [] + for error in sorted(errors): + for pattern, meaning in FATAL_SLIDE_ERRORS: + if pattern.search(error): + out.append(f"{meaning}: {error}") + break + return out diff --git a/powerpoint/scripts/office/helpers/pptx_theme.py b/powerpoint/scripts/office/helpers/pptx_theme.py new file mode 100644 index 0000000..8446620 --- /dev/null +++ b/powerpoint/scripts/office/helpers/pptx_theme.py @@ -0,0 +1,114 @@ +"""Find masters sharing a theme part in the way PowerPoint refuses to open. + +Reports only; the fix is to move back to directly after + in ppt/presentation.xml. +""" + + +from __future__ import annotations + +import posixpath +import re +from typing import Mapping + +from . import part_text + +THEME_REL_TYPE = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/theme" + +_MASTER_RE = re.compile( + r"^ppt/(?PslideMasters|notesMasters|handoutMasters)/" + r"(?:slide|notes|handout)Master(?P\d+)\.xml$" +) +_GROUP_ORDER = {"slideMasters": 0, "notesMasters": 1, "handoutMasters": 2} + +_RELATIONSHIP_RE = re.compile( + r"]*?(?:/>|>.*?)", re.DOTALL +) + + +def _sort_key(name: str) -> tuple[int, int]: + m = _MASTER_RE.match(name) + assert m is not None + return (_GROUP_ORDER[m.group("group")], int(m.group("num"))) + + +def _rels_path(part: str) -> str: + directory, base = posixpath.split(part) + return f"{directory}/_rels/{base}.rels" + + +def _resolve(rels_path: str, target: str) -> str: + if target.startswith("/"): + return target.lstrip("/") + part_dir = posixpath.dirname(posixpath.dirname(rels_path)) + return posixpath.normpath(posixpath.join(part_dir, target)) + + +def _theme_rel(files: Mapping[str, bytes], master: str): + rels_path = _rels_path(master) + rels = files.get(rels_path) + if rels is None: + return None + for element in _RELATIONSHIP_RE.findall(part_text(rels)): + if f'Type="{THEME_REL_TYPE}"' not in element: + continue + target = re.search(r'\bTarget="([^"]+)"', element) + if target is None: + continue + return rels_path, element, _resolve(rels_path, target.group(1)) + return None + + +def _masters(files: Mapping[str, bytes]) -> list[str]: + return sorted((n for n in files if _MASTER_RE.match(n)), key=_sort_key) + + +_PRESENTATION = "ppt/presentation.xml" +_NOTES_MASTERS = "ppt/notesMasters/" +_IGNORABLE_RE = re.compile(r"|<\?.*?\?>", re.DOTALL) +_AFTER_SLDIDLST_RE = re.compile( + r"]*/>|[^>]*>.*?)\s*(<[^>\s/]+)", re.DOTALL +) + + +def _notes_master_share_is_inert(files: Mapping[str, bytes]) -> bool: + data = files.get(_PRESENTATION) + if data is None: + return False + match = _AFTER_SLDIDLST_RE.search(_IGNORABLE_RE.sub("", part_text(data))) + return match is not None and match.group(1) == " bool: + return inert_notes and master.startswith(_NOTES_MASTERS) + + +def find_shared_master_themes(files: Mapping[str, bytes]) -> list[str]: + return [ + f"{master} shares {theme} with {first}" + for master, _, _, theme, first in _shares(files) + ] + + +def live_shared_master_themes(files: Mapping[str, bytes]) -> list[str]: + inert_notes = _notes_master_share_is_inert(files) + return [ + f"{master} shares {theme} with {first}" + for master, _, _, theme, first in _shares(files) + if not _is_inert(master, inert_notes) + ] diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd new file mode 100644 index 0000000..6454ef9 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chart.xsd @@ -0,0 +1,1499 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd new file mode 100644 index 0000000..afa4f46 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-chartDrawing.xsd @@ -0,0 +1,146 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd new file mode 100644 index 0000000..64e66b8 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-diagram.xsd @@ -0,0 +1,1085 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd new file mode 100644 index 0000000..687eea8 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-lockedCanvas.xsd @@ -0,0 +1,11 @@ + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd new file mode 100644 index 0000000..6ac81b0 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-main.xsd @@ -0,0 +1,3081 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd new file mode 100644 index 0000000..1dbf051 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-picture.xsd @@ -0,0 +1,23 @@ + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd new file mode 100644 index 0000000..f1af17d --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-spreadsheetDrawing.xsd @@ -0,0 +1,185 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd new file mode 100644 index 0000000..0a185ab --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/dml-wordprocessingDrawing.xsd @@ -0,0 +1,287 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd new file mode 100644 index 0000000..14ef488 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/pml.xsd @@ -0,0 +1,1676 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd new file mode 100644 index 0000000..c20f3bf --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-additionalCharacteristics.xsd @@ -0,0 +1,28 @@ + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd new file mode 100644 index 0000000..ac60252 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-bibliography.xsd @@ -0,0 +1,144 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd new file mode 100644 index 0000000..424b8ba --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-commonSimpleTypes.xsd @@ -0,0 +1,174 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd new file mode 100644 index 0000000..2bddce2 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlDataProperties.xsd @@ -0,0 +1,25 @@ + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd new file mode 100644 index 0000000..8a8c18b --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-customXmlSchemaProperties.xsd @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd new file mode 100644 index 0000000..5c42706 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd @@ -0,0 +1,59 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd new file mode 100644 index 0000000..853c341 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd @@ -0,0 +1,56 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd new file mode 100644 index 0000000..da835ee --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-documentPropertiesVariantTypes.xsd @@ -0,0 +1,195 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd new file mode 100644 index 0000000..87ad265 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-math.xsd @@ -0,0 +1,582 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd new file mode 100644 index 0000000..9e86f1b --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/shared-relationshipReference.xsd @@ -0,0 +1,25 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd new file mode 100644 index 0000000..d0be42e --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/sml.xsd @@ -0,0 +1,4439 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd new file mode 100644 index 0000000..8821dd1 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-main.xsd @@ -0,0 +1,570 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd new file mode 100644 index 0000000..ca2575c --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-officeDrawing.xsd @@ -0,0 +1,509 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd new file mode 100644 index 0000000..dd079e6 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-presentationDrawing.xsd @@ -0,0 +1,12 @@ + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd new file mode 100644 index 0000000..3dd6cf6 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-spreadsheetDrawing.xsd @@ -0,0 +1,108 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd new file mode 100644 index 0000000..f1041e3 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/vml-wordprocessingDrawing.xsd @@ -0,0 +1,96 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd new file mode 100644 index 0000000..9c5b7a6 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/wml.xsd @@ -0,0 +1,3646 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd new file mode 100644 index 0000000..0f13678 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ISO-IEC29500-4_2016/xml.xsd @@ -0,0 +1,116 @@ + + + + + + See http://www.w3.org/XML/1998/namespace.html and + http://www.w3.org/TR/REC-xml for information about this namespace. + + This schema document describes the XML namespace, in a form + suitable for import by other schema documents. + + Note that local names in this namespace are intended to be defined + only by the World Wide Web Consortium or its subgroups. The + following names are currently defined in this namespace and should + not be used with conflicting semantics by any Working Group, + specification, or document instance: + + base (as an attribute name): denotes an attribute whose value + provides a URI to be used as the base for interpreting any + relative URIs in the scope of the element on which it + appears; its value is inherited. This name is reserved + by virtue of its definition in the XML Base specification. + + lang (as an attribute name): denotes an attribute whose value + is a language code for the natural language of the content of + any element; its value is inherited. This name is reserved + by virtue of its definition in the XML specification. + + space (as an attribute name): denotes an attribute whose + value is a keyword indicating what whitespace processing + discipline is intended for the content of the element; its + value is inherited. This name is reserved by virtue of its + definition in the XML specification. + + Father (in any context at all): denotes Jon Bosak, the chair of + the original XML Working Group. This name is reserved by + the following decision of the W3C XML Plenary and + XML Coordination groups: + + In appreciation for his vision, leadership and dedication + the W3C XML Plenary on this 10th day of February, 2000 + reserves for Jon Bosak in perpetuity the XML name + xml:Father + + + + + This schema defines attributes and an attribute group + suitable for use by + schemas wishing to allow xml:base, xml:lang or xml:space attributes + on elements they define. + + To enable this, such a schema must import this schema + for the XML namespace, e.g. as follows: + <schema . . .> + . . . + <import namespace="http://www.w3.org/XML/1998/namespace" + schemaLocation="http://www.w3.org/2001/03/xml.xsd"/> + + Subsequently, qualified reference to any of the attributes + or the group defined below will have the desired effect, e.g. + + <type . . .> + . . . + <attributeGroup ref="xml:specialAttrs"/> + + will define a type which will schema-validate an instance + element with any of those attributes + + + + In keeping with the XML Schema WG's standard versioning + policy, this schema document will persist at + http://www.w3.org/2001/03/xml.xsd. + At the date of issue it can also be found at + http://www.w3.org/2001/xml.xsd. + The schema document at that URI may however change in the future, + in order to remain compatible with the latest version of XML Schema + itself. In other words, if the XML Schema namespace changes, the version + of this document at + http://www.w3.org/2001/xml.xsd will change + accordingly; the version at + http://www.w3.org/2001/03/xml.xsd will not change. + + + + + + In due course, we should install the relevant ISO 2- and 3-letter + codes as the enumerated possible values . . . + + + + + + + + + + + + + + + See http://www.w3.org/TR/xmlbase/ for + information about this attribute. + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-contentTypes.xsd b/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-contentTypes.xsd new file mode 100644 index 0000000..a6de9d2 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-contentTypes.xsd @@ -0,0 +1,42 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-coreProperties.xsd b/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-coreProperties.xsd new file mode 100644 index 0000000..10e978b --- /dev/null +++ b/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-coreProperties.xsd @@ -0,0 +1,50 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-digSig.xsd b/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-digSig.xsd new file mode 100644 index 0000000..4248bf7 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-digSig.xsd @@ -0,0 +1,49 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-relationships.xsd b/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-relationships.xsd new file mode 100644 index 0000000..5649746 --- /dev/null +++ b/powerpoint/scripts/office/schemas/ecma/fourth-edition/opc-relationships.xsd @@ -0,0 +1,33 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/mce/mc.xsd b/powerpoint/scripts/office/schemas/mce/mc.xsd new file mode 100644 index 0000000..ef72545 --- /dev/null +++ b/powerpoint/scripts/office/schemas/mce/mc.xsd @@ -0,0 +1,75 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/microsoft/wml-2010.xsd b/powerpoint/scripts/office/schemas/microsoft/wml-2010.xsd new file mode 100644 index 0000000..f65f777 --- /dev/null +++ b/powerpoint/scripts/office/schemas/microsoft/wml-2010.xsd @@ -0,0 +1,560 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/microsoft/wml-2012.xsd b/powerpoint/scripts/office/schemas/microsoft/wml-2012.xsd new file mode 100644 index 0000000..6b00755 --- /dev/null +++ b/powerpoint/scripts/office/schemas/microsoft/wml-2012.xsd @@ -0,0 +1,67 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/microsoft/wml-2018.xsd b/powerpoint/scripts/office/schemas/microsoft/wml-2018.xsd new file mode 100644 index 0000000..f321d33 --- /dev/null +++ b/powerpoint/scripts/office/schemas/microsoft/wml-2018.xsd @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/microsoft/wml-cex-2018.xsd b/powerpoint/scripts/office/schemas/microsoft/wml-cex-2018.xsd new file mode 100644 index 0000000..364c6a9 --- /dev/null +++ b/powerpoint/scripts/office/schemas/microsoft/wml-cex-2018.xsd @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/microsoft/wml-cid-2016.xsd b/powerpoint/scripts/office/schemas/microsoft/wml-cid-2016.xsd new file mode 100644 index 0000000..fed9d15 --- /dev/null +++ b/powerpoint/scripts/office/schemas/microsoft/wml-cid-2016.xsd @@ -0,0 +1,13 @@ + + + + + + + + + + + + + diff --git a/powerpoint/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd b/powerpoint/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd new file mode 100644 index 0000000..680cf15 --- /dev/null +++ b/powerpoint/scripts/office/schemas/microsoft/wml-sdtdatahash-2020.xsd @@ -0,0 +1,4 @@ + + + + diff --git a/powerpoint/scripts/office/schemas/microsoft/wml-symex-2015.xsd b/powerpoint/scripts/office/schemas/microsoft/wml-symex-2015.xsd new file mode 100644 index 0000000..89ada90 --- /dev/null +++ b/powerpoint/scripts/office/schemas/microsoft/wml-symex-2015.xsd @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/powerpoint/scripts/office/soffice.py b/powerpoint/scripts/office/soffice.py new file mode 100644 index 0000000..eec1687 --- /dev/null +++ b/powerpoint/scripts/office/soffice.py @@ -0,0 +1,192 @@ +""" +Helper for running LibreOffice (soffice) in environments where AF_UNIX +sockets may be blocked (e.g., sandboxed VMs). Detects the restriction +at runtime and applies an LD_PRELOAD shim if needed. + +Usage: + from office.soffice import run_soffice + + result = run_soffice(["--headless", "--convert-to", "pdf", "input.docx"]) + +Call soffice through run_soffice, not through subprocess with get_soffice_env(): +the env dict carries the shim but names no user profile, and a non-root sandbox +cannot bootstrap the default one -- soffice aborts with "User installation could +not be completed" and converts nothing. get_soffice_env() stays public for the +callers that build their own argv (they must pass -env:UserInstallation too). +""" + +import contextlib +import os +import socket +import subprocess +import tempfile +from collections.abc import Iterable +from pathlib import Path + + +def get_soffice_env() -> dict: + env = os.environ.copy() + env["SAL_USE_VCLPLUGIN"] = "svp" + + if _needs_shim(): + shim = _ensure_shim() + env["LD_PRELOAD"] = str(shim) + + return env + + +def run_soffice(args: Iterable[str], **kwargs) -> subprocess.CompletedProcess: + args = list(args) + with contextlib.ExitStack() as stack: + if not any(str(a).startswith("-env:UserInstallation") for a in args): + profile = stack.enter_context( + tempfile.TemporaryDirectory(prefix="lo_profile_", ignore_cleanup_errors=True) + ) + args = [f"-env:UserInstallation={Path(profile).as_uri()}"] + args + return subprocess.run(["soffice"] + args, env=get_soffice_env(), **kwargs) + + + +_SHIM_SO = Path(tempfile.gettempdir()) / "lo_socket_shim.so" + + +def _needs_shim() -> bool: + try: + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s.close() + return False + except OSError: + return True + + +def _ensure_shim() -> Path: + if _SHIM_SO.exists(): + return _SHIM_SO + + src = Path(tempfile.gettempdir()) / "lo_socket_shim.c" + src.write_text(_SHIM_SOURCE, encoding="utf-8") + subprocess.run( + ["gcc", "-shared", "-fPIC", "-o", str(_SHIM_SO), str(src), "-ldl"], + check=True, + capture_output=True, + ) + src.unlink() + return _SHIM_SO + + + +_SHIM_SOURCE = r""" +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include + +static int (*real_socket)(int, int, int); +static int (*real_socketpair)(int, int, int, int[2]); +static int (*real_listen)(int, int); +static int (*real_accept)(int, struct sockaddr *, socklen_t *); +static int (*real_close)(int); +static int (*real_read)(int, void *, size_t); + +/* Per-FD bookkeeping (FDs >= 1024 are passed through unshimmed). */ +static int is_shimmed[1024]; +static int peer_of[1024]; +static int wake_r[1024]; /* accept() blocks reading this */ +static int wake_w[1024]; /* close() writes to this */ +static int listener_fd = -1; /* FD that received listen() */ + +__attribute__((constructor)) +static void init(void) { + real_socket = dlsym(RTLD_NEXT, "socket"); + real_socketpair = dlsym(RTLD_NEXT, "socketpair"); + real_listen = dlsym(RTLD_NEXT, "listen"); + real_accept = dlsym(RTLD_NEXT, "accept"); + real_close = dlsym(RTLD_NEXT, "close"); + real_read = dlsym(RTLD_NEXT, "read"); + for (int i = 0; i < 1024; i++) { + peer_of[i] = -1; + wake_r[i] = -1; + wake_w[i] = -1; + } +} + +/* ---- socket ---------------------------------------------------------- */ +int socket(int domain, int type, int protocol) { + if (domain == AF_UNIX) { + int fd = real_socket(domain, type, protocol); + if (fd >= 0) return fd; + /* socket(AF_UNIX) blocked – fall back to socketpair(). */ + int sv[2]; + if (real_socketpair(domain, type, protocol, sv) == 0) { + if (sv[0] >= 0 && sv[0] < 1024) { + is_shimmed[sv[0]] = 1; + peer_of[sv[0]] = sv[1]; + int wp[2]; + if (pipe(wp) == 0) { + wake_r[sv[0]] = wp[0]; + wake_w[sv[0]] = wp[1]; + } + } + return sv[0]; + } + errno = EPERM; + return -1; + } + return real_socket(domain, type, protocol); +} + +/* ---- listen ---------------------------------------------------------- */ +int listen(int sockfd, int backlog) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + listener_fd = sockfd; + return 0; + } + return real_listen(sockfd, backlog); +} + +/* ---- accept ---------------------------------------------------------- */ +int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + /* Block until close() writes to the wake pipe. */ + if (wake_r[sockfd] >= 0) { + char buf; + real_read(wake_r[sockfd], &buf, 1); + } + errno = ECONNABORTED; + return -1; + } + return real_accept(sockfd, addr, addrlen); +} + +/* ---- close ----------------------------------------------------------- */ +int close(int fd) { + if (fd >= 0 && fd < 1024 && is_shimmed[fd]) { + int was_listener = (fd == listener_fd); + is_shimmed[fd] = 0; + + if (wake_w[fd] >= 0) { /* unblock accept() */ + char c = 0; + write(wake_w[fd], &c, 1); + real_close(wake_w[fd]); + wake_w[fd] = -1; + } + if (wake_r[fd] >= 0) { real_close(wake_r[fd]); wake_r[fd] = -1; } + if (peer_of[fd] >= 0) { real_close(peer_of[fd]); peer_of[fd] = -1; } + + if (was_listener) + _exit(0); /* conversion done – exit */ + } + return real_close(fd); +} +""" + + + +if __name__ == "__main__": + import sys + result = run_soffice(sys.argv[1:]) + sys.exit(result.returncode) diff --git a/powerpoint/scripts/office/validate.py b/powerpoint/scripts/office/validate.py new file mode 100755 index 0000000..29ca186 --- /dev/null +++ b/powerpoint/scripts/office/validate.py @@ -0,0 +1,173 @@ +""" +Command line tool to validate Office document XML files against XSD schemas and tracked changes. + +Usage: + python validate.py [--original ] [--auto-repair] [--author NAME] + +The first argument can be either: +- An unpacked directory containing the Office document XML files +- A packed Office file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx template) which will be unpacked to a temp directory + +Auto-repair fixes: +- paraId/durableId values that exceed OOXML limits +- Missing xml:space="preserve" on w:t elements with whitespace +""" + +import argparse +import sys +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.ElementTree as ET +from defusedxml.common import DefusedXmlException + +from helpers import OOXML_FAMILY, rezip, safe_extract +from validators import DOCXSchemaValidator, PPTXSchemaValidator, RedliningValidator + +WORD_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + + +def _fail(message: str): + print(f"Error: {message}", file=sys.stderr) + sys.exit(2) + + +def _has_tracked_changes(unpacked_dir: Path) -> bool: + document = unpacked_dir / "word" / "document.xml" + if not document.is_file(): + return False + try: + root = ET.parse(document).getroot() + except (ET.ParseError, DefusedXmlException): + return False + tracked = {f"{{{WORD_NS}}}ins", f"{{{WORD_NS}}}del"} + return any(elem.tag in tracked for elem in root.iter()) + + +def main(): + parser = argparse.ArgumentParser(description="Validate Office document XML files") + parser.add_argument( + "path", + help="Path to unpacked directory or packed Office file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx)", + ) + parser.add_argument( + "--original", + required=False, + default=None, + help="Path to original file (.docx/.pptx/.xlsx or .dotx/.potx/.xltx). If omitted, all XSD errors are reported and redlining validation is skipped.", + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + help="Enable verbose output", + ) + parser.add_argument( + "--auto-repair", + action="store_true", + help="Automatically repair common issues (hex IDs, whitespace preservation). " + "Modifies the input in place: repairs to a packed file are written back to it.", + ) + parser.add_argument( + "--author", + default=None, + help="The name you are redlining under. Passing it turns on the " + "tracked-change check: any text differing from --original without a " + "/ recording it is reported. Untracked edits carry no " + "author, so the check covers them whoever made them — the name marks " + "the run as redlining work and is not used to filter. Requires " + "--original; docx only.", + ) + args = parser.parse_args() + + if args.author is not None and not args.original: + _fail("--author requires --original") + + path = Path(args.path) + if not path.exists(): + _fail(f"{path} does not exist") + + original_file = None + if args.original: + original_file = Path(args.original) + if not original_file.is_file(): + _fail(f"{original_file} is not a file") + if original_file.suffix.lower() not in OOXML_FAMILY: + _fail(f"{original_file} must be one of: {', '.join(sorted(OOXML_FAMILY))}") + + family = OOXML_FAMILY.get((original_file or path).suffix.lower()) + if family is None: + _fail( + f"Cannot determine file type from {path}. Use --original or provide one of: {', '.join(sorted(OOXML_FAMILY))}." + ) + + if args.author is not None and family != "docx": + _fail(f"--author only applies to docx files, not {family}") + + packed_file = None + temp_dir_ctx = None + if path.is_file() and path.suffix.lower() in OOXML_FAMILY: + packed_file = path + temp_dir_ctx = tempfile.TemporaryDirectory() + unpacked_dir = Path(temp_dir_ctx.name) + try: + with zipfile.ZipFile(path, "r") as zf: + safe_extract(zf, unpacked_dir) + except (zipfile.BadZipFile, ValueError, OSError) as e: + _fail(f"cannot unpack {path}: {e}") + else: + if not path.is_dir(): + _fail(f"{path} is not a directory or Office file") + unpacked_dir = path + + match family: + case "docx": + validators = [ + DOCXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), + ] + if args.author is not None: + validators.append( + RedliningValidator(unpacked_dir, original_file, verbose=args.verbose) + ) + elif original_file and _has_tracked_changes(unpacked_dir): + print( + "Note: this document has tracked changes; they were not " + "checked against the original (pass --author to check)." + ) + case "pptx": + validators = [ + PPTXSchemaValidator(unpacked_dir, original_file, verbose=args.verbose), + ] + case "xlsx": + exts = ", ".join(k for k, v in sorted(OOXML_FAMILY.items()) if v == "xlsx") + print( + f"No XSD schema validation is performed for xlsx-family files ({exts}). " + "For formula-error checking, use scripts/recalc.py instead." + ) + sys.exit(0) + case _: + print(f"Error: Validation not supported for file type {family}") + sys.exit(1) + + if args.auto_repair: + total_repairs = sum(v.repair() for v in validators) + if total_repairs: + print(f"Auto-repaired {total_repairs} issue(s)") + if packed_file is not None: + rezip(unpacked_dir, packed_file) + print(f"Wrote repaired file to {packed_file}") + + success = all([v.validate() for v in validators]) + + if temp_dir_ctx is not None: + temp_dir_ctx.cleanup() + + if success: + print("All validations PASSED!") + + sys.exit(0 if success else 1) + + +if __name__ == "__main__": + main() diff --git a/powerpoint/scripts/office/validators/__init__.py b/powerpoint/scripts/office/validators/__init__.py new file mode 100644 index 0000000..db092ec --- /dev/null +++ b/powerpoint/scripts/office/validators/__init__.py @@ -0,0 +1,15 @@ +""" +Validation modules for Word document processing. +""" + +from .base import BaseSchemaValidator +from .docx import DOCXSchemaValidator +from .pptx import PPTXSchemaValidator +from .redlining import RedliningValidator + +__all__ = [ + "BaseSchemaValidator", + "DOCXSchemaValidator", + "PPTXSchemaValidator", + "RedliningValidator", +] diff --git a/powerpoint/scripts/office/validators/base.py b/powerpoint/scripts/office/validators/base.py new file mode 100644 index 0000000..f928b2c --- /dev/null +++ b/powerpoint/scripts/office/validators/base.py @@ -0,0 +1,875 @@ +""" +Base validator with common validation logic for document files. +""" + +import re +from pathlib import Path + +import defusedxml.minidom +from functools import lru_cache + +import lxml.etree + +from helpers import safe_extract + + +@lru_cache(maxsize=None) +def _load_schema(schema_path: str): + with open(schema_path, "rb") as xsd_file: + xsd_doc = lxml.etree.parse( + xsd_file, parser=lxml.etree.XMLParser(), base_url=schema_path + ) + return lxml.etree.XMLSchema(xsd_doc) + +class BaseSchemaValidator: + + IGNORED_VALIDATION_ERRORS = [ + "hyphenationZone", + "purl.org/dc/terms", + ] + + UNIQUE_ID_REQUIREMENTS = { + "comment": ("id", "file"), + "commentrangestart": ("id", "file"), + "commentrangeend": ("id", "file"), + "bookmarkstart": ("id", "file"), + "bookmarkend": ("id", "file"), + "sldid": ("id", "file"), + "sldmasterid": ("id", "global"), + "sldlayoutid": ("id", "global"), + "cm": ("authorid", "file"), + "sheet": ("sheetid", "file"), + "definedname": ("id", "file"), + "cxnsp": ("id", "file"), + "sp": ("id", "file"), + "pic": ("id", "file"), + "grpsp": ("id", "file"), + } + + EXCLUDED_ID_CONTAINERS = { + "sectionlst", + } + + ELEMENT_RELATIONSHIP_TYPES = {} + + SCHEMA_MAPPINGS = { + "word": "ISO-IEC29500-4_2016/wml.xsd", + "ppt": "ISO-IEC29500-4_2016/pml.xsd", + "xl": "ISO-IEC29500-4_2016/sml.xsd", + "[Content_Types].xml": "ecma/fourth-edition/opc-contentTypes.xsd", + "app.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesExtended.xsd", + "core.xml": "ecma/fourth-edition/opc-coreProperties.xsd", + "custom.xml": "ISO-IEC29500-4_2016/shared-documentPropertiesCustom.xsd", + ".rels": "ecma/fourth-edition/opc-relationships.xsd", + "people.xml": "microsoft/wml-2012.xsd", + "commentsIds.xml": "microsoft/wml-cid-2016.xsd", + "commentsExtensible.xml": "microsoft/wml-cex-2018.xsd", + "commentsExtended.xml": "microsoft/wml-2012.xsd", + "chart": "ISO-IEC29500-4_2016/dml-chart.xsd", + "theme": "ISO-IEC29500-4_2016/dml-main.xsd", + "drawing": "ISO-IEC29500-4_2016/dml-main.xsd", + } + + MC_NAMESPACE = "http://schemas.openxmlformats.org/markup-compatibility/2006" + XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace" + + PACKAGE_RELATIONSHIPS_NAMESPACE = ( + "http://schemas.openxmlformats.org/package/2006/relationships" + ) + OFFICE_RELATIONSHIPS_NAMESPACE = ( + "http://schemas.openxmlformats.org/officeDocument/2006/relationships" + ) + CONTENT_TYPES_NAMESPACE = ( + "http://schemas.openxmlformats.org/package/2006/content-types" + ) + + MAIN_CONTENT_FOLDERS = {"word", "ppt", "xl"} + + OOXML_NAMESPACES = { + "http://schemas.openxmlformats.org/officeDocument/2006/math", + "http://schemas.openxmlformats.org/officeDocument/2006/relationships", + "http://schemas.openxmlformats.org/schemaLibrary/2006/main", + "http://schemas.openxmlformats.org/drawingml/2006/main", + "http://schemas.openxmlformats.org/drawingml/2006/chart", + "http://schemas.openxmlformats.org/drawingml/2006/chartDrawing", + "http://schemas.openxmlformats.org/drawingml/2006/diagram", + "http://schemas.openxmlformats.org/drawingml/2006/picture", + "http://schemas.openxmlformats.org/drawingml/2006/spreadsheetDrawing", + "http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing", + "http://schemas.openxmlformats.org/wordprocessingml/2006/main", + "http://schemas.openxmlformats.org/presentationml/2006/main", + "http://schemas.openxmlformats.org/spreadsheetml/2006/main", + "http://schemas.openxmlformats.org/officeDocument/2006/sharedTypes", + "http://www.w3.org/XML/1998/namespace", + } + + def __init__(self, unpacked_dir, original_file=None, verbose=False): + self.unpacked_dir = Path(unpacked_dir).resolve() + self.original_file = Path(original_file) if original_file else None + self.verbose = verbose + + self.schemas_dir = Path(__file__).parent.parent / "schemas" + + patterns = ["*.xml", "*.rels"] + self.xml_files = [ + f for pattern in patterns for f in self.unpacked_dir.rglob(pattern) + ] + + if not self.xml_files: + print(f"Warning: No XML files found in {self.unpacked_dir}") + + def validate(self): + raise NotImplementedError("Subclasses must implement the validate method") + + def repair(self) -> int: + return self.repair_whitespace_preservation() + + def repair_whitespace_preservation(self) -> int: + repairs = 0 + + for xml_file in self.xml_files: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + pending = [] + + for elem in dom.getElementsByTagName("*"): + local_name = elem.tagName.rsplit(":", 1)[-1] + if local_name in ("t", "delText", "instrText", "delInstrText"): + text = "".join( + child.data + for child in elem.childNodes + if child.nodeType in (child.TEXT_NODE, child.CDATA_SECTION_NODE) + ) + ws = (" ", "\t", "\n", "\r") + if text and (text.startswith(ws) or text.endswith(ws)): + if elem.getAttribute("xml:space") != "preserve": + elem.setAttribute("xml:space", "preserve") + text_preview = repr(text[:30]) + "..." if len(text) > 30 else repr(text) + pending.append(f" Repaired: {xml_file.name}: Added xml:space='preserve' to {elem.tagName}: {text_preview}") + + if pending: + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + for message in pending: + print(message) + repairs += len(pending) + + except Exception: + pass + + return repairs + + def validate_xml(self): + errors = [] + + for xml_file in self.xml_files: + try: + lxml.etree.parse(str(xml_file)) + except lxml.etree.XMLSyntaxError as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {e.lineno}: {e.msg}" + ) + except Exception as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Unexpected error: {str(e)}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} XML violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All XML files are well-formed") + return True + + def validate_namespaces(self): + errors = [] + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + declared = set(root.nsmap.keys()) - {None} + + for attr_val in [ + v for k, v in root.attrib.items() if k.endswith("Ignorable") + ]: + undeclared = set(attr_val.split()) - declared + errors.extend( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Namespace '{ns}' in Ignorable but not declared" + for ns in undeclared + ) + except lxml.etree.XMLSyntaxError: + continue + + if errors: + print(f"FAILED - {len(errors)} namespace issues:") + for error in errors: + print(error) + return False + if self.verbose: + print("PASSED - All namespace prefixes properly declared") + return True + + def validate_unique_ids(self): + errors = [] + global_ids = {} + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + file_ids = {} + + mc_elements = root.xpath( + ".//mc:AlternateContent", namespaces={"mc": self.MC_NAMESPACE} + ) + for elem in mc_elements: + elem.getparent().remove(elem) + + for elem in root.iter(): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + tag = ( + elem.tag.split("}")[-1].lower() + if "}" in elem.tag + else elem.tag.lower() + ) + + if tag in self.UNIQUE_ID_REQUIREMENTS: + in_excluded_container = any( + ancestor.tag.split("}")[-1].lower() in self.EXCLUDED_ID_CONTAINERS + for ancestor in elem.iterancestors() + ) + if in_excluded_container: + continue + + attr_name, scope = self.UNIQUE_ID_REQUIREMENTS[tag] + + id_value = None + for attr, value in elem.attrib.items(): + attr_local = ( + attr.split("}")[-1].lower() + if "}" in attr + else attr.lower() + ) + if attr_local == attr_name: + id_value = value + break + + if id_value is not None: + if scope == "global": + if id_value in global_ids: + prev_file, prev_line, prev_tag = global_ids[ + id_value + ] + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: Global ID '{id_value}' in <{tag}> " + f"already used in {prev_file} at line {prev_line} in <{prev_tag}>" + ) + else: + global_ids[id_value] = ( + xml_file.relative_to(self.unpacked_dir), + elem.sourceline, + tag, + ) + elif scope == "file": + key = (tag, attr_name) + if key not in file_ids: + file_ids[key] = {} + + if id_value in file_ids[key]: + prev_line = file_ids[key][id_value] + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: Duplicate {attr_name}='{id_value}' in <{tag}> " + f"(first occurrence at line {prev_line})" + ) + else: + file_ids[key][id_value] = elem.sourceline + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} ID uniqueness violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All required IDs are unique") + return True + + def validate_file_references(self): + errors = [] + + rels_files = list(self.unpacked_dir.rglob("*.rels")) + + if not rels_files: + if self.verbose: + print("PASSED - No .rels files found") + return True + + all_files = [] + for file_path in self.unpacked_dir.rglob("*"): + if ( + file_path.is_file() + and file_path.name != "[Content_Types].xml" + and not file_path.name.endswith(".rels") + ): + all_files.append(file_path.resolve()) + + all_referenced_files = set() + + if self.verbose: + print( + f"Found {len(rels_files)} .rels files and {len(all_files)} target files" + ) + + for rels_file in rels_files: + try: + rels_root = lxml.etree.parse(str(rels_file)).getroot() + + rels_dir = rels_file.parent + + referenced_files = set() + broken_refs = [] + + for rel in rels_root.findall( + ".//ns:Relationship", + namespaces={"ns": self.PACKAGE_RELATIONSHIPS_NAMESPACE}, + ): + target = rel.get("Target") + if rel.get("TargetMode") == "External": + continue + if target and not target.startswith( + ("http", "mailto:") + ): + if target.startswith("/"): + target_path = self.unpacked_dir / target.lstrip("/") + elif rels_file.name == ".rels": + target_path = self.unpacked_dir / target + else: + base_dir = rels_dir.parent + target_path = base_dir / target + + try: + target_path = target_path.resolve() + if target_path.exists() and target_path.is_file(): + referenced_files.add(target_path) + all_referenced_files.add(target_path) + else: + broken_refs.append((target, rel.sourceline)) + except (OSError, ValueError): + broken_refs.append((target, rel.sourceline)) + + if broken_refs: + rel_path = rels_file.relative_to(self.unpacked_dir) + for broken_ref, line_num in broken_refs: + errors.append( + f" {rel_path}: Line {line_num}: Broken reference to {broken_ref}" + ) + + except Exception as e: + rel_path = rels_file.relative_to(self.unpacked_dir) + errors.append(f" Error parsing {rel_path}: {e}") + + unreferenced_files = set(all_files) - all_referenced_files + + if unreferenced_files: + for unref_file in sorted(unreferenced_files): + unref_rel_path = unref_file.relative_to(self.unpacked_dir) + errors.append(f" Unreferenced file: {unref_rel_path}") + + if errors: + print(f"FAILED - Found {len(errors)} relationship validation errors:") + for error in errors: + print(error) + print( + "CRITICAL: These errors will cause the document to appear corrupt. " + + "Broken references MUST be fixed, " + + "and unreferenced files MUST be referenced or removed." + ) + return False + else: + if self.verbose: + print( + "PASSED - All references are valid and all files are properly referenced" + ) + return True + + def validate_all_relationship_ids(self): + import lxml.etree + + errors = [] + + for xml_file in self.xml_files: + if xml_file.suffix == ".rels": + continue + + rels_dir = xml_file.parent / "_rels" + rels_file = rels_dir / f"{xml_file.name}.rels" + + if not rels_file.exists(): + continue + + try: + rels_root = lxml.etree.parse(str(rels_file)).getroot() + rid_to_type = {} + + for rel in rels_root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rid = rel.get("Id") + rel_type = rel.get("Type", "") + if rid: + if rid in rid_to_type: + rels_rel_path = rels_file.relative_to(self.unpacked_dir) + errors.append( + f" {rels_rel_path}: Line {rel.sourceline}: " + f"Duplicate relationship ID '{rid}' (IDs must be unique)" + ) + type_name = ( + rel_type.split("/")[-1] if "/" in rel_type else rel_type + ) + rid_to_type[rid] = type_name + + xml_root = lxml.etree.parse(str(xml_file)).getroot() + + r_ns = self.OFFICE_RELATIONSHIPS_NAMESPACE + rid_attrs_to_check = ["id", "embed", "link"] + for elem in xml_root.iter(): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + for attr_name in rid_attrs_to_check: + rid_attr = elem.get(f"{{{r_ns}}}{attr_name}") + if not rid_attr: + continue + xml_rel_path = xml_file.relative_to(self.unpacked_dir) + elem_name = ( + elem.tag.split("}")[-1] if "}" in elem.tag else elem.tag + ) + + if rid_attr not in rid_to_type: + errors.append( + f" {xml_rel_path}: Line {elem.sourceline}: " + f"<{elem_name}> r:{attr_name} references non-existent relationship '{rid_attr}' " + f"(valid IDs: {', '.join(sorted(rid_to_type.keys())[:5])}{'...' if len(rid_to_type) > 5 else ''})" + ) + elif attr_name == "id" and self.ELEMENT_RELATIONSHIP_TYPES: + expected_type = self._get_expected_relationship_type( + elem_name + ) + if expected_type: + actual_type = rid_to_type[rid_attr] + if expected_type not in actual_type.lower(): + errors.append( + f" {xml_rel_path}: Line {elem.sourceline}: " + f"<{elem_name}> references '{rid_attr}' which points to '{actual_type}' " + f"but should point to a '{expected_type}' relationship" + ) + + except Exception as e: + xml_rel_path = xml_file.relative_to(self.unpacked_dir) + errors.append(f" Error processing {xml_rel_path}: {e}") + + if errors: + print(f"FAILED - Found {len(errors)} relationship ID reference errors:") + for error in errors: + print(error) + print("\nThese ID mismatches will cause the document to appear corrupt!") + return False + else: + if self.verbose: + print("PASSED - All relationship ID references are valid") + return True + + def _get_expected_relationship_type(self, element_name): + elem_lower = element_name.lower() + + if elem_lower in self.ELEMENT_RELATIONSHIP_TYPES: + return self.ELEMENT_RELATIONSHIP_TYPES[elem_lower] + + if elem_lower.endswith("id") and len(elem_lower) > 2: + prefix = elem_lower[:-2] + if prefix.endswith("master"): + return prefix.lower() + elif prefix.endswith("layout"): + return prefix.lower() + else: + if prefix == "sld": + return "slide" + return prefix.lower() + + if elem_lower.endswith("reference") and len(elem_lower) > 9: + prefix = elem_lower[:-9] + return prefix.lower() + + return None + + def validate_content_types(self): + errors = [] + + content_types_file = self.unpacked_dir / "[Content_Types].xml" + if not content_types_file.exists(): + print("FAILED - [Content_Types].xml file not found") + return False + + try: + root = lxml.etree.parse(str(content_types_file)).getroot() + declared_parts = set() + declared_extensions = set() + + for override in root.findall( + f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Override" + ): + part_name = override.get("PartName") + if part_name is not None: + declared_parts.add(part_name.lstrip("/")) + + for default in root.findall( + f".//{{{self.CONTENT_TYPES_NAMESPACE}}}Default" + ): + extension = default.get("Extension") + if extension is not None: + declared_extensions.add(extension.lower()) + + declarable_roots = { + "sld", + "sldLayout", + "sldMaster", + "presentation", + "document", + "workbook", + "worksheet", + "theme", + } + + media_extensions = { + "png": "image/png", + "jpg": "image/jpeg", + "jpeg": "image/jpeg", + "gif": "image/gif", + "bmp": "image/bmp", + "tiff": "image/tiff", + "wmf": "image/x-wmf", + "emf": "image/x-emf", + } + + all_files = list(self.unpacked_dir.rglob("*")) + all_files = [f for f in all_files if f.is_file()] + + for xml_file in self.xml_files: + path_str = str(xml_file.relative_to(self.unpacked_dir)).replace( + "\\", "/" + ) + + if any( + skip in path_str + for skip in [".rels", "[Content_Types]", "docProps/", "_rels/"] + ): + continue + + try: + root_tag = lxml.etree.parse(str(xml_file)).getroot().tag + root_name = root_tag.split("}")[-1] if "}" in root_tag else root_tag + + if root_name in declarable_roots and path_str not in declared_parts: + errors.append( + f" {path_str}: File with <{root_name}> root not declared in [Content_Types].xml" + ) + + except Exception: + continue + + for file_path in all_files: + if file_path.suffix.lower() in {".xml", ".rels"}: + continue + if file_path.name == "[Content_Types].xml": + continue + if "_rels" in file_path.parts or "docProps" in file_path.parts: + continue + + extension = file_path.suffix.lstrip(".").lower() + if extension and extension not in declared_extensions: + if extension in media_extensions: + relative_path = file_path.relative_to(self.unpacked_dir) + errors.append( + f' {relative_path}: File with extension \'{extension}\' not declared in [Content_Types].xml - should add: ' + ) + + except Exception as e: + errors.append(f" Error parsing [Content_Types].xml: {e}") + + if errors: + print(f"FAILED - Found {len(errors)} content type declaration errors:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print( + "PASSED - All content files are properly declared in [Content_Types].xml" + ) + return True + + def validate_file_against_xsd(self, xml_file, verbose=False): + xml_file = Path(xml_file).resolve() + unpacked_dir = self.unpacked_dir.resolve() + + is_valid, current_errors = self._validate_single_file_xsd( + xml_file, unpacked_dir + ) + + if is_valid is None: + return None, set() + elif is_valid: + return True, set() + + original_errors = self._get_original_file_errors(xml_file) + + assert current_errors is not None + new_errors = current_errors - original_errors + + new_errors = { + e for e in new_errors + if not any(pattern in e for pattern in self.IGNORED_VALIDATION_ERRORS) + } + + if new_errors: + if verbose: + relative_path = xml_file.relative_to(unpacked_dir) + print(f"FAILED - {relative_path}: {len(new_errors)} new error(s)") + for error in list(new_errors)[:3]: + truncated = error[:250] + "..." if len(error) > 250 else error + print(f" - {truncated}") + return False, new_errors + else: + if verbose: + print( + f"PASSED - No new errors (original had {len(current_errors)} errors)" + ) + return True, set() + + def validate_against_xsd(self): + new_errors = [] + original_error_count = 0 + valid_count = 0 + skipped_count = 0 + + for xml_file in self.xml_files: + relative_path = str(xml_file.relative_to(self.unpacked_dir)) + is_valid, new_file_errors = self.validate_file_against_xsd( + xml_file, verbose=False + ) + + if is_valid is None: + skipped_count += 1 + continue + elif is_valid and not new_file_errors: + valid_count += 1 + continue + elif is_valid: + original_error_count += 1 + valid_count += 1 + continue + + new_errors.append(f" {relative_path}: {len(new_file_errors)} new error(s)") + for error in list(new_file_errors)[:3]: + new_errors.append( + f" - {error[:250]}..." if len(error) > 250 else f" - {error}" + ) + + if self.verbose: + print(f"Validated {len(self.xml_files)} files:") + print(f" - Valid: {valid_count}") + print(f" - Skipped (no schema): {skipped_count}") + if original_error_count: + print(f" - With original errors (ignored): {original_error_count}") + print( + f" - With NEW errors: {len(new_errors) > 0 and len([e for e in new_errors if not e.startswith(' ')]) or 0}" + ) + + if new_errors: + print("\nFAILED - Found NEW validation errors:") + for error in new_errors: + print(error) + return False + else: + if self.verbose: + print("\nPASSED - No new XSD validation errors introduced") + return True + + def _get_schema_path(self, xml_file): + if xml_file.name in self.SCHEMA_MAPPINGS: + return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.name] + + if xml_file.suffix == ".rels": + return self.schemas_dir / self.SCHEMA_MAPPINGS[".rels"] + + if "charts/" in str(xml_file) and xml_file.name.startswith("chart"): + return self.schemas_dir / self.SCHEMA_MAPPINGS["chart"] + + if "theme/" in str(xml_file) and xml_file.name.startswith("theme"): + return self.schemas_dir / self.SCHEMA_MAPPINGS["theme"] + + if xml_file.parent.name in self.MAIN_CONTENT_FOLDERS: + return self.schemas_dir / self.SCHEMA_MAPPINGS[xml_file.parent.name] + + return None + + def _clean_ignorable_namespaces(self, xml_doc): + xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") + xml_copy = lxml.etree.fromstring(xml_string) + + for elem in xml_copy.iter(): + attrs_to_remove = [] + + for attr in elem.attrib: + if "{" in attr: + ns = attr.split("}")[0][1:] + if ns not in self.OOXML_NAMESPACES: + attrs_to_remove.append(attr) + + for attr in attrs_to_remove: + del elem.attrib[attr] + + self._remove_ignorable_elements(xml_copy) + + return lxml.etree.ElementTree(xml_copy) + + def _remove_ignorable_elements(self, root): + elements_to_remove = [] + + for elem in list(root): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + + tag_str = str(elem.tag) + if tag_str.startswith("{"): + ns = tag_str.split("}")[0][1:] + if ns not in self.OOXML_NAMESPACES: + elements_to_remove.append(elem) + continue + + self._remove_ignorable_elements(elem) + + for elem in elements_to_remove: + root.remove(elem) + + def _preprocess_for_mc_ignorable(self, xml_doc): + root = xml_doc.getroot() + + if f"{{{self.MC_NAMESPACE}}}Ignorable" in root.attrib: + del root.attrib[f"{{{self.MC_NAMESPACE}}}Ignorable"] + + return xml_doc + + def _preprocess_for_schema(self, xml_doc, relative_path): + return xml_doc + + def _validate_single_file_xsd(self, xml_file, base_path, schema_path=None): + schema_path = schema_path or self._get_schema_path(xml_file) + if not schema_path: + return None, None + + try: + schema = _load_schema(str(schema_path)) + + with open(xml_file, "rb") as f: + xml_doc = lxml.etree.parse(f) + + xml_doc, _ = self._remove_template_tags_from_text_nodes(xml_doc) + xml_doc = self._preprocess_for_mc_ignorable(xml_doc) + + relative_path = xml_file.relative_to(base_path) + if ( + relative_path.parts + and relative_path.parts[0] in self.MAIN_CONTENT_FOLDERS + ): + xml_doc = self._clean_ignorable_namespaces(xml_doc) + + xml_doc = self._preprocess_for_schema(xml_doc, relative_path) + + if schema.validate(xml_doc): + return True, set() + else: + errors = set() + for error in schema.error_log: + errors.add(error.message) + return False, errors + + except Exception as e: + return False, {str(e)} + + def _get_original_file_errors(self, xml_file, schema_path=None): + if self.original_file is None: + return set() + + import tempfile + import zipfile + + xml_file = Path(xml_file).resolve() + unpacked_dir = self.unpacked_dir.resolve() + relative_path = xml_file.relative_to(unpacked_dir) + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + try: + with zipfile.ZipFile(self.original_file, "r") as zip_ref: + safe_extract(zip_ref, temp_path) + except (zipfile.BadZipFile, ValueError, OSError): + return set() + + original_xml_file = temp_path / relative_path + + if not original_xml_file.exists(): + return set() + + is_valid, errors = self._validate_single_file_xsd( + original_xml_file, temp_path, schema_path=schema_path + ) + return errors if errors else set() + + def _remove_template_tags_from_text_nodes(self, xml_doc): + warnings = [] + template_pattern = re.compile(r"\{\{[^}]*\}\}") + + xml_string = lxml.etree.tostring(xml_doc, encoding="unicode") + xml_copy = lxml.etree.fromstring(xml_string) + + def process_text_content(text, content_type): + if not text: + return text + matches = list(template_pattern.finditer(text)) + if matches: + for match in matches: + warnings.append( + f"Found template tag in {content_type}: {match.group()}" + ) + return template_pattern.sub("", text) + return text + + for elem in xml_copy.iter(): + if not hasattr(elem, "tag") or callable(elem.tag): + continue + tag_str = str(elem.tag) + if tag_str.endswith("}t") or tag_str == "t": + continue + + elem.text = process_text_content(elem.text, "text content") + elem.tail = process_text_content(elem.tail, "tail content") + + return lxml.etree.ElementTree(xml_copy), warnings + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/powerpoint/scripts/office/validators/docx.py b/powerpoint/scripts/office/validators/docx.py new file mode 100644 index 0000000..b181499 --- /dev/null +++ b/powerpoint/scripts/office/validators/docx.py @@ -0,0 +1,466 @@ +""" +Validator for Word document XML files against XSD schemas. +""" + +import random +import re +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.minidom +import lxml.etree + +from helpers import safe_extract + +from .base import BaseSchemaValidator + + +class DOCXSchemaValidator(BaseSchemaValidator): + + WORD_2006_NAMESPACE = "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + W14_NAMESPACE = "http://schemas.microsoft.com/office/word/2010/wordml" + W16CID_NAMESPACE = "http://schemas.microsoft.com/office/word/2016/wordml/cid" + + ELEMENT_RELATIONSHIP_TYPES = {} + + def validate(self): + if not self.validate_xml(): + return False + + all_valid = True + if not self.validate_namespaces(): + all_valid = False + + if not self.validate_unique_ids(): + all_valid = False + + if not self.validate_file_references(): + all_valid = False + + if not self.validate_content_types(): + all_valid = False + + if not self.validate_against_xsd(): + all_valid = False + + if not self.validate_whitespace_preservation(): + all_valid = False + + if not self.validate_deletions(): + all_valid = False + + if not self.validate_insertions(): + all_valid = False + + if not self.validate_all_relationship_ids(): + all_valid = False + + if not self.validate_id_constraints(): + all_valid = False + + if not self.validate_comment_markers(): + all_valid = False + + self.compare_paragraph_counts() + + return all_valid + + def validate_whitespace_preservation(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + + for elem in root.iter(f"{{{self.WORD_2006_NAMESPACE}}}t"): + if elem.text: + text = elem.text + if re.search(r"^[ \t\n\r]", text) or re.search( + r"[ \t\n\r]$", text + ): + xml_space_attr = f"{{{self.XML_NAMESPACE}}}space" + if ( + xml_space_attr not in elem.attrib + or elem.attrib[xml_space_attr] != "preserve" + ): + text_preview = ( + repr(text)[:50] + "..." + if len(repr(text)) > 50 + else repr(text) + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: w:t element with whitespace missing xml:space='preserve': {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} whitespace preservation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All whitespace is properly preserved") + return True + + def validate_deletions(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + for t_elem in root.xpath(".//w:del//w:t", namespaces=namespaces): + if t_elem.text: + text_preview = ( + repr(t_elem.text)[:50] + "..." + if len(repr(t_elem.text)) > 50 + else repr(t_elem.text) + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {t_elem.sourceline}: found within : {text_preview}" + ) + + for instr_elem in root.xpath( + ".//w:del//w:instrText", namespaces=namespaces + ): + text_preview = ( + repr(instr_elem.text or "")[:50] + "..." + if len(repr(instr_elem.text or "")) > 50 + else repr(instr_elem.text or "") + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {instr_elem.sourceline}: found within (use ): {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} deletion validation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - No w:t elements found within w:del elements") + return True + + def count_paragraphs_in_unpacked(self): + count = 0 + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") + count = len(paragraphs) + except Exception as e: + print(f"Error counting paragraphs in unpacked document: {e}") + + return count + + def count_paragraphs_in_original(self): + original = self.original_file + if original is None: + return 0 + + count = 0 + + try: + with tempfile.TemporaryDirectory() as temp_dir: + with zipfile.ZipFile(original, "r") as zip_ref: + safe_extract(zip_ref, Path(temp_dir)) + + doc_xml_path = temp_dir + "/word/document.xml" + root = lxml.etree.parse(doc_xml_path).getroot() + + paragraphs = root.findall(f".//{{{self.WORD_2006_NAMESPACE}}}p") + count = len(paragraphs) + + except Exception as e: + print(f"Error counting paragraphs in original document: {e}") + + return count + + def validate_insertions(self): + errors = [] + + for xml_file in self.xml_files: + if xml_file.name != "document.xml": + continue + + try: + root = lxml.etree.parse(str(xml_file)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + invalid_elements = root.xpath( + ".//w:ins//w:delText[not(ancestor::w:del)]", namespaces=namespaces + ) + + for elem in invalid_elements: + text_preview = ( + repr(elem.text or "")[:50] + "..." + if len(repr(elem.text or "")) > 50 + else repr(elem.text or "") + ) + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: within : {text_preview}" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} insertion validation violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - No w:delText elements within w:ins elements") + return True + + def compare_paragraph_counts(self): + new_count = self.count_paragraphs_in_unpacked() + if self.original_file is None: + print(f"\nParagraphs: {new_count}") + return + + original_count = self.count_paragraphs_in_original() + diff = new_count - original_count + diff_str = f"+{diff}" if diff > 0 else str(diff) + print(f"\nParagraphs: {original_count} → {new_count} ({diff_str})") + + def _parse_id_value(self, val: str, base: int = 16) -> int: + return int(val, base) + + def validate_id_constraints(self): + errors = [] + para_id_attr = f"{{{self.W14_NAMESPACE}}}paraId" + durable_id_attr = f"{{{self.W16CID_NAMESPACE}}}durableId" + + for xml_file in self.xml_files: + try: + for elem in lxml.etree.parse(str(xml_file)).iter(): + if val := elem.get(para_id_attr): + try: + if self._parse_id_value(val, base=16) >= 0x80000000: + errors.append( + f" {xml_file.name}:{elem.sourceline}: paraId={val} >= 0x80000000" + ) + except ValueError: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"paraId={val} is not valid hex" + ) + + if val := elem.get(durable_id_attr): + if xml_file.name == "numbering.xml": + try: + if self._parse_id_value(val, base=10) >= 0x7FFFFFFF: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} >= 0x7FFFFFFF" + ) + except ValueError: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} must be decimal in numbering.xml" + ) + else: + try: + if self._parse_id_value(val, base=16) >= 0x7FFFFFFF: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} >= 0x7FFFFFFF" + ) + except ValueError: + errors.append( + f" {xml_file.name}:{elem.sourceline}: " + f"durableId={val} is not valid hex" + ) + except lxml.etree.XMLSyntaxError: + continue + + if errors: + print(f"FAILED - {len(errors)} ID constraint violations:") + for e in errors: + print(e) + elif self.verbose: + print("PASSED - All paraId/durableId values within constraints") + return not errors + + def validate_comment_markers(self): + errors = [] + + document_xml = None + comments_xml = None + for xml_file in self.xml_files: + if xml_file.name == "document.xml" and "word" in str(xml_file): + document_xml = xml_file + elif xml_file.name == "comments.xml": + comments_xml = xml_file + + if not document_xml: + if self.verbose: + print("PASSED - No document.xml found (skipping comment validation)") + return True + + try: + doc_root = lxml.etree.parse(str(document_xml)).getroot() + namespaces = {"w": self.WORD_2006_NAMESPACE} + + range_starts = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentRangeStart", namespaces=namespaces + ) + } + range_ends = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentRangeEnd", namespaces=namespaces + ) + } + references = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in doc_root.xpath( + ".//w:commentReference", namespaces=namespaces + ) + } + + orphaned_ends = range_ends - range_starts + for comment_id in sorted( + orphaned_ends, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + errors.append( + f' document.xml: commentRangeEnd id="{comment_id}" has no matching commentRangeStart' + ) + + orphaned_starts = range_starts - range_ends + for comment_id in sorted( + orphaned_starts, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + errors.append( + f' document.xml: commentRangeStart id="{comment_id}" has no matching commentRangeEnd' + ) + + comment_ids = set() + if comments_xml and comments_xml.exists(): + comments_root = lxml.etree.parse(str(comments_xml)).getroot() + comment_ids = { + elem.get(f"{{{self.WORD_2006_NAMESPACE}}}id") + for elem in comments_root.xpath( + ".//w:comment", namespaces=namespaces + ) + } + + marker_ids = range_starts | range_ends | references + invalid_refs = marker_ids - comment_ids + for comment_id in sorted( + invalid_refs, key=lambda x: int(x) if x and x.isdigit() else 0 + ): + if comment_id: + errors.append( + f' document.xml: marker id="{comment_id}" references non-existent comment' + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append(f" Error parsing XML: {e}") + + if errors: + print(f"FAILED - {len(errors)} comment marker violations:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All comment markers properly paired") + return True + + def repair(self) -> int: + repairs = super().repair() + repairs += self.repair_durableId() + return repairs + + def repair_durableId(self) -> int: + DURABLE_ID_ATTRS = ("w16cid:durableId", "w16cex:durableId") + repairs = 0 + renames: dict = {} + + for xml_file in self.xml_files: + try: + content = xml_file.read_text(encoding="utf-8") + dom = defusedxml.minidom.parseString(content) + is_numbering = xml_file.name == "numbering.xml" + base = 10 if is_numbering else 16 + pending = [] + seen_in_file = set() + modified = False + + for elem in dom.getElementsByTagName("*"): + for attr_name in DURABLE_ID_ATTRS: + if not elem.hasAttribute(attr_name): + continue + + durable_id = elem.getAttribute(attr_name) + try: + key = self._parse_id_value(durable_id, base=base) + needs_repair = key >= 0x7FFFFFFF + except ValueError: + key = durable_id + needs_repair = True + + if needs_repair: + if key in seen_in_file: + value = random.randint(1, 0x7FFFFFFE) + else: + seen_in_file.add(key) + if key not in renames: + renames[key] = random.randint(1, 0x7FFFFFFE) + value = renames[key] + new_id = str(value) if is_numbering else f"{value:08X}" + + elem.setAttribute(attr_name, new_id) + pending.append( + f" Repaired: {xml_file.name}: durableId {durable_id} → {new_id}" + ) + modified = True + + if modified: + xml_file.write_bytes(dom.toxml(encoding="UTF-8")) + for message in pending: + print(message) + repairs += len(pending) + + except Exception: + pass + + return repairs + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/powerpoint/scripts/office/validators/pptx.py b/powerpoint/scripts/office/validators/pptx.py new file mode 100644 index 0000000..318f0e6 --- /dev/null +++ b/powerpoint/scripts/office/validators/pptx.py @@ -0,0 +1,441 @@ +""" +Validator for PowerPoint presentation XML files against XSD schemas. +""" + +import re +from pathlib import Path + +from helpers import opc_target, rels_source_part, safe_extract + +from .base import BaseSchemaValidator + + +class PPTXSchemaValidator(BaseSchemaValidator): + + PRESENTATIONML_NAMESPACE = ( + "http://schemas.openxmlformats.org/presentationml/2006/main" + ) + + ELEMENT_RELATIONSHIP_TYPES = { + "sldid": "slide", + "sldmasterid": "slidemaster", + "notesmasterid": "notesmaster", + "sldlayoutid": "slidelayout", + "themeid": "theme", + "tablestyleid": "tablestyles", + } + + def validate(self): + if not self.validate_xml(): + return False + + all_valid = True + if not self.validate_namespaces(): + all_valid = False + + if not self.validate_unique_ids(): + all_valid = False + + if not self.validate_uuid_ids(): + all_valid = False + + if not self.validate_file_references(): + all_valid = False + + if not self.validate_slide_layout_ids(): + all_valid = False + + if not self.validate_content_types(): + all_valid = False + + if not self.validate_against_xsd(): + all_valid = False + + if not self.validate_notes_slide_references(): + all_valid = False + + if not self.validate_all_relationship_ids(): + all_valid = False + + if not self.validate_no_duplicate_slide_layouts(): + all_valid = False + + if not self.validate_master_theme_uniqueness(): + all_valid = False + + if not self.validate_charts(): + all_valid = False + + if not self.validate_slides(): + all_valid = False + + return all_valid + + def _package_map(self) -> dict: + wanted = [] + wanted += list(self.unpacked_dir.glob("[[]Content_Types[]].xml")) + wanted += list(self.unpacked_dir.glob("ppt/presentation.xml")) + wanted += list(self.unpacked_dir.glob("ppt/theme/*.xml")) + wanted += list(self.unpacked_dir.glob("ppt/theme/_rels/*.rels")) + wanted += list(self.unpacked_dir.glob("ppt/charts/chart*.xml")) + for group in ("slideMasters", "notesMasters", "handoutMasters"): + wanted += list(self.unpacked_dir.glob(f"ppt/{group}/*.xml")) + wanted += list(self.unpacked_dir.glob(f"ppt/{group}/_rels/*.rels")) + return { + p.relative_to(self.unpacked_dir).as_posix(): p.read_bytes() + for p in wanted + if p.is_file() + } + + def validate_master_theme_uniqueness(self): + from helpers.pptx_theme import _NOTES_MASTERS, live_shared_master_themes + + shared = live_shared_master_themes(self._package_map()) + if shared: + print(f"FAILED - Found {len(shared)} master(s) sharing a theme part:") + for message in shared: + print(f" {message}") + if any(m.startswith(_NOTES_MASTERS) for m in shared): + print(" Fix: in ppt/presentation.xml, move back to " + "directly after . PowerPoint reads that happily.") + else: + print(" Fix: give each master its own theme part.") + return False + + if self.verbose: + print("PASSED - No master shares a theme part in a way PowerPoint refuses") + return True + + def validate_charts(self): + from helpers.pptx_chart import find_chart_problems + + problems = find_chart_problems(self._package_map()) + if problems: + print(f"FAILED - Found {len(problems)} chart problem(s) PowerPoint rejects:") + for message in problems: + print(f" {message}") + return False + + if self.verbose: + print("PASSED - Charts satisfy the constraints PowerPoint enforces") + return True + + def _original_slide_defects(self, schema) -> set[str]: + import tempfile + import zipfile + + from helpers.pptx_slide import SLIDE_PART_RE, fatal_slide_errors + + if self.original_file is None: + return set() + + found: set[str] = set() + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + try: + with zipfile.ZipFile(self.original_file, "r") as zf: + safe_extract(zf, temp_path) + except (zipfile.BadZipFile, ValueError, OSError): + return set() + + for part in sorted(temp_path.rglob("*.xml")): + relative = part.relative_to(temp_path).as_posix() + if not SLIDE_PART_RE.fullmatch(relative): + continue + ok, errors = self._validate_single_file_xsd( + part.resolve(), temp_path.resolve(), schema_path=schema + ) + if ok is None or ok or not errors: + continue + found |= set(fatal_slide_errors(set(errors))) + return found + + def validate_slides(self): + from helpers.pptx_slide import ( + SLIDE_PART_RE, + fatal_slide_errors, + is_schema_verdict, + ) + + schema = self.schemas_dir / self.SCHEMA_MAPPINGS["ppt"] + inherited = self._original_slide_defects(schema) + problems: list[str] = [] + broken: list[str] = [] + + for xml_file in self.xml_files: + relative = xml_file.relative_to(self.unpacked_dir).as_posix() + if not SLIDE_PART_RE.fullmatch(relative): + continue + ok, errors = self._validate_single_file_xsd( + xml_file.resolve(), self.unpacked_dir.resolve(), schema_path=schema + ) + if ok is None or not errors: + continue + + unreadable = [f"{relative}: {e}" for e in errors if not is_schema_verdict(e)] + if unreadable: + broken.extend(unreadable) + continue + if ok: + continue + + for message in fatal_slide_errors(set(errors)): + if message in inherited: + continue + problems.append(f"{relative}: {message}") + + if broken: + print(f"FAILED - Could not check {len(broken)} slide part(s):") + for message in sorted(broken): + print(f" {message[:240]}") + + if problems: + print(f"FAILED - Found {len(problems)} slide problem(s) PowerPoint rejects:") + for message in sorted(problems): + print(f" {message[:240]}") + + if broken or problems: + return False + + if self.verbose: + print("PASSED - Slide XML has none of the defects PowerPoint refuses") + return True + + def _get_schema_path(self, xml_file): + if xml_file.parent.name == "charts" and xml_file.name.startswith("chart"): + return None + return super()._get_schema_path(xml_file) + + def _preprocess_for_schema(self, xml_doc, relative_path): + if relative_path.as_posix() != "ppt/presentation.xml": + return xml_doc + + root = xml_doc.getroot() + ns = f"{{{self.PRESENTATIONML_NAMESPACE}}}" + notes = root.find(f"{ns}notesMasterIdLst") + slides = root.find(f"{ns}sldIdLst") + if notes is None or slides is None: + return xml_doc + + children = list(root) + if children.index(notes) < children.index(slides): + return xml_doc + + root.remove(notes) + root.insert(list(root).index(slides), notes) + return xml_doc + + def validate_uuid_ids(self): + import lxml.etree + + errors = [] + uuid_pattern = re.compile( + r"^[\{\(]?[0-9A-Fa-f]{8}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{4}-?[0-9A-Fa-f]{12}[\}\)]?$" + ) + + for xml_file in self.xml_files: + try: + root = lxml.etree.parse(str(xml_file)).getroot() + + for elem in root.iter(): + for attr, value in elem.attrib.items(): + attr_name = attr.split("}")[-1].lower() + if attr_name == "id" or attr_name.endswith("id"): + if self._looks_like_uuid(value): + if not uuid_pattern.match(value): + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: " + f"Line {elem.sourceline}: ID '{value}' appears to be a UUID but contains invalid hex characters" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {xml_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} UUID ID validation errors:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All UUID-like IDs contain valid hex values") + return True + + def _looks_like_uuid(self, value): + clean_value = value.strip("{}()").replace("-", "") + return len(clean_value) == 32 and all(c.isalnum() for c in clean_value) + + def validate_slide_layout_ids(self): + import lxml.etree + + errors = [] + + slide_masters = list(self.unpacked_dir.glob("ppt/slideMasters/*.xml")) + + if not slide_masters: + if self.verbose: + print("PASSED - No slide masters found") + return True + + for slide_master in slide_masters: + try: + root = lxml.etree.parse(str(slide_master)).getroot() + + rels_file = slide_master.parent / "_rels" / f"{slide_master.name}.rels" + + if not rels_file.exists(): + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: " + f"Missing relationships file: {rels_file.relative_to(self.unpacked_dir)}" + ) + continue + + rels_root = lxml.etree.parse(str(rels_file)).getroot() + + valid_layout_rids = set() + for rel in rels_root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rel_type = rel.get("Type", "") + if "slideLayout" in rel_type: + valid_layout_rids.add(rel.get("Id")) + + for sld_layout_id in root.findall( + f".//{{{self.PRESENTATIONML_NAMESPACE}}}sldLayoutId" + ): + r_id = sld_layout_id.get( + f"{{{self.OFFICE_RELATIONSHIPS_NAMESPACE}}}id" + ) + layout_id = sld_layout_id.get("id") + + if r_id and r_id not in valid_layout_rids: + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: " + f"Line {sld_layout_id.sourceline}: sldLayoutId with id='{layout_id}' " + f"references r:id='{r_id}' which is not found in slide layout relationships" + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {slide_master.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print(f"FAILED - Found {len(errors)} slide layout ID validation errors:") + for error in errors: + print(error) + print( + "Remove invalid references or add missing slide layouts to the relationships file." + ) + return False + else: + if self.verbose: + print("PASSED - All slide layout IDs reference valid slide layouts") + return True + + def validate_no_duplicate_slide_layouts(self): + import lxml.etree + + errors = [] + slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) + + for rels_file in slide_rels_files: + try: + root = lxml.etree.parse(str(rels_file)).getroot() + + layout_rels = [ + rel + for rel in root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ) + if "slideLayout" in rel.get("Type", "") + ] + + if len(layout_rels) > 1: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: has {len(layout_rels)} slideLayout references" + ) + + except Exception as e: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + if errors: + print("FAILED - Found slides with duplicate slideLayout references:") + for error in errors: + print(error) + return False + else: + if self.verbose: + print("PASSED - All slides have exactly one slideLayout reference") + return True + + def validate_notes_slide_references(self): + import lxml.etree + + errors = [] + notes_slide_references = {} + + slide_rels_files = list(self.unpacked_dir.glob("ppt/slides/_rels/*.xml.rels")) + + if not slide_rels_files: + if self.verbose: + print("PASSED - No slide relationship files found") + return True + + for rels_file in slide_rels_files: + try: + root = lxml.etree.parse(str(rels_file)).getroot() + + for rel in root.findall( + f".//{{{self.PACKAGE_RELATIONSHIPS_NAMESPACE}}}Relationship" + ): + rel_type = rel.get("Type", "") + if "notesSlide" in rel_type: + part = opc_target( + rel.get("Target", ""), + rels_source_part(rels_file, self.unpacked_dir), + rel.get("TargetMode", ""), + ) + if part: + slide_name = rels_file.stem.replace( + ".xml", "" + ) + + notes_slide_references.setdefault(part, []).append( + (slide_name, rels_file) + ) + + except (lxml.etree.XMLSyntaxError, Exception) as e: + errors.append( + f" {rels_file.relative_to(self.unpacked_dir)}: Error: {e}" + ) + + for target, references in notes_slide_references.items(): + if len(references) > 1: + slide_names = [ref[0] for ref in references] + errors.append( + f" Notes slide '{target}' is referenced by multiple slides: {', '.join(slide_names)}" + ) + for slide_name, rels_file in references: + errors.append(f" - {rels_file.relative_to(self.unpacked_dir)}") + + if errors: + print( + f"FAILED - Found {len([e for e in errors if not e.startswith(' ')])} notes slide reference validation errors:" + ) + for error in errors: + print(error) + print("Each slide may optionally have its own slide file.") + return False + else: + if self.verbose: + print("PASSED - All notes slide references are unique") + return True + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/powerpoint/scripts/office/validators/redlining.py b/powerpoint/scripts/office/validators/redlining.py new file mode 100644 index 0000000..7d3a082 --- /dev/null +++ b/powerpoint/scripts/office/validators/redlining.py @@ -0,0 +1,299 @@ +""" +Validator for tracked changes in Word documents. + +Detects untracked edits in word/document.xml: text that differs from the +original without a / wrapper recording it. The tracked changes +that are new relative to the original are undone, and the result is compared +against the original; whatever text still differs was edited without being +tracked. + +Only the document body is compared. Headers, footers, footnotes and endnotes +are separate parts and are not checked. +""" + +import subprocess +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.ElementTree as ET +from defusedxml.common import DefusedXmlException + +from helpers import rendered_text, safe_extract + + +class RedliningValidator: + + def __init__(self, unpacked_dir, original_docx, verbose=False): + self.unpacked_dir = Path(unpacked_dir) + self.original_docx = Path(original_docx) + self.verbose = verbose + self.namespaces = { + "w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main" + } + + def repair(self) -> int: + return 0 + + def validate(self): + modified_file = self.unpacked_dir / "word" / "document.xml" + if not modified_file.exists(): + print(f"FAILED - Modified document.xml not found at {modified_file}") + return False + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + try: + with zipfile.ZipFile(self.original_docx, "r") as zip_ref: + safe_extract(zip_ref, temp_path) + except Exception as e: + print(f"FAILED - Error unpacking original docx: {e}") + return False + + original_file = temp_path / "word" / "document.xml" + if not original_file.exists(): + print( + f"FAILED - Original document.xml not found in {self.original_docx}" + ) + return False + + try: + modified_tree = ET.parse(modified_file) + modified_root = modified_tree.getroot() + original_tree = ET.parse(original_file) + original_root = original_tree.getroot() + except (ET.ParseError, DefusedXmlException) as e: + print(f"FAILED - Error parsing XML files: {e}") + return False + + new_changes = self._new_tracked_changes(original_root, modified_root) + self._remove_tracked_changes(modified_root, new_changes) + + modified_text = self._extract_text_content(modified_root) + original_text = self._extract_text_content(original_root) + + if modified_text != original_text: + error_message = self._generate_detailed_diff( + original_text, modified_text + ) + print(error_message) + return False + + if self.verbose: + print( + f"PASSED - All {len(new_changes)} change(s) against the original " + "are properly tracked" + ) + return True + + def _tracked_change_elements(self, root): + ins_tag = f"{{{self.namespaces['w']}}}ins" + del_tag = f"{{{self.namespaces['w']}}}del" + return [elem for elem in root.iter() if elem.tag in (ins_tag, del_tag)] + + def _rendered_text(self, elem): + preserve = elem.get("{http://www.w3.org/XML/1998/namespace}space") == "preserve" + return rendered_text(elem.text or "", preserve) + + def _text_elements(self, elem): + w = self.namespaces["w"] + return [ + node + for node in elem.iter() + if node.tag in (f"{{{w}}}t", f"{{{w}}}delText") + ] + + def _tracked_change_key(self, elem): + w = self.namespaces["w"] + text = "".join(self._rendered_text(node) for node in self._text_elements(elem)) + return (elem.tag, elem.get(f"{{{w}}}author"), elem.get(f"{{{w}}}date"), text) + + def _new_tracked_changes(self, original_root, modified_root): + original = self._tracked_change_elements(original_root) + modified = self._tracked_change_elements(modified_root) + + pool = {} + for elem in original: + pool.setdefault(self._tracked_change_key(elem), []).append(elem) + + matched, leftover = set(), [] + for elem in modified: + bucket = pool.get(self._tracked_change_key(elem)) + if bucket: + matched.add(bucket.pop()) + else: + leftover.append(elem) + + def group(elem): + return self._tracked_change_key(elem)[:3] + + def text_of(elems): + return "".join(self._tracked_change_key(e)[3] for e in elems) + + unmatched_original = {} + for elem in original: + if elem not in matched: + unmatched_original.setdefault(group(elem), []).append(elem) + + by_group = {} + for elem in leftover: + by_group.setdefault(group(elem), []).append(elem) + + new = set() + for key, elems in by_group.items(): + rebuilt = text_of(elems) + if rebuilt and rebuilt == text_of(unmatched_original.get(key, [])): + continue + new.update(elems) + return new + + def _generate_detailed_diff(self, original_text, modified_text): + error_parts = [ + "FAILED - Document text doesn't match after removing the tracked changes", + "", + "Likely causes:", + " 1. Modified text inside another author's or tags", + " 2. Made edits without proper tracked changes", + " 3. Didn't nest inside when deleting another's insertion", + " 4. Rewrote another author's / and changed its text on", + " the way. A tracked change from the original is recognised by its", + " author, date and text; anything that doesn't reproduce one exactly", + " reads as new, and the text it carried is reported missing.", + "", + "For pre-redlined documents, use correct patterns:", + " - To reject another's INSERTION: Nest inside their ", + " - To reject PART of one: nest around only the runs you reject.", + " Their may be split around it, so long as the pieces keep", + " their author and date and still spell out the same text.", + " - To restore another's DELETION: Add new AFTER their ", + "", + ] + + git_diff = self._get_git_word_diff(original_text, modified_text) + if git_diff: + error_parts.extend(["Differences:", "============", git_diff]) + else: + error_parts.append("Unable to generate word diff (git not available)") + + return "\n".join(error_parts) + + def _get_git_word_diff(self, original_text, modified_text): + try: + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + + original_file = temp_path / "original.txt" + modified_file = temp_path / "modified.txt" + + original_file.write_text(original_text, encoding="utf-8") + modified_file.write_text(modified_text, encoding="utf-8") + + result = subprocess.run( + [ + "git", + "diff", + "--word-diff=plain", + "--word-diff-regex=.", + "-U0", + "--no-index", + str(original_file), + str(modified_file), + ], + capture_output=True, + text=True, encoding="utf-8", errors="replace", + ) + + if result.stdout.strip(): + lines = result.stdout.split("\n") + content_lines = [] + in_content = False + for line in lines: + if line.startswith("@@"): + in_content = True + continue + if in_content and line.strip(): + content_lines.append(line) + + if content_lines: + return "\n".join(content_lines) + + result = subprocess.run( + [ + "git", + "diff", + "--word-diff=plain", + "-U0", + "--no-index", + str(original_file), + str(modified_file), + ], + capture_output=True, + text=True, encoding="utf-8", errors="replace", + ) + + if result.stdout.strip(): + lines = result.stdout.split("\n") + content_lines = [] + in_content = False + for line in lines: + if line.startswith("@@"): + in_content = True + continue + if in_content and line.strip(): + content_lines.append(line) + return "\n".join(content_lines) + + except (subprocess.CalledProcessError, FileNotFoundError, Exception): + pass + + return None + + def _remove_tracked_changes(self, root, targets): + ins_tag = f"{{{self.namespaces['w']}}}ins" + del_tag = f"{{{self.namespaces['w']}}}del" + + for parent in root.iter(): + to_remove = [] + for child in parent: + if child.tag == ins_tag and child in targets: + to_remove.append(child) + for elem in to_remove: + parent.remove(elem) + + deltext_tag = f"{{{self.namespaces['w']}}}delText" + t_tag = f"{{{self.namespaces['w']}}}t" + + for parent in root.iter(): + to_process = [] + for child in parent: + if child.tag == del_tag and child in targets: + to_process.append((child, list(parent).index(child))) + + for del_elem, del_index in reversed(to_process): + for elem in del_elem.iter(): + if elem.tag == deltext_tag: + elem.tag = t_tag + + for child in reversed(list(del_elem)): + parent.insert(del_index, child) + parent.remove(del_elem) + + def _extract_text_content(self, root): + p_tag = f"{{{self.namespaces['w']}}}p" + t_tag = f"{{{self.namespaces['w']}}}t" + + paragraphs = [] + for p_elem in root.findall(f".//{p_tag}"): + text_parts = [] + for t_elem in p_elem.findall(f".//{t_tag}"): + text_parts.append(self._rendered_text(t_elem)) + paragraph_text = "".join(text_parts) + if paragraph_text: + paragraphs.append(paragraph_text) + + return "\n".join(paragraphs) + + +if __name__ == "__main__": + raise RuntimeError("This module should not be run directly.") diff --git a/powerpoint/scripts/thumbnail.py b/powerpoint/scripts/thumbnail.py new file mode 100755 index 0000000..8735bc6 --- /dev/null +++ b/powerpoint/scripts/thumbnail.py @@ -0,0 +1,313 @@ +"""Create thumbnail grids from PowerPoint presentation slides. + +Creates a grid layout of slide thumbnails for quick visual analysis. +Labels each thumbnail with its XML filename (e.g., slide1.xml). +Hidden slides are shown with a placeholder pattern. + +Usage: + python thumbnail.py input.pptx [output_prefix] [--cols N] + +Examples: + python thumbnail.py presentation.pptx + # Creates: thumbnails.jpg + + python thumbnail.py template.pptx grid --cols 4 + # Creates: grid.jpg (or grid-1.jpg, grid-2.jpg for large decks) +""" + +import argparse +import posixpath +import subprocess +import sys +import tempfile +import zipfile +from pathlib import Path + +import defusedxml.minidom +from defusedxml import ElementTree +from office.helpers import SLIDE_REL_TYPE, opc_target +from office.soffice import run_soffice +from PIL import Image, ImageDraw, ImageFont + + +THUMBNAIL_WIDTH = 300 +CONVERSION_DPI = 100 +MAX_COLS = 6 +DEFAULT_COLS = 3 +JPEG_QUALITY = 95 +GRID_PADDING = 20 +BORDER_WIDTH = 2 +FONT_SIZE_RATIO = 0.10 +LABEL_PADDING_RATIO = 0.4 + + +def main(): + parser = argparse.ArgumentParser( + description="Create thumbnail grids from PowerPoint slides." + ) + parser.add_argument("input", help="Input PowerPoint file (.pptx)") + parser.add_argument( + "output_prefix", + nargs="?", + default="thumbnails", + help="Output prefix for image files (default: thumbnails)", + ) + parser.add_argument( + "--cols", + type=int, + default=DEFAULT_COLS, + help=f"Number of columns (default: {DEFAULT_COLS}, max: {MAX_COLS})", + ) + + args = parser.parse_args() + + cols = min(args.cols, MAX_COLS) + if args.cols > MAX_COLS: + print(f"Warning: Columns limited to {MAX_COLS}") + + input_path = Path(args.input) + if not input_path.exists() or input_path.suffix.lower() != ".pptx": + print(f"Error: Invalid PowerPoint file: {args.input}", file=sys.stderr) + sys.exit(1) + + output_path = Path(f"{args.output_prefix}.jpg") + + try: + slide_info = get_slide_info(input_path) + + with tempfile.TemporaryDirectory() as temp_dir: + temp_path = Path(temp_dir) + visible_images = convert_to_images(input_path, temp_path) + + if not visible_images and not any(s["hidden"] for s in slide_info): + print("Error: No slides found", file=sys.stderr) + sys.exit(1) + + slides = build_slide_list(slide_info, visible_images, temp_path) + + grid_files = create_grids(slides, cols, THUMBNAIL_WIDTH, output_path) + + print(f"Created {len(grid_files)} grid(s):") + for grid_file in grid_files: + print(f" {grid_file}") + + except Exception as e: + print(f"Error: {e}", file=sys.stderr) + sys.exit(1) + + +def _is_hidden(zf: zipfile.ZipFile, part: str) -> bool: + try: + with zf.open(part) as f: + for _, root in ElementTree.iterparse(f, events=("start",)): + return root.get("show") in ("0", "false") + except (KeyError, ElementTree.ParseError): + return False + return False + + +def get_slide_info(pptx_path: Path) -> list[dict]: + with zipfile.ZipFile(pptx_path, "r") as zf: + rels_content = zf.read("ppt/_rels/presentation.xml.rels").decode("utf-8") + rels_dom = defusedxml.minidom.parseString(rels_content) + + rid_to_part = {} + for rel in rels_dom.getElementsByTagName("Relationship"): + if rel.getAttribute("Type") != SLIDE_REL_TYPE: + continue + part = opc_target( + rel.getAttribute("Target"), + "ppt/presentation.xml", + rel.getAttribute("TargetMode"), + ) + if part is not None: + rid_to_part[rel.getAttribute("Id")] = part + + pres_content = zf.read("ppt/presentation.xml").decode("utf-8") + pres_dom = defusedxml.minidom.parseString(pres_content) + + present = set(zf.namelist()) + + slides = [] + for sld_id in pres_dom.getElementsByTagName("p:sldId"): + part = rid_to_part.get(sld_id.getAttribute("r:id")) + if part is not None and part in present: + slides.append( + {"name": posixpath.basename(part), "hidden": _is_hidden(zf, part)} + ) + + return slides + + +def build_slide_list( + slide_info: list[dict], + visible_images: list[Path], + temp_dir: Path, +) -> list[tuple[Path, str]]: + visible_count = sum(1 for info in slide_info if not info["hidden"]) + rendered_hidden = len(visible_images) == len(slide_info) != visible_count + + if not rendered_hidden and visible_count != len(visible_images): + raise ValueError( + f"LibreOffice rendered {len(visible_images)} page(s) for {visible_count} " + f"visible slide(s) of {len(slide_info)}; thumbnails would be mislabeled" + ) + + if visible_images: + with Image.open(visible_images[0]) as img: + placeholder_size = img.size + else: + placeholder_size = (1920, 1080) + + slides = [] + visible_idx = 0 + + for info in slide_info: + if info["hidden"] and not rendered_hidden: + placeholder_path = temp_dir / f"hidden-{info['name']}.jpg" + placeholder_img = create_hidden_placeholder(placeholder_size) + placeholder_img.save(placeholder_path, "JPEG") + slides.append((placeholder_path, f"{info['name']} (hidden)")) + else: + label = f"{info['name']} (hidden)" if info["hidden"] else info["name"] + slides.append((visible_images[visible_idx], label)) + visible_idx += 1 + + return slides + + +def create_hidden_placeholder(size: tuple[int, int]) -> Image.Image: + img = Image.new("RGB", size, color="#F0F0F0") + draw = ImageDraw.Draw(img) + line_width = max(5, min(size) // 100) + draw.line([(0, 0), size], fill="#CCCCCC", width=line_width) + draw.line([(size[0], 0), (0, size[1])], fill="#CCCCCC", width=line_width) + return img + + +def convert_to_images(pptx_path: Path, temp_dir: Path) -> list[Path]: + pdf_path = temp_dir / f"{pptx_path.stem}.pdf" + + result = run_soffice( + ["--headless", "--convert-to", "pdf", "--outdir", str(temp_dir), str(pptx_path)], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + ) + if result.returncode != 0 or not pdf_path.exists(): + detail = (result.stderr or result.stdout or "").strip() + raise RuntimeError(f"PDF conversion failed: {detail}" if detail else "PDF conversion failed") + + result = subprocess.run( + [ + "pdftoppm", + "-jpeg", + "-r", + str(CONVERSION_DPI), + str(pdf_path), + str(temp_dir / "slide"), + ], + capture_output=True, + text=True, encoding="utf-8", errors="replace", + ) + if result.returncode != 0: + raise RuntimeError("Image conversion failed") + + return sorted(temp_dir.glob("slide-*.jpg")) + + +def create_grids( + slides: list[tuple[Path, str]], + cols: int, + width: int, + output_path: Path, +) -> list[str]: + max_per_grid = cols * (cols + 1) + grid_files = [] + + for chunk_idx, start_idx in enumerate(range(0, len(slides), max_per_grid)): + end_idx = min(start_idx + max_per_grid, len(slides)) + chunk_slides = slides[start_idx:end_idx] + + grid = create_grid(chunk_slides, cols, width) + + if len(slides) <= max_per_grid: + grid_filename = output_path + else: + stem = output_path.stem + suffix = output_path.suffix + grid_filename = output_path.parent / f"{stem}-{chunk_idx + 1}{suffix}" + + grid_filename.parent.mkdir(parents=True, exist_ok=True) + grid.save(str(grid_filename), quality=JPEG_QUALITY) + grid_files.append(str(grid_filename)) + + return grid_files + + +def create_grid( + slides: list[tuple[Path, str]], + cols: int, + width: int, +) -> Image.Image: + font_size = int(width * FONT_SIZE_RATIO) + label_padding = int(font_size * LABEL_PADDING_RATIO) + + with Image.open(slides[0][0]) as img: + aspect = img.height / img.width + height = int(width * aspect) + + rows = (len(slides) + cols - 1) // cols + grid_w = cols * width + (cols + 1) * GRID_PADDING + grid_h = rows * (height + font_size + label_padding * 2) + (rows + 1) * GRID_PADDING + + grid = Image.new("RGB", (grid_w, grid_h), "white") + draw = ImageDraw.Draw(grid) + + try: + font = ImageFont.load_default(size=font_size) + except Exception: + font = ImageFont.load_default() + + for i, (img_path, slide_name) in enumerate(slides): + row, col = i // cols, i % cols + x = col * width + (col + 1) * GRID_PADDING + y_base = ( + row * (height + font_size + label_padding * 2) + (row + 1) * GRID_PADDING + ) + + label = slide_name + bbox = draw.textbbox((0, 0), label, font=font) + text_w = bbox[2] - bbox[0] + draw.text( + (x + (width - text_w) // 2, y_base + label_padding), + label, + fill="black", + font=font, + ) + + y_thumbnail = y_base + label_padding + font_size + label_padding + + with Image.open(img_path) as img: + img.thumbnail((width, height), Image.Resampling.LANCZOS) + w, h = img.size + tx = x + (width - w) // 2 + ty = y_thumbnail + (height - h) // 2 + grid.paste(img, (tx, ty)) + + if BORDER_WIDTH > 0: + draw.rectangle( + [ + (tx - BORDER_WIDTH, ty - BORDER_WIDTH), + (tx + w + BORDER_WIDTH - 1, ty + h + BORDER_WIDTH - 1), + ], + outline="gray", + width=BORDER_WIDTH, + ) + + return grid + + +if __name__ == "__main__": + main() diff --git a/teams-meeting-pipeline/SKILL.md b/teams-meeting-pipeline/SKILL.md new file mode 100644 index 0000000..775e3a5 --- /dev/null +++ b/teams-meeting-pipeline/SKILL.md @@ -0,0 +1,116 @@ +--- +name: teams-meeting-pipeline +description: Teams meeting summaries, job replay, Graph subscriptions. +version: 1.1.0 +author: Hermes Agent + Teknium +license: MIT +prerequisites: + env_vars: [MSGRAPH_TENANT_ID, MSGRAPH_CLIENT_ID, MSGRAPH_CLIENT_SECRET] + commands: [hermes] +metadata: + hermes: + tags: [Teams, Microsoft Graph, Meetings, Productivity, Operations] + related_docs: + - /docs/guides/microsoft-graph-app-registration + - /docs/user-guide/messaging/teams-meetings + - /docs/guides/operate-teams-meeting-pipeline +--- + +# Teams Meeting Pipeline + +Use this skill whenever the user asks about Microsoft Teams meeting summaries, transcripts, recordings, action items, Graph subscriptions, or any operational question about the Teams meeting pipeline. Works in any language — the triggers below are examples, not an exhaustive list. + +Everything operator-facing is a `hermes teams-pipeline` subcommand run via the terminal tool. There are no new model tools for this pipeline — the CLI is the surface. + +## When to use this skill + +The user is asking to: +- summarize a Teams meeting / extract action items / pull meeting notes +- check pipeline status, inspect a stored meeting job, or see recent meetings +- replay / re-run a stored job that failed or needs a fresh summary +- validate Microsoft Graph setup after changing env or config +- troubleshoot "meeting summary never arrived" or "no new meetings are ingesting" +- manage Graph webhook subscriptions (create, renew, delete, inspect) +- set up automated subscription renewal (see pitfall below) + +Multilingual trigger examples (not exhaustive): +- English: "summarize the Teams meeting", "pipeline status", "replay job X" +- Turkish: "Teams meeting özetle", "action item çıkar", "toplantı notu", "pipeline durumu", "replay job" + +## Prerequisites + +Before using the pipeline, verify these are set in `${HERMES_HOME:-~/.hermes}/.env`: + +```bash +MSGRAPH_TENANT_ID=... +MSGRAPH_CLIENT_ID=... +MSGRAPH_CLIENT_SECRET=... +``` + +If any are missing, direct the user to the Azure app registration guide at `/docs/guides/microsoft-graph-app-registration` — they need an Azure AD app registration with admin-consented Graph application permissions before the pipeline will work. + +## Command reference + +### Status and inspection (start here) + +```bash +hermes teams-pipeline validate # config snapshot — run first after any change +hermes teams-pipeline token-health # Graph token status +hermes teams-pipeline token-health --force-refresh # force a fresh token acquisition +hermes teams-pipeline list # recent meeting jobs +hermes teams-pipeline list --status failed # only failed jobs +hermes teams-pipeline show # full detail of one job +hermes teams-pipeline subscriptions # current Graph webhook subscriptions +``` + +### Re-running / debugging + +```bash +hermes teams-pipeline run # replay a stored job (re-summarize, re-deliver) +hermes teams-pipeline fetch --meeting-id # dry-run: resolve meeting + transcript without persisting +hermes teams-pipeline fetch --join-web-url "" # dry-run by join URL +``` + +### Subscription management + +```bash +hermes teams-pipeline subscribe \ + --resource communications/onlineMeetings/getAllTranscripts \ + --notification-url https:///msgraph/webhook \ + --client-state "$MSGRAPH_WEBHOOK_CLIENT_STATE" + +hermes teams-pipeline renew-subscription --expiration +hermes teams-pipeline delete-subscription +hermes teams-pipeline maintain-subscriptions # renew near-expiry ones +hermes teams-pipeline maintain-subscriptions --dry-run # show what would be renewed +``` + +## Decision tree for common asks + +- User asks "why didn't I get a summary for today's meeting?" → start with `list --status failed`, then `show ` on the relevant row. If the job doesn't exist at all, check `subscriptions` — the webhook may have expired (see pitfall below). +- User asks "is setup working?" → `validate`, then `token-health`, then `subscriptions`. If all three pass, request a test meeting and check `list` for a fresh row. +- User asks "re-run summary for meeting X" → `list` to find the job ID, `run ` to replay. If it fails again, `show ` to inspect the error and `fetch --meeting-id` to dry-run the artifact resolution. +- User asks "add meeting X to the pipeline" → usually you don't — the pipeline is subscription-driven, not per-meeting. If they want a specific past meeting summarized, use `fetch` to pull transcript + `run` after a job is created. + +## Critical pitfall: Graph subscriptions expire in 72 hours + +Microsoft Graph caps webhook subscriptions at 72 hours and **will not auto-renew them**. If `maintain-subscriptions` is not scheduled, meeting notifications silently stop arriving 3 days after any manual subscription creation. + +When the user reports "the pipeline worked yesterday but nothing is arriving today": +1. Run `hermes teams-pipeline subscriptions` — if it's empty or all entries show `expirationDateTime` in the past, that's the cause. +2. Recreate with `subscribe` as shown above. +3. **Set up automated renewal immediately** via `hermes cron add`, a systemd timer, or plain crontab. The operator runbook at `/docs/guides/operate-teams-meeting-pipeline#automating-subscription-renewal-required-for-production` has all three options. 12-hour interval is safe (6x headroom against the 72h limit). + +## Other pitfalls + +- **Transcript not available yet.** Teams takes some time after a meeting ends to generate the transcript artifact. `fetch --meeting-id` on a just-ended meeting may return empty. Wait 2-5 minutes and retry, or let the Graph webhook drive ingestion naturally. +- **Delivery mode mismatch.** If summaries are produced (`list` shows success) but nothing lands in Teams, check `platforms.teams.extra.delivery_mode` and the matching target config (`incoming_webhook_url` OR `chat_id` OR `team_id`+`channel_id`). The writer reads these from config.yaml or `TEAMS_*` env vars. +- **Graph app permissions.** A token acquires cleanly (`token-health` passes) but Graph API calls return 401/403 when permissions were added but admin consent wasn't re-granted. Have the user revisit the app registration in the Azure portal and click "Grant admin consent" again. + +## Related docs + +Point the user to these when they need more depth than this skill covers: +- Azure app registration walkthrough: `/docs/guides/microsoft-graph-app-registration` +- Full pipeline setup: `/docs/user-guide/messaging/teams-meetings` +- Operator runbook (renewal automation, troubleshooting, go-live checklist): `/docs/guides/operate-teams-meeting-pipeline` +- Webhook listener setup: `/docs/user-guide/messaging/msgraph-webhook` diff --git a/xlsx/LICENSE.txt b/xlsx/LICENSE.txt new file mode 100644 index 0000000..c55ab42 --- /dev/null +++ b/xlsx/LICENSE.txt @@ -0,0 +1,30 @@ +© 2025 Anthropic, PBC. All rights reserved. + +LICENSE: Use of these materials (including all code, prompts, assets, files, +and other components of this Skill) is governed by your agreement with +Anthropic regarding use of Anthropic's services. If no separate agreement +exists, use is governed by Anthropic's Consumer Terms of Service or +Commercial Terms of Service, as applicable: +https://www.anthropic.com/legal/consumer-terms +https://www.anthropic.com/legal/commercial-terms +Your applicable agreement is referred to as the "Agreement." "Services" are +as defined in the Agreement. + +ADDITIONAL RESTRICTIONS: Notwithstanding anything in the Agreement to the +contrary, users may not: + +- Extract these materials from the Services or retain copies of these + materials outside the Services +- Reproduce or copy these materials, except for temporary copies created + automatically during authorized use of the Services +- Create derivative works based on these materials +- Distribute, sublicense, or transfer these materials to any third party +- Make, offer to sell, sell, or import any inventions embodied in these + materials +- Reverse engineer, decompile, or disassemble these materials + +The receipt, viewing, or possession of these materials does not convey or +imply any license or right beyond those expressly granted above. + +Anthropic retains all right, title, and interest in these materials, +including all copyrights, patents, and other intellectual property rights. diff --git a/xlsx/SKILL.md b/xlsx/SKILL.md new file mode 100644 index 0000000..7a7c1f3 --- /dev/null +++ b/xlsx/SKILL.md @@ -0,0 +1,105 @@ +--- +name: xlsx +description: "Create, read, edit Excel .xlsx spreadsheets and CSVs." +version: 1.0.0 +author: Anthropic (adapted by Nous Research) +license: Proprietary. LICENSE.txt has complete terms +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [Excel, XLSX, Spreadsheets, Office, Productivity] + category: productivity + related_skills: [docx, pdf, powerpoint] +--- + +# XLSX Skill + +Create, read, and edit Excel workbooks — formulas, formatting, charts, data cleaning, and format conversion. Every formula-bearing output must be recalculated and error-free before delivery. + +## When to Use + +Use this skill any time a spreadsheet file is the primary input or output: opening, reading, editing, or fixing an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file; creating a new spreadsheet from scratch or from other data; converting between tabular formats; cleaning messy tabular data into a proper spreadsheet. Trigger whenever the user references a spreadsheet file by name or path — even casually. Do NOT trigger when the deliverable is a Word document (`docx` skill), HTML report, standalone script, or Google Sheets API integration. For finance-grade modeling conventions (DCF, LBO, three-statement), the optional `excel-author` skill adds stricter standards on top of this one. + +## Prerequisites + +```bash +pip install openpyxl pandas "markitdown[xlsx]" +which soffice || sudo apt install -y libreoffice # formula recalculation (scripts/recalc.py) +``` + +macOS: `brew install libreoffice`. + +## Quick Reference + +| Task | Approach | +|---|---| +| **Create** or **edit** with formulas/formatting | `openpyxl` — see gotchas below | +| **Bulk data** in or out | `pandas` (`read_excel`, `to_excel`) | +| **Quick look** at a sheet | `markitdown file.xlsx` — `## SheetName` per sheet; reads `.xlsm` too. No cell coordinates, so don't plan edits from it. (`read_file` also auto-extracts .xlsx) | +| **Read** a model (formulas *and* values) | two `load_workbook` passes — see gotchas | + +> Script paths below are relative to this skill's directory. + +## Requirements for every output + +- **Professional font** (Arial, Times New Roman) throughout, unless the user says otherwise. +- **Zero formula errors.** Never ship while `recalc.py` reports `errors_found`. If you think an error predates you, prove it: load the *original* with `data_only=True` and look at that cell. An error you introduced looks exactly like one you inherited. +- **Use formulas, never hardcoded results.** Write `sheet['B10'] = '=SUM(B2:B9)'`, not the Python-computed total. The sheet must recalculate when its inputs change. +- **Follow the user's spec literally.** Exact tab names, exact column headers, and the formula they spelled out. A redesign that computes something else fails, however elegant. +- **Document every assumption and hardcoded number** where the reader will see it — a cell comment, or an adjacent cell at a table's end. Cite a real source when one exists; when the number came from the user, say so plainly. +- **A workbook *you create* for someone to fill in** needs a short legend naming which cells to edit, and one example row of realistic values showing the expected format. Never add such a row to a file you were asked to edit. +- **Editing an existing file: match its conventions exactly.** They override every guideline here. Find its designated input cells first — a distinct font color, fill, or shading marks them — write only there, and leave every existing formula untouched. + +## Recalculate (mandatory whenever the file contains formulas) + +openpyxl writes formulas as strings with **no cached values**. Until you recalculate, every formula cell reads back as `None` to anything reading cached values — `pandas`, `load_workbook(data_only=True)`, and most previewers. + +```bash +python scripts/recalc.py output.xlsx [timeout_seconds] # default 30 +``` + +LibreOffice computes every formula, the file is **rewritten in place**, and you get JSON: `status` (`success` | `errors_found`), `total_formulas`, `total_errors`, and an `error_summary` naming up to 100 cells per error type (`locations_truncated` says how many it withheld — trust `total_errors`, not the length of the list). Fix what it names and run it again. **JSON with an `error` key instead of a `status` means nothing was recalculated**, and only that case exits non-zero — `errors_found` exits 0, so never treat a clean exit as a clean workbook. + +**A green recalc proves your formulas *evaluate*, not that they are *right*.** An off-by-one range or a reference to the wrong row yields a clean, error-free file with wrong numbers. Write 2–3 formulas first and check they pull the values you expect, before building out a grid. + +**A workbook that links to another file loses those links** if you re-save it with openpyxl and then recalculate. Such a formula reads `='[1]Returns Analysis'!$B$2` — the `[1]` is an index into the workbook's external-reference list, naming a *separate file on disk*, not a sheet. That file is rarely present, so the cell's cached value is the only thing holding its data. openpyxl strips that value on save; LibreOffice then has to resolve the reference for real, fails, writes `#NAME?`, and deletes every link. `recalc.py` refuses to run in that state — copy those cells' values out of the original before you save over them (`--force` overrides, and accepts the loss). + +## Choosing formulas that survive verification + +LibreOffice implements fewer functions than Excel, and one it cannot evaluate becomes a literal `#NAME?` baked into the file you deliver. + +- **Prefer Excel-2007-era functions** — `SUMIFS`, `INDEX`, `MATCH`, `IFERROR`, `SUMPRODUCT` — which need no prefix. +- **Six post-2007 functions work, but only with an `_xlfn.` prefix**, because openpyxl writes your formula into the XML verbatim and Excel stores post-2007 names prefixed (its UI hides the prefix): `_xlfn.TEXTJOIN`, `_xlfn.CONCAT`, `_xlfn.IFS`, `_xlfn.SWITCH`, `_xlfn.MAXIFS`, `_xlfn.MINIFS`. Written bare, each yields `#NAME?`. +- **Never use `XLOOKUP`, `XMATCH`, `SORT`, `FILTER`, `UNIQUE`, or `SEQUENCE`.** LibreOffice cannot reliably evaluate them; newer builds that do are spilling array functions, and an openpyxl-written file has no spill metadata, so only the top-left cell of the range gets a value — and `recalc.py` reports `total_errors: 0` on the truncated result. Use `INDEX`/`MATCH` for lookups, and sort, filter, and de-duplicate in Python before writing the cells. +- A formula LibreOffice could not parse is written back **lowercased** — a quick tell beside a `#NAME?`. + +## openpyxl gotchas + +- **Reading a model takes two loads.** `data_only=True` yields cached values with the formulas gone; the default yields formula strings with no values. One pass cannot give you both. +- **`data_only=True` is destructive if you save.** That workbook has no formulas left, so saving replaces every one with a literal — permanently. +- **`data_only=True` on a file openpyxl just wrote returns `None` everywhere** — run `recalc.py` first. (A formula whose result is `""` also reads back as `None`.) +- **Merged cells: write the top-left anchor only.** Every other cell in the range is a `MergedCell` whose `.value` is read-only. +- **`.xlsm` loses its macros unless you pass `keep_vba=True`** to `load_workbook`. +- **A sheet name containing a space must be quoted** in a cross-sheet reference: `='Assumptions Inputs'!$B$5`. Unquoted, it evaluates to `#VALUE!`. + +## Financial models + +Unless the user says otherwise, or the existing file already does something else. + +**Color:** blue text (`0,0,255`) for hardcoded inputs and scenario levers · black for formulas · green (`0,128,0`) for links to another sheet · red (`255,0,0`) for links to another file · yellow fill (`255,255,0`) for key assumptions and cells the user should fill in. + +**Numbers:** currency `$#,##0`, with the unit named in the header (`Revenue ($mm)`) · zeros render as `-`, including in percentages (`$#,##0;($#,##0);-`) · negatives in parentheses · percentages `0.0%`, **stored as fractions** (`0.15` renders `15.0%`; storing `15` renders `1500.0%`) · valuation multiples `0.0x` · years as text (`"2024"`, never `2,024`). + +**Structure:** every assumption in its own labeled cell, referenced by the formulas that use it (`=B5*(1+$B$6)`, never `=B5*1.05`) · formulas consistent across every projection period, since a lone edited cell mid-row is the commonest silent error · guard denominators that can be zero. + +For full investment-banking conventions (balance checks, sensitivity tables, named ranges), install the optional skill: `hermes skills install official/finance/excel-author`. + +## Verification + +1. `python scripts/recalc.py output.xlsx` → `status: success`, `total_errors: 0`. +2. Spot-check 2–3 computed cells against expected values (`load_workbook(data_only=True)` *after* recalc). +3. `markitdown output.xlsx` — scan for missing sheets, misplaced headers, leftover placeholders. + +## Related skills + +`docx` (Word documents), `pdf` (PDF work), `powerpoint` (decks), optional `excel-author` (finance-grade modeling standards). diff --git a/xlsx/scripts/office/soffice.py b/xlsx/scripts/office/soffice.py new file mode 100644 index 0000000..eec1687 --- /dev/null +++ b/xlsx/scripts/office/soffice.py @@ -0,0 +1,192 @@ +""" +Helper for running LibreOffice (soffice) in environments where AF_UNIX +sockets may be blocked (e.g., sandboxed VMs). Detects the restriction +at runtime and applies an LD_PRELOAD shim if needed. + +Usage: + from office.soffice import run_soffice + + result = run_soffice(["--headless", "--convert-to", "pdf", "input.docx"]) + +Call soffice through run_soffice, not through subprocess with get_soffice_env(): +the env dict carries the shim but names no user profile, and a non-root sandbox +cannot bootstrap the default one -- soffice aborts with "User installation could +not be completed" and converts nothing. get_soffice_env() stays public for the +callers that build their own argv (they must pass -env:UserInstallation too). +""" + +import contextlib +import os +import socket +import subprocess +import tempfile +from collections.abc import Iterable +from pathlib import Path + + +def get_soffice_env() -> dict: + env = os.environ.copy() + env["SAL_USE_VCLPLUGIN"] = "svp" + + if _needs_shim(): + shim = _ensure_shim() + env["LD_PRELOAD"] = str(shim) + + return env + + +def run_soffice(args: Iterable[str], **kwargs) -> subprocess.CompletedProcess: + args = list(args) + with contextlib.ExitStack() as stack: + if not any(str(a).startswith("-env:UserInstallation") for a in args): + profile = stack.enter_context( + tempfile.TemporaryDirectory(prefix="lo_profile_", ignore_cleanup_errors=True) + ) + args = [f"-env:UserInstallation={Path(profile).as_uri()}"] + args + return subprocess.run(["soffice"] + args, env=get_soffice_env(), **kwargs) + + + +_SHIM_SO = Path(tempfile.gettempdir()) / "lo_socket_shim.so" + + +def _needs_shim() -> bool: + try: + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s.close() + return False + except OSError: + return True + + +def _ensure_shim() -> Path: + if _SHIM_SO.exists(): + return _SHIM_SO + + src = Path(tempfile.gettempdir()) / "lo_socket_shim.c" + src.write_text(_SHIM_SOURCE, encoding="utf-8") + subprocess.run( + ["gcc", "-shared", "-fPIC", "-o", str(_SHIM_SO), str(src), "-ldl"], + check=True, + capture_output=True, + ) + src.unlink() + return _SHIM_SO + + + +_SHIM_SOURCE = r""" +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include + +static int (*real_socket)(int, int, int); +static int (*real_socketpair)(int, int, int, int[2]); +static int (*real_listen)(int, int); +static int (*real_accept)(int, struct sockaddr *, socklen_t *); +static int (*real_close)(int); +static int (*real_read)(int, void *, size_t); + +/* Per-FD bookkeeping (FDs >= 1024 are passed through unshimmed). */ +static int is_shimmed[1024]; +static int peer_of[1024]; +static int wake_r[1024]; /* accept() blocks reading this */ +static int wake_w[1024]; /* close() writes to this */ +static int listener_fd = -1; /* FD that received listen() */ + +__attribute__((constructor)) +static void init(void) { + real_socket = dlsym(RTLD_NEXT, "socket"); + real_socketpair = dlsym(RTLD_NEXT, "socketpair"); + real_listen = dlsym(RTLD_NEXT, "listen"); + real_accept = dlsym(RTLD_NEXT, "accept"); + real_close = dlsym(RTLD_NEXT, "close"); + real_read = dlsym(RTLD_NEXT, "read"); + for (int i = 0; i < 1024; i++) { + peer_of[i] = -1; + wake_r[i] = -1; + wake_w[i] = -1; + } +} + +/* ---- socket ---------------------------------------------------------- */ +int socket(int domain, int type, int protocol) { + if (domain == AF_UNIX) { + int fd = real_socket(domain, type, protocol); + if (fd >= 0) return fd; + /* socket(AF_UNIX) blocked – fall back to socketpair(). */ + int sv[2]; + if (real_socketpair(domain, type, protocol, sv) == 0) { + if (sv[0] >= 0 && sv[0] < 1024) { + is_shimmed[sv[0]] = 1; + peer_of[sv[0]] = sv[1]; + int wp[2]; + if (pipe(wp) == 0) { + wake_r[sv[0]] = wp[0]; + wake_w[sv[0]] = wp[1]; + } + } + return sv[0]; + } + errno = EPERM; + return -1; + } + return real_socket(domain, type, protocol); +} + +/* ---- listen ---------------------------------------------------------- */ +int listen(int sockfd, int backlog) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + listener_fd = sockfd; + return 0; + } + return real_listen(sockfd, backlog); +} + +/* ---- accept ---------------------------------------------------------- */ +int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen) { + if (sockfd >= 0 && sockfd < 1024 && is_shimmed[sockfd]) { + /* Block until close() writes to the wake pipe. */ + if (wake_r[sockfd] >= 0) { + char buf; + real_read(wake_r[sockfd], &buf, 1); + } + errno = ECONNABORTED; + return -1; + } + return real_accept(sockfd, addr, addrlen); +} + +/* ---- close ----------------------------------------------------------- */ +int close(int fd) { + if (fd >= 0 && fd < 1024 && is_shimmed[fd]) { + int was_listener = (fd == listener_fd); + is_shimmed[fd] = 0; + + if (wake_w[fd] >= 0) { /* unblock accept() */ + char c = 0; + write(wake_w[fd], &c, 1); + real_close(wake_w[fd]); + wake_w[fd] = -1; + } + if (wake_r[fd] >= 0) { real_close(wake_r[fd]); wake_r[fd] = -1; } + if (peer_of[fd] >= 0) { real_close(peer_of[fd]); peer_of[fd] = -1; } + + if (was_listener) + _exit(0); /* conversion done – exit */ + } + return real_close(fd); +} +""" + + + +if __name__ == "__main__": + import sys + result = run_soffice(sys.argv[1:]) + sys.exit(result.returncode) diff --git a/xlsx/scripts/recalc.py b/xlsx/scripts/recalc.py new file mode 100755 index 0000000..2317ce6 --- /dev/null +++ b/xlsx/scripts/recalc.py @@ -0,0 +1,308 @@ +""" +Excel Formula Recalculation Script +Recalculates all formulas in an Excel file using LibreOffice +""" + +import contextlib +import json +import os +import platform +import re +import shutil +import subprocess +import sys +import tempfile +import time +import zipfile +from pathlib import Path + +from office.soffice import get_soffice_env, run_soffice + +from openpyxl import load_workbook + +MACRO_FILENAME = "Module1.xba" +SOFFICE_MISSING = "soffice not found on PATH; LibreOffice is required to recalculate" + +MAX_LOCATIONS = 100 + +EXTERNAL_REF_RE = re.compile(r"""(? + + + Sub RecalculateAndSave() + ThisComponent.calculateAll() + ThisComponent.store() + ThisComponent.close(True) + End Sub +""" + + +def has_gtimeout(): + try: + subprocess.run( + ["gtimeout", "--version"], capture_output=True, timeout=1, check=False + ) + return True + except (FileNotFoundError, subprocess.TimeoutExpired): + return False + + +def _stamp(path): + st = os.stat(path) + return st.st_mtime_ns, st.st_size + + +def setup_libreoffice_macro(profile_dir: Path, timeout=30): + url = profile_dir.as_uri() + try: + run_soffice( + ["--headless", "--terminate_after_init", f"-env:UserInstallation={url}"], + capture_output=True, + timeout=timeout, + ) + except FileNotFoundError: + return None, SOFFICE_MISSING + except subprocess.TimeoutExpired: + return None, "LibreOffice timed out creating its profile; formulas were NOT recalculated" + + macro_dir = profile_dir / "user" / "basic" / "Standard" + if not macro_dir.exists(): + return None, "LibreOffice did not create a usable profile; formulas were NOT recalculated" + + try: + (macro_dir / MACRO_FILENAME).write_text(RECALCULATE_MACRO, encoding="utf-8") + except OSError as e: + return None, f"Could not install the recalculation macro: {e}" + + return url, None + + +def external_links_at_risk(filename): + try: + with zipfile.ZipFile(filename) as archive: + names = archive.namelist() + except (zipfile.BadZipFile, OSError): + return [] + if not any(n.startswith("xl/externalLinks/") for n in names): + return [] + + with contextlib.ExitStack() as stack: + formulas = load_workbook(filename, data_only=False) + stack.callback(formulas.close) + values = load_workbook(filename, data_only=True) + stack.callback(values.close) + + external_names = [ + name + for name, dn in formulas.defined_names.items() + if isinstance(getattr(dn, "value", None), str) and EXTERNAL_REF_RE.search(dn.value) + ] + name_re = ( + re.compile(r"\b(" + "|".join(re.escape(n) for n in external_names) + r")\b") + if external_names + else None + ) + + at_risk = [] + for sheet in formulas.sheetnames: + ws = formulas[sheet] + if not hasattr(ws, "iter_rows"): + continue + cached = values[sheet] + for row in ws.iter_rows(): + for cell in row: + v = cell.value + if not (isinstance(v, str) and v.startswith("=")): + continue + reaches_out = EXTERNAL_REF_RE.search(v) or (name_re and name_re.search(v)) + if reaches_out and cached[cell.coordinate].value is None: + at_risk.append(f"{sheet}!{cell.coordinate}") + return at_risk + + +def recalc(filename, timeout=30, force=False): + if not Path(filename).exists(): + return {"error": f"File {filename} does not exist"} + + abs_path = str(Path(filename).absolute()) + + if not os.access(abs_path, os.W_OK): + return {"error": f"{filename} is not writable; recalculation rewrites the file in place"} + + try: + get_soffice_env() + except Exception as e: + return {"error": f"Could not prepare the LibreOffice environment: {e}"} + + if not force: + try: + at_risk = external_links_at_risk(filename) + except Exception as e: + return {"error": f"Could not inspect {filename} for external links: {e}"} + if at_risk: + shown = at_risk[:MAX_LOCATIONS] + return { + "error": ( + "Refusing to recalculate: this workbook links to another workbook, and " + f"{len(at_risk)} linked cell(s) have lost their cached value (openpyxl strips " + "these on save). Recalculating would resolve them to #NAME? and delete the " + "external links for good. Copy those cells' values from the original file " + "before saving, or pass --force to accept the loss. Charts and conditional " + "formats can hold external references too, so this list may not be exhaustive." + ), + "external_link_cells": shown, + "external_link_cells_truncated": max(0, len(at_risk) - len(shown)), + } + + with tempfile.TemporaryDirectory( + prefix="recalc-lo-profile-", ignore_cleanup_errors=True + ) as profile_dir: + return _recalc_with_profile(filename, abs_path, timeout, Path(profile_dir)) + + +def _recalc_with_profile(filename, abs_path, timeout, profile_dir: Path): + started = time.monotonic() + profile_url, err = setup_libreoffice_macro(profile_dir, timeout=timeout) + if err: + return {"error": err} + + timeout = max(5, int(timeout - (time.monotonic() - started))) + + before = _stamp(abs_path) + + cmd = [ + "soffice", + "--headless", + "--norestore", + f"-env:UserInstallation={profile_url}", + "vnd.sun.star.script:Standard.Module1.RecalculateAndSave?language=Basic&location=application", + abs_path, + ] + + if platform.system() == "Linux" and shutil.which("timeout"): + cmd = ["timeout", str(timeout)] + cmd + elif platform.system() == "Darwin" and has_gtimeout(): + cmd = ["gtimeout", str(timeout)] + cmd + + timed_out = f"LibreOffice timed out after {timeout}s; formulas were NOT recalculated. Re-run with a longer timeout." + + try: + result = subprocess.run( + cmd, capture_output=True, text=True, encoding="utf-8", errors="replace", env=get_soffice_env(), timeout=timeout + 15 + ) + except subprocess.TimeoutExpired: + return {"error": timed_out} + except FileNotFoundError: + return {"error": SOFFICE_MISSING} + + if result.returncode == 124: + return {"error": timed_out} + + if result.returncode != 0: + detail = (result.stderr or "").strip() or f"soffice exited {result.returncode}" + return {"error": f"LibreOffice failed to recalculate: {detail}"} + + if _stamp(abs_path) == before: + return { + "error": ( + "LibreOffice exited cleanly but never rewrote the file, so nothing was " + "recalculated. Check that no other LibreOffice instance is running, then retry." + ) + } + + try: + wb = load_workbook(filename, data_only=True) + + excel_errors = [ + "#VALUE!", + "#DIV/0!", + "#REF!", + "#NAME?", + "#NULL!", + "#NUM!", + "#N/A", + ] + error_details = {err: [] for err in excel_errors} + total_errors = 0 + + for sheet_name in wb.sheetnames: + ws = wb[sheet_name] + if not hasattr(ws, "iter_rows"): + continue + for row in ws.iter_rows(): + for cell in row: + if cell.value is not None and isinstance(cell.value, str): + for err in excel_errors: + if err in cell.value: + location = f"{sheet_name}!{cell.coordinate}" + error_details[err].append(location) + total_errors += 1 + break + + result = { + "status": "success" if total_errors == 0 else "errors_found", + "total_errors": total_errors, + "error_summary": {}, + } + + for err_type, locations in error_details.items(): + if locations: + entry = {"count": len(locations), "locations": locations[:MAX_LOCATIONS]} + if len(locations) > MAX_LOCATIONS: + entry["locations_truncated"] = len(locations) - MAX_LOCATIONS + result["error_summary"][err_type] = entry + + wb.close() + + wb_formulas = load_workbook(filename, data_only=False) + formula_count = 0 + for sheet_name in wb_formulas.sheetnames: + ws = wb_formulas[sheet_name] + if not hasattr(ws, "iter_rows"): + continue + for row in ws.iter_rows(): + for cell in row: + if ( + cell.value + and isinstance(cell.value, str) + and cell.value.startswith("=") + ): + formula_count += 1 + wb_formulas.close() + + result["total_formulas"] = formula_count + + return result + + except Exception as e: + return {"error": str(e)} + + +def main(): + args = [a for a in sys.argv[1:] if a != "--force"] + force = "--force" in sys.argv[1:] + + if not args: + print("Usage: python recalc.py [timeout_seconds] [--force]") + print("\nRecalculates all formulas in an Excel file using LibreOffice") + print("\nReturns JSON with error details:") + print(" - status: 'success' or 'errors_found'") + print(" - total_errors: Total number of Excel errors found") + print(" - total_formulas: Number of formulas in the file") + print(" - error_summary: Breakdown by error type with locations") + print(" - #VALUE!, #DIV/0!, #REF!, #NAME?, #NULL!, #NUM!, #N/A") + print("\nOn any failure the JSON has an 'error' key and no 'status'.") + print("--force recalculates even when it would destroy external links.") + sys.exit(1) + + filename = args[0] + timeout = int(args[1]) if len(args) > 1 else 30 + + result = recalc(filename, timeout, force=force) + print(json.dumps(result, indent=2)) + sys.exit(1 if "error" in result else 0) + + +if __name__ == "__main__": + main()