From 778cb72c8a0902bd0b8159ebd3bb7eff93f28c83 Mon Sep 17 00:00:00 2001
From: grm <grm@eyesin.space>
Date: Mon, 14 Sep 2026 23:51:55 +0300
Subject: Turn the image library into a file library, with a per-blog upload
 limit

Bloggers want to attach PDFs, archives, audio and other files to posts,
not only images. The Images tab becomes Files: any type is accepted,
listed by kind with search, paging, rename and multi-file upload, and the
editor's paste/drop/"Insert file" takes anything (images are shown,
everything else becomes a link). The default limit goes from 5 to 10 MB
and the superadmin can override it per blog from /admin/.

Files stay in Postgres so one pg_dump is still the whole blog. The bytea
column is STORAGE EXTERNAL and /media streams it in substring() slices,
so serving never holds a whole file in memory whatever limit a blog gets.

Serving any type on the root domain, which carries the session cookie,
needs a policy: uploads are typed by sniffing (the extension may only
refine a generic sniff to an allowlisted type) and only images, PDF,
plain text, audio and video render inline; HTML, SVG, XML, scripts,
archives and binaries always go out as application/octet-stream with
Content-Disposition: attachment.

The body cap moves out of requireAuth into guardPOST, which runs after
withBlog has resolved the blog and so knows its limit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sd8UPWrvyYCLj97JexNw3A
---
 AGENTS.md | 80 +++++++++++++++++++++++++++++++++++++++++++++------------------
 1 file changed, 58 insertions(+), 22 deletions(-)

(limited to 'AGENTS.md')

diff --git a/AGENTS.md b/AGENTS.md
index 09d9b7d..4a04077 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -22,12 +22,14 @@ table and deployment notes.
   needs JS, keep it small and give it a sensible fallback where cheap
   (e.g. the `<noscript>` button in `posts.html`).
 - Server-rendered `html/template`; no SPA, no frontend toolchain.
-- Postgres holds **everything**, including images (`images.data bytea`).
+- Postgres holds **everything**, including uploaded files (`files.data bytea`,
+  `STORAGE EXTERNAL`, served in `substring()` slices so a download never
+  loads the whole blob).
   **Each blog is its own database** (`blog_<sub>`, see Key mechanics) so a
   blog is backed up and restored with plain `pg_dump`/`psql`; the control
   database (`DATABASE_URL`) holds only `users` and the `blogs` registry.
 - Posts are Markdown (goldmark → bluemonday). No WYSIWYG. The editor's
-  "Insert image" is the one scripted convenience (`partials/editor.html`);
+  "Insert file" is the one scripted convenience (`partials/editor.html`);
   it must keep its no-JS fallback (file appended on save).
 - Changes are live immediately — there is no draft/preview system. The UX
   is "save, then refresh your blog tab"; keep the "View blog ↗" links.
@@ -49,14 +51,15 @@ internal/config/      env → Config; RootSubdomain = "www"
 internal/db/          Cluster (control pool + lazy per-blog pools, CREATE/DROP DATABASE),
                       goose providers; migrations/control/*.sql, migrations/blog/*.sql
 internal/store/       Store = control DB (users, blog registry, create/delete blog);
-                      BlogStore = one blog's DB (settings, pages, posts, images, sections, modules, menu)
+                      BlogStore = one blog's DB (settings, pages, posts, files, sections, modules, menu)
 internal/auth/        bcrypt, JWT issue/parse, cookie, HMAC CSRF
 internal/markdown/    Render(md) → sanitized HTML
 internal/i18n/        languages, T/Tf (English keys → catalog), FormatDate/Month, Accept-Language Match;
                       el.go is the Greek catalog, el_months.go the month tables
 internal/slug/        Make/Valid/WithSuffix (Greek transliteration included)
 internal/web/         server.go (host router, middleware, render helpers)
-                      routes.go (all routes), handlers_*.go (sections = announcements, layout = modules + menu)
+                      routes.go (all routes), handlers_*.go (sections = announcements, layout = modules + menu,
+                      files = the upload library), filetype.go (what an upload is, how it may be served)
                       theme.go (colours/fonts/layout switches), layout.go (module kinds, archive grouping)
                       templates/ (embedded; layouts/, partials/, auth/, dashboard/, admin/, blog/)
                       static/ (dashboard.css "paper & ink" look — plain CSS, no variables/flex;
@@ -67,7 +70,7 @@ internal/web/         server.go (host router, middleware, render helpers)
 
 - **One database per blog**: the registry row (`blogs`: id, owner, subdomain,
   `db_name`) lives in the control DB; everything else — a one-row `settings`
-  table (title, tagline, language, theme) plus pages, posts, images, sections, modules,
+  table (title, tagline, language, theme) plus pages, posts, files, sections, modules,
   menu_items — lives in `blog_<sub>` (`db.DBName`: dashes → underscores, so
   subdomains are capped at 58 chars). No table in a blog DB carries a
   blog id; the database is the scope. `store.Store` (control) hands out a
@@ -103,9 +106,16 @@ internal/web/         server.go (host router, middleware, render helpers)
   7 days). Claims carry `uid` + `ver` (= `users.token_version`); the
   session middleware re-loads the user every request and drops the session
   if disabled or version mismatch. Password change / reset / disable bump
-  `token_version`. Every POST behind `requireAuth` must include
+  `token_version`. Every management POST must include
   `<input type="hidden" name="_csrf" value="{{.CSRF}}">`; the check runs in
-  `requireAuth`, which also caps the body at `MAX_UPLOAD_MB + 1 MB`.
+  `guardPOST`, which first caps the body at the limit it is given + 1 MB
+  and parses the form. `requireLogin` only redirects anonymous users;
+  `requireAuth` = login + `guardPOST(0)` (dashboard, password, admin: no
+  uploads); `withBlog` = login → `resolveBlog` → owner-or-superadmin →
+  `guardPOST(blog.UploadLimit(cfg))`, and `withBlogFiles(n, …)` allows n
+  limits for the Files page's multi-upload (`maxUploadFiles` = 10). Errors
+  from `guardPOST` go through `s.fail`, which answers JSON when the request
+  has `Accept: application/json` (the upload scripts).
 - **Templates**: each page file is parsed together with its layout
   (`layouts/dashboard.html` or `layouts/blog.html` for `blog/*`) and all
   `partials/*.html`. Page files define `content` (and optionally `title`).
@@ -126,7 +136,8 @@ internal/web/         server.go (host router, middleware, render helpers)
   Optional colours (`LinkHover`, `NavHover`) are `""` = inherit and come
   from a colour input paired with a `<name>_custom` checkbox, shown/hidden
   with CSS only (`.hoverpick > input:not(:checked) ~ …`). Image fields
-  use the `imagepick` partial's radios: `none` clears, a uuid selects, no
+  use the `imagepick` partial's radios (the library's `kind = image` files,
+  `ListFiles(ctx, "image", "", 0, 0)`): `none` clears, a uuid selects, no
   value keeps (`pickImage`); an upload in `<name>_file` wins.
   `Presets()`/`WithPreset` are the colour schemes (`POST /b/{sub}/design/preset`,
   colours only); `POST /b/{sub}/design/reset` stores `DefaultTheme()` and
@@ -166,20 +177,44 @@ internal/web/         server.go (host router, middleware, render helpers)
   blog layout links instead. `GET /favicon.ico` on every host
   (`handleFavicon`) serves the png or redirects to the blog's image, so
   icon-probing browsers never hit the 404 page; `favicon.ico` is reserved.
-- **Images**: `/media/{uuid}` served on every host with immutable cache
-  headers, from the host's blog database only (the root domain serves the
-  root blog's images). Dashboard previews for another blog on the root host
-  therefore use `GET /b/{sub}/media/{id}` (`withBlog`): `dashboard/images.html`
-  and the `imagepick` partial (`sub` arg). Markdown keeps the relative
-  `/media/…` form because post bodies render on the blog host.
-  Uploads are content-sniffed (png/jpeg/gif/webp/ico). Deleting an
-  image clears theme references to it. `POST /b/{sub}/images/upload`
-  answers JSON (`{id, filename, markdown}` / `{error}`) when the request
-  has `Accept: application/json`; that is what the editor script calls.
+- **Files** (`files` table, `store/files.go`, `handlers_files.go`,
+  `filetype.go`, `/b/{sub}/files…`): the upload library, any type. A row is
+  `id, filename, content_type, kind, size, data`; `kind` (image, document,
+  audio, video, archive, other) is decided at upload by `fileType`, which
+  trusts `http.DetectContentType` first and lets the extension refine only a
+  generic sniff (text/plain, octet-stream) to a type on the allowlists in
+  `filetype.go`; anything unknown is stored as `application/octet-stream`.
+  `/media/{uuid}` is served on every host with immutable cache headers, from
+  the host's blog database only (the root domain serves the root blog's
+  files); dashboard previews for another blog on the root host use
+  `GET /b/{sub}/media/{id}` (`withBlog`). **The root domain carries the
+  session cookie, so `servedAs` renders inline only images, PDF, plain text,
+  audio and video; everything else (HTML, SVG, XML, JS, archives, binaries)
+  goes out as `application/octet-stream` + `Content-Disposition: attachment`.**
+  `?download` forces attachment. SVG is therefore never an image (download
+  only, and refused by the design page's `readUpload(…, imagesOnly)`); ICO
+  is. The bytes are streamed by `BlogStore.FileReader` (a `chunkReader` over
+  `substring()`, 512 KiB per query, Range requests included), which is why
+  the per-blog limit is capped at 1024 MB (`maxUploadMB`: int4 offsets).
+  Ids are immutable, so a renamed file keeps its old download name in
+  browsers that cached it. Markdown keeps the relative `/media/…` form
+  because post bodies render on the blog host; `fileMarkdown` writes
+  `![name](…)` for images and `[name](…)` for the rest. Deleting a file
+  clears theme references to it. `POST /b/{sub}/files/upload` takes several
+  `file` parts (the no-JS `<input multiple>`), or answers JSON
+  (`{id, filename, kind, size, markdown}` / `{error}`) for one file when the
+  request has `Accept: application/json` — what the editor and the Files
+  page scripts call. `dashboard/files.html` lists by `?kind=&q=&p=`
+  (`ListFiles`, 50 per page, `pageBounds` clamps), shows `FileUsage` and the
+  blog's limit; rename/delete return to the `back` field. The per-blog limit
+  is `blogs.max_upload_bytes` in the control DB (NULL = `MAX_UPLOAD_MB`, now
+  10), set at `POST /admin/blogs/{id}/upload-limit` from `admin/index.html`;
+  `Blog.UploadLimit(cfg)` resolves it.
 - **Editor** (`partials/editor.html`, args via `dict`: name, value, rows,
-  tall, upload, csrf): textarea + "Insert image" + cheat-sheet. Forms using
-  it must be `multipart/form-data` and their save handler must call
-  `s.readUpload(r, "inline_image")` + `appendImageMD` (the no-JS path).
+  tall, upload, csrf): textarea + "Insert file" + cheat-sheet; paste and
+  drop take any file. Forms using it must be `multipart/form-data` and
+  their save handler must call `s.readUpload(r, "inline_file", false)` +
+  `appendFileMD` (the no-JS path).
 - **Announcements** (`sections` table, `store/sections.go`,
   `handlers_sections.go`, `/b/{sub}/announcements…`): per-blog notices with
   `placement` (`<column>-<position>`: left|main|right × top|bottom, split by
@@ -253,7 +288,8 @@ superadmin password to `admin`. Production refuses both.
   (runs in every blog database — it must not reference `users`/`blogs`) and
   `internal/db/migrations/control/` for users and the registry; goose
   `-- +goose Up/Down` sections; they run automatically at startup. Never
-  edit an applied migration. Blog chain so far: `00001_init`, `00002_language`. Both chains were re-baselined at 00001 after
+  edit an applied migration. Blog chain so far: `00001_init`, `00002_language`,
+  `00003_files`; control: `00001_init`, `00002_upload_limit`. Both chains were re-baselined at 00001 after
   the move to per-blog databases; deployments from before it have
   `goose_db_version` rows 2–7 in the control DB that must be deleted once
   (README "Upgrading from a single database") or the next control migration
-- 
cgit v1.2.3