# Deploying Plantoo_tech Handover instructions. Written 2026-09-03 for T17; the environment-specific values are marked `«fill in»` rather than guessed, because getting them wrong is worse than leaving them blank. ## What exists today | Environment | State | |---|---| | CI (GitHub Actions, `.github/workflows/ci.yml`) | **Real.** Runs the PHPUnit suite against MySQL 8 + MongoDB 7 service containers on every push and PR to `main`, plus an `env-diff` job that fails the build on an undocumented variable | | Testing / Staging | **Real**, driven by an external deploy tool (`plantoo_tech_testing`, `plantoo_tech_staging`), not by this repo | | `.github/workflows/staging-deploy.yml` | **A deliberate stub.** Gated behind `vars.STAGING_DEPLOY_ENABLED == 'true'`; its deploy and smoke steps echo and exit. Wire it or delete it — do not leave it looking like a pipeline | | Production | **Does not exist** (confirmed 2026-08-19, still true 2026-09-03). Building it is T18 | ## Requirements on the host - PHP **8.3** with `mbstring`, `intl`, `pdo_mysql`, **`mongodb`**, and — for the concurrency suite — `pcntl` + `posix`. Without those last two, `BookingConcurrencyTest` **skips silently** and a build goes green having never run the race it exists for. - MySQL **8** and MongoDB **7**. Both, always: `phpunit.xml` sets `DB_CONNECTION=sqlite`, but that only changes Laravel's *default* connection — every model pins `mysql` or `mongodb` explicitly. - Node for the asset build (`npm ci && npm run build`) — and the SAME `node_modules` is the Node worker's runtime, so it must survive the build rather than being pruned. Whoever runs the install must own the directory: on 21/09/2026 the testing box had a `node_modules` written by Jenkins and un-writable by the deploy user, and the worker crash-looped for hours on a missing `mysql2`. - **`node` must be executable by the SERVICE ACCOUNT, not just by you.** Under `on_demand` the queue worker (`www-data`) is what spawns it, and on 21/09/2026 `/usr/bin/node` was a symlink into `/home/arfu` that `www-data` could not traverse: `Permission denied`, from a path that worked perfectly for the deploy user. `sudo -u www-data node -v` is the check. `chmod o+x /home/arfu` fixes it; installing node system-wide is the version that does not depend on a person's home directory continuing to exist. `NODE_BINARY` exists for hosts where a different path is wanted — it does not help when the problem is traversal, since it is the same file behind the same directory. - **CJK fonts for the renderer.** A written document goes snapshot → HTML → headless Chromium, and a host with no Chinese font renders every Chinese glyph as a hollow box — silently, in a finished PDF nobody re-reads. Found on 21/09/2026 in the bilingual hire-note fixture, which exists to catch exactly this: `sudo apt-get install -y fonts-noto-cjk`, then restart the worker. - Three DNS names, one per panel: `ADMIN_DOMAIN`, `ORG_DOMAIN`, `USER_DOMAIN`. The panels are separated by **host**, not by path prefix, so one hostname is not enough. ## Environment file Copy `.env.example` and read the annotations — two blocks are easy to get wrong: - **`DB_*` is MySQL. `DB_*_ALT` is MongoDB** (`config/database.php`). The example file used to declare `DB_PORT_ALT=3306`, which pointed Mongo at the MySQL port; a fresh clone following it could not start. Both blocks are annotated now and CI's `env-diff` gates them. - `QUEUE_CONNECTION=database` and `SESSION_DRIVER=database` are **deliberate** for the MVP. Redis is full-product scope — do not "fix" this. ### Organization-owned Stripe test mode (FP-T7/T8) There is no platform Stripe configuration in `.env`. Each organization enters its own test publishable key and secret key under Payments; Plantoo encrypts both in that tenant's `stripe_account` row and never returns them. Test Org is the deliberate sandbox-only exception: its Plantoo test-account pair is pinned by `TestOrgSeeder` so the shared acceptance fixture works after every rebuild. The screen displays a tenant-specific webhook URL. Add it to that organization's Stripe test dashboard, subscribe to PaymentIntent events, then paste the endpoint signing secret back into Payments. `TestOrgSeeder` creates `Test P1 — Stripe card payment`, Test Org's stable callback row and its sandbox credentials. The endpoint signing secret is still entered after creating the callback in Stripe. Live keys and live webhook events remain out of scope until FP-T24. ## ⚠️ The staging script as it stands (reviewed 2026-09-03) — three problems The build step currently running on staging has one destructive line and two omissions. ### 1. `php artisan db:seed --force` DELETES DATA ON EVERY DEPLOY — remove it `DatabaseSeeder` calls `AdminSeeder`, which does `table("admin")->truncate()`, and `OrganizationSeeder`, which does `table("organization")->truncate()` and then re-inserts **only** org 1 (Anoath) and org 2 (Day Day Help) with the password `abcd1234`. So every deploy to staging: - **deletes every organization created through the UI** — anything beyond ids 1 and 2; - **resets the passwords** of orgs 1 and 2 back to the seed value; - wipes and re-seeds `admin`; - leaves MongoDB **untouched**, so `user_org_belonging`, `chatroom`, `flow_run`, `flow_version`, `message_template` and every `*_detail` document goes on pointing at organization ids that no longer exist. This is where orphaned documents come from. Seeding is a **first-boot** action, not a deploy action. Run it by hand once when a box is built. ### 2. `config:clear` is missing before `config:cache` A box that ever ran a broken build keeps serving the cache that 404'd all three panels. Clearing first is cheap; not clearing is a panel-wide outage that presents as a routing bug. ### 3. Nothing here starts the queue or the scheduler `queue:restart` does **not** start a worker — it signals existing workers to exit after their current job so they reboot onto the new code. If nothing supervises `queue:work`, that line is a no-op. And there is no crontab entry, so `schedule:run` never fires, so `flow:timer-sweep` never runs: no chases, no give-ups, no appointment reminders, no reschedule chases, no slot-hold releases. See *Long-running processes* below — this is the failure that cost a morning on 2026-09-03. ### Bonus, found while reviewing it: uploaded files were 404ing on the server **Do not add `storage:link`** — this project serves uploads through Laravel, never through a symlink (owner decision, 03/09/2026). It already did, via `filesystems.php`'s `serve => true`… but the framework registers that URL as a **closure**, and `FilesystemServiceProvider::serveFiles()` begins `if ($this->app->routesAreCached()) return;`. The deploy runs `route:cache`. So on staging that route was never registered and **every chat attachment, evidence file and receipt 404'd** — while working perfectly in local development, where routes are not cached. Fixed 03/09/2026: `serve => false`, and `App\Http\Controllers\StorageController` owns `/storage/{path}` as an **invokable**, which is the repo's standing route rule for exactly this reason. `StorageServingTest` guards it, including an assertion that the route is not a closure. ### Corrected build step ```bash set -xe cd /var/www/html/plantoo_tech_staging git reset --hard git pull composer install --no-dev --optimize-autoloader --no-interaction --prefer-dist /home/arfu/.nvm/versions/node/v25.3.0/bin/npm ci /home/arfu/.nvm/versions/node/v25.3.0/bin/npm run build # db — migrations only. NEVER db:seed here: the seeders truncate admin and organization. php artisan migrate --force # cache — clear BEFORE cache php artisan config:clear php artisan config:cache php artisan route:cache php artisan view:cache # queue — LAST of the PHP steps. Restarts existing workers onto the new code; does not start one. php artisan queue:restart # worker — ONLY under WORKER_MODE=supervised, and then it is a supervisor program like the queue # worker. Under `on_demand` there is nothing to restart here: the queue worker runs it per job, and # `queue:restart` above already put that onto the new code. # sudo systemctl restart plantoo_testing_render ``` ## Deploy sequence Order matters, and every line here exists because leaving it out broke something once. ```bash git pull # or however the deploy tool delivers the code composer install --no-dev --optimize-autoloader --no-interaction npm ci && npm run build php artisan migrate --force # migrations are production-facing; see CLAUDE.md php artisan config:clear # BEFORE config:cache, always — see note below php artisan config:cache php artisan route:cache php artisan view:cache php artisan queue:restart # MUST be the last line — see note below ``` **`config:clear` before `config:cache`.** A box that ever ran the broken build keeps serving the cache that 404'd all three panels. Clearing is cheap; not clearing is a panel-wide outage that looks like a routing bug. **`queue:restart` last.** A long-lived `queue:work` keeps executing the code it booted with, so a deployed job-class change never takes effect — it presents as "the fix didn't work". It is the commit point of the deploy: with `set -xe`, an earlier failure should leave the old workers running on the old code rather than half-restarted onto a broken deploy. **Routes must be cacheable.** Every route is an invokable or a controller action, never a closure, because `route:cache` fails on a closure. Keep it that way. ## Long-running processes Three, and the product is visibly broken without the first two: ```bash php artisan queue:work --tries=3 # supervisor: autostart, autorestart, 1+ process php artisan schedule:work # or the standard crontab entry, below node worker/index.js # supervisor: autostart, autorestart, stopwaitsecs=30 ``` ```cron * * * * * cd /path/to/app && php artisan schedule:run >> /dev/null 2>&1 ``` **Under `WORKER_MODE=on_demand` the third one goes away** — the queue worker runs the Node worker when a step asks for a document — so a host needs exactly two things per environment: a supervised `queue:work`, and one cron line driving the scheduler. **Use a SYSTEMD UNIT, because the host already does.** The sibling application on the same box runs `anoath_testing_worker.service` and `anoath_staging_worker.service`, and its scheduler from the deploy user's crontab. A second convention on one host is a second thing to remember at three in the morning, so this follows it rather than inventing. (Its own runbook says supervisor; supervisor is not installed there. Believe the box, not the document — checked 21/09/2026.) `/etc/systemd/system/plantoo_testing_worker.service`: ```ini [Unit] Description=Plantoo Testing queue worker After=network.target mysql.service [Service] User=www-data Group=www-data WorkingDirectory=/var/www/html/plantoo_tech_testing ExecStart=/usr/bin/php /var/www/html/plantoo_tech_testing/artisan queue:work --sleep=1 --tries=3 --max-time=3600 Restart=always RestartSec=5 # NOT in the sibling's unit, so systemd's default 90s applies there. A document render can outlast # that, and a worker killed mid-render leaves a `render_job` row Running that nothing reclaims. TimeoutStopSec=600 [Install] WantedBy=multi-user.target ``` ```bash sudo systemctl daemon-reload sudo systemctl enable --now plantoo_testing_worker systemctl status plantoo_testing_worker # expect active (running) ``` One unit per environment, differing only in name and the two paths. `--sleep=1` rather than the sibling's 3: a queued render is something a customer is watching the chat for. And the scheduler, in the deploy user's crontab — the same shape the sibling uses: ```cron * * * * * cd /var/www/html/plantoo_tech_testing && php artisan schedule:run >> /dev/null 2>&1 ``` `queue:restart` in the deploy then does what it is for: the worker exits after its current job and systemd starts it again on the new code. **Linux keeps nothing alive by itself** — that is the only reason a supervisor appears in this document at all. The scheduler runs `flow:timer-sweep` every minute (`routes/console.php`). **Everything time-based depends on it** — reminder chases, give-ups, appointment reminders, reschedule chases and slot-hold releases are all `Timer` rows that something must come back and fire, and each firing schedules the next one. With no runner a reminder loop stops dead at its first link, which reads as a broken flow rather than a missing worker. It cost a morning on 2026-09-03. `failed_jobs` is the triage table; see the README runbook. ### The Node worker (FP-T26) A second runtime, and the only one. It does the two things PHP should not: fill an official PDF, and (when D2 lands) run a workbook's formulas. **One process, two job types** — the cost of supervising, deploying and watching a process is the same whether it does one job or two, which is why the second arrives as a handler rather than as a second service. - **It needs `npm ci --omit=dev` on the host.** The build already runs npm for the frontend; this adds a runtime dependency on `node_modules` remaining present after the build rather than being discarded. Three runtime packages: `pdf-lib`, `mysql2` and `@univerjs/preset-sheets-node-core`. - **A browser for LAYOUT, never for arithmetic**, and the two are separate decisions that happen to land in the same process. The spreadsheet ran in headless Chromium at one point; it worked, and it was replaced — the board's own words are *"a browser cannot be trusted with a statutory figure"*, and a browser adds a rendering pipeline, a font stack and a window's worth of state to something that is arithmetic. The sums are Univer's own engine, headless, in Node. **Rendering a written document is the opposite case**: a browser is the best text layout engine anybody has, and the figures reach it already decided. So `playwright` is a runtime dependency again, and the host needs its browser. - **Install the browser into a SHARED path, not a home directory.** Playwright resolves browsers under `$HOME/.cache/ms-playwright`, and on a server the user who installs one is never the user who launches it: you install as yourself, Jenkins installs as `jenkins`, and the render runs as `www-data`, whose home holds neither. The failure is `browserType.launch: Executable doesn't exist at /var/www/.cache/ms-playwright/...` on a checkout where the identical job worked by hand an hour earlier — 21/09/2026, and the flow took its `on_error` branch in front of a customer. So, once per host: ```bash sudo PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright npx playwright install --with-deps chromium sudo chmod -R a+rX /opt/ms-playwright ``` then `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright` in `.env` (Laravel passes it down to the worker under both `on_demand` and `scheduler`) and the same line as `Environment=` in the unit under `supervised`. The build step installs into the same path rather than Jenkins's home: `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright ./node_modules/.bin/playwright install chromium`. - **`playwright` is pinned to the same version as `@playwright/test`.** Two Playwright versions in one project fight over browser builds — installing one prunes the other's, and whichever ran last wins. Keep them equal when either moves. - **`stopwaitsecs=30`.** The loop normally finishes the job in hand before exiting. If the process is killed mid-render, the collector reclaims a `Running` row after `WORKER_LEASE_SECONDS` (600 by default), removes its partial private output, and retries it. After `WORKER_MAX_ATTEMPTS` claims (3 by default), it is marked failed instead of looping forever. Keep both values explicit in each deployed environment and alert on exhausted jobs. - **Unlike the two above, nothing is broken while it is down.** Rendering is not on any customer's path today; jobs queue up in `render_job` and drain when it comes back. Worth knowing during a deploy, and worth an alert before anything depends on it. - Logs are one JSON line per job to stdout — `claimed`, `done` with the hash, `failed` with the reason. Point the supervisor's `stdout_logfile` somewhere and that is the whole observability story for now. `worker/README.md` carries the supervisor stanza. - **`WORKER_MODE` decides how this is driven**, and it is the setting to read before deciding what to supervise. Read through `config("plantoo.worker")`, never `env()` — see the panel-domain note above. | mode | what runs it | latency | what the host must supervise | |---|---|---|---| | `supervised` | a systemd unit keeps `worker/index.js` alive | a second | queue worker, scheduler, **worker** | | `scheduler` | the schedule runs `--once` every minute | up to a minute | queue worker, scheduler | | `on_demand` (default) | a QUEUED JOB runs `--once` when a step asks for a document, then collects | seconds | **queue worker only** | `on_demand` is the one that needs no scheduler for documents at all: `App\Jobs\RunRenderWorker` runs the worker and then calls `render:collect` itself, because producing a document and telling the conversation about it are two steps and a host with no scheduler would do only the first. It is `ShouldBeUnique`, so a burst of steps starts one process rather than ten. **The scheduler is still required for everything else** — `flow:timer-sweep` is what fires reminders and deadlines, and nothing dispatches that on demand. `node worker/index.js --once` also runs by hand, which is how a backlog is drained after any of this has been misconfigured — `routes/console.php` then runs it every minute under the same schedule as everything else, with `withoutOverlapping()` so a render that outlasts a minute cannot be joined by a second copy. Set `NODE_BINARY` too where node is not on the scheduler's PATH, which is every nvm host. That takes the box from three long-running processes to two, and costs a document up to a minute of latency plus Chromium's start-up on every run. Choose it where fewer processes matter more than seconds; supervise the worker where a customer is watching the chat for their statement. **Never both at once**: two workers racing one table is safe — the rows are claimed under a lock — but it is twice the CPU for nothing. - **CI/CD does not keep it alive — it only restarts it.** A deploy shell runs once and exits, so anything it starts dies with that session. Under `WORKER_MODE=supervised` the host needs a unit of its own beside the queue worker: ```ini [Service] User=www-data WorkingDirectory=/var/www/html/plantoo_tech_testing Environment=PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright ExecStart=/usr/bin/node worker/index.js Restart=always TimeoutStopSec=30 ``` `Environment=` is not optional here: nothing else sets it for a supervised process, and without it the unit looks for Chromium in `www-data`'s home. The other two modes get it from `.env`. `TimeoutStopSec=30` matters: the loop finishes the job in hand before exiting, and a deploy that kills it mid-render leaves a `render_job` row `Running` that nothing reclaims. Add `sudo systemctl restart plantoo_testing_render` to the deploy, since `git pull` changes its code too. Found on 21/09/2026: the deploy script had never mentioned the worker at all, and documents queued for 104 minutes while the panel correctly said the service was stopped. ## Seeding — read before running it anywhere with real data `AdminSeeder` and `OrganizationSeeder` **truncate**. Running `db:seed` on a live environment DELETES both organizations and leaves `user_org_belonging` documents pointing at ids that no longer exist. Safe on a fresh box, destructive the moment anyone has created anything. A full reset is two commands **in this order**: ```bash php artisan db:wipe --database=mongodb --force # FIRST php artisan migrate:fresh --seed --force ``` `migrate:fresh` only drops MySQL. Without the Mongo wipe first, every document survives what looks like a complete reset — and the result is a MySQL schema with no rows pointing at a Mongo store still full of the old ones. A clean seed produces exactly **one admin and two organizations**, and nothing else: no demo flow, no demo service, no demo user. That is deliberate — building those through the UI *is* the test (`docs/mvp-test-plan.html`). ## Post-deploy smoke check Five minutes, in this order. Anything failing here is a broken deploy, not a flaky test. 1. `curl -fsS https://«org host»/api/health` → `{"status":"ok"}`. 2. Load each of the three panel hosts signed out. Each shows its login form; **none returns 500** (an unauthenticated navigation returning 500 is a regression the browser suite has caught before). 3. Sign into the org panel. The dashboard renders — a blank page means the bundle died on an un-imported constant, which has happened and which no PHPUnit test can see. 4. Confirm both processes are up: `php artisan queue:work` and the scheduler. Then check `select count(*) from timer where status = 1 and fire_at <= now()` is not growing — a rising count is the signature of a dead sweep. 5. Run one flow end to end as a customer in a throwaway organization, and check the message arrives in the org Inbox. Never smoke-test in a real organization. Build a dedicated testing org. ## Demo walkthrough Six scenarios, in order, against a freshly built environment. Preserved here from the MVP release card because the content is release-time work, not a task. **It needs the content built through the UI first**: a clean seed creates two EMPTY organizations — no demo flow, no service, no user. 1. **Tenancy.** Two organizations; suspend one; its chat entry shows unavailable; reactivate. 2. **1-way.** An announcement flow; the message appears in the customer panel. 3. **Booking.** Complete the flow in org A; the appointment shows in *My Schedule* and in the org panel; the slot is gone from availability. 4. **2-way.** Go quiet on a choice node; the timer fires; the reminder arrives; acknowledge. (Needs the scheduler running — see *Long-running processes*.) 5. **3-way.** Upload evidence; approve it in the org's Verifications; the receipt appears; confirmed. 6. **Isolation and concurrency.** A cross-org URL is not-found; parallel booking gives 1×201 and 19×409. ## Not automated, on purpose `npm run e2e` (Playwright, 13 specs) is not in the deploy path. It needs a running app with real hosts, and wiring it is T12's open checkbox — see `docs/progress.md`.