fchinembiri
/
geocrop-platform.
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
geocrop-platform./apps/nextgen/.harness/changelogs/2026-07-17-p2-backup.md

8.1 KiB
Raw Blame History

Changelog — 2026-07-17 — P2 backup daemon (Track G) + P2-5 bulk marks audit

Branch: ops/sqlite-backup Worktree: .worktrees/ops-sqlite-backup/ Owner: developer (devops) Date: 2026-07-17

This change adds the production-host backup / restore / verify tooling for the Africa Alert SQLite database (P2-1), and documents an audit of the existing bulk marks CSV import path (P2-5). No application code is touched and no schema changes are introduced.

Files added

Path Purpose
scripts/backup/africa-alert-backup.sh Hot online backup via sqlite3 .backup. Configurable via DB_PATH, BACKUP_DIR, LOG_DIR, RETAIN_DAILY env vars. Copies -wal/-shm sidecars if present, writes a sha256sum next to the snapshot, and prunes the local retention window.
scripts/backup/install-backup-cron.sh One-shot installer. Appends a 0 */12 * * * cron entry (configurable via CRON_SCHEDULE), idempotent via a crontab -l | grep -q africa-alert guard. Logs the next-run estimate to logs/backup-install.log.
scripts/backup/restore.sh Destructive restore. Prompts for typed "yes" at a TTY (or RESTORE_CONFIRM=yes from automation), stops africa-alert.service via systemctl, takes a pre-restore-YYYYmmdd-HHMMSS safety snapshot, copies the backup (and its sidecars) over the live DB, removes any stale live sidecars that do not have a matching backup sidecar, then restarts the service.
scripts/backup/verify-restore.sh Smoke-test. Copies the backup to a temp dir, spawns the server with DB_PATH and PORT overrides, polls /api/auth/login until the route is ready, asserts a 200 with the admin creds (admin@school.com / admin123), and cleans up. Exits 0 on success, 1 on failure.
docs/BACKUP.md Operator runbook. Installation one-liner, manual trigger, restore drill, retention policy (30 daily + 12 weekly with a copy-pasteable find command), failure-alerting contract, and a full disaster-recovery scenario for a bare-metal rebuild.

No .sh script in this change was executed on Windows. They are written for the production Ubuntu 22.04 host. Verification on this dev machine is limited to bash -n syntax checks (results in §Verification below).

Bulk marks audit

Audit target: server/src/controllers/marks.controller.js — the POST /api/marks/bulk endpoint that the teacher CSV importer calls.

Expected CSV header

assignment_id,student_id,score,feedback

The controller uses csv-parse/sync with columns: true, so the first non-empty row is the header. Columns are lowercased by trim: true; the parser trims whitespace on each value and tolerates a UTF-8 BOM (bom: true).

Per-row behaviour (summary)

  1. Validate the four required fields: assignment_id and student_id must be present and resolve to a real assignments.id / users.id; score must be numeric, non-negative, and <= assignments.max_score when max_score is non-null. The student must have role='student'.
  2. Look up an existing submissions row by (assignment_id, student_id). If found, UPDATE the row's grade/feedback/graded_by/graded_at, force status='graded', set sync_status='pending', and bump updated_at. If not found, INSERT a new row with the same fields, a freshly generated uid, and sync_status='pending'.
  3. The whole loop is wrapped in a single better-sqlite3 transaction, so the per-row counters are consistent. Per-row errors are collected into an errors array and the loop continues; a single bad row never aborts the rest of the batch.
  4. After the transaction, the controller writes a best-effort audit log (AuditService.log, action BULK_UPLOAD_MARKS, entity type submission) with { total_rows, inserted, updated, error_count, filename }. Audit failures do not fail the response.
  5. Response: { inserted, updated, errors: [{ row, reason, data }], total_rows }.

Gap analysis

  • No header validation. A CSV with the wrong header (e.g. student_name, score because the teacher used the wrong export) will produce 0 inserts and N errors all of the form missing assignment_id, with no upfront "wrong header" message. The recommended fix is a small header check at the top of the handler that returns 400 with a clear message naming the expected columns.
  • No teacher ownership / class-scope guard. The adminOrTeacher middleware is permissive: any teacher can grade any student for any assignment. A teacher could overwrite another teacher's grades by re-uploading the same CSV. A teacher.assigned_classes / assignments.created_by check would close this gap; not done here because the existing RBAC refactor (Track C P1-10) is the right place to add it consistently.
  • No dry_run mode. The handler always commits. A query flag (?dry_run=true) that runs the parse + validation + assignment/student pre-cache but skips the transaction would let operators catch malformed CSVs cheaply.
  • Score type is loosely validated. Number(scoreRaw) accepts floats (75.5) even when the assignment expects an integer. The upper-bound check (score > a.max_score) catches overshoots but a fractional score that happens to be ≤ max_score will be stored as a float. Either pre-round or require integer explicitly.
  • MIME allowlist is permissive on purpose (application/octet-stream and empty MIME are accepted so that browsers and curl both work). This is documented in the controller comment, but is worth a re-review when the file-attachment work in Track H lands a unified multer configuration.

Recommendation

Keep as-is for this PR — these gaps are real but small, and the fixes are best done in the same PR as the broader RBAC refactor (Track C) and the dry-run work (Track B tests). Track G's scope is backup tooling, not the marks controller. The audit findings above are captured here so the next agent who picks up Track C / Track B has them.

Verification

The repo's dev machine is Windows 11. The .sh scripts target the production Ubuntu 22.04 host. Verification on this box is limited to bash -n (syntax-only) using Git Bash. The actual end-to-end smoke (bash africa-alert-backup.sh && bash verify-restore.sh) is run on the production host after the branch is deployed.

Check Result
bash -n scripts/backup/africa-alert-backup.sh SYNTAX_OK
bash -n scripts/backup/install-backup-cron.sh SYNTAX_OK
bash -n scripts/backup/restore.sh SYNTAX_OK
bash -n scripts/backup/verify-restore.sh SYNTAX_OK
wc -l docs/BACKUP.md 283 docs/BACKUP.md (target: ≥80)
Marks bulk-audit findings documented above; one-sentence recommendation: keep as-is, fix in Track B / Track C

Deviations from the plan

  • .gitignore covered already: logs/ is created at the project root for backup.log / verify-restore.log / backup-install.log. The existing *.log entry in .gitignore (line 9) catches the per-run log files, so no .gitignore change is needed.
  • G.8 (Playwright bulk-marks.spec.ts) deferred to Track B. The plan notes "Skip this if it would risk breaking the existing 24/24 Playwright baseline." The Playwright test infrastructure is on a different worktree (feature/playwright-e2e); adding specs from this branch would either duplicate fixtures (fragile) or skip them (defeats the purpose). Track B owns the test infra; the three specs listed in the plan should be added there.

Follow-ups

  • On the production host, after deploy: run bash -n once more (paranoia), then sudo /usr/local/bin/install-backup-cron.sh and crontab -l to confirm the entry. Run one manual backup, then verify-restore.sh against the new snapshot to confirm the round trip.
  • Wire logs/backup.log to the operator's monitoring (Promtail, mailx, Telegram webhook, etc.). The contract is in docs/BACKUP.md §6.
  • Promote the 12-weekly retention in docs/BACKUP.md §5 to a real cron entry. The runbook gives the script; the cron line is left for the operator to paste.
  • Add the four server/src/controllers/marks.controller.js audit items to the Track C / Track B backlogs.