geocrop-platform./apps/nextgen/.harness/plans/2026-07-22-phase2-analytics.md

Phase 2 — Analytics Dashboard Rebuild

Date: 2026-07-22 Owner: Mavis (orchestrator) + backend-expert / frontend-expert / database-expert / tester reins Branch (target): feature/analytics-dashboard-2026-07-22 (worktree from dev, branched after Phase 1 lands) Status: plan (not started) Assignee: Arthur Companion plans: Phase 1 — auxiliary roles + cohorts + assignments · Phase 3 — offboarding · superseded 7-feature draft Parallelizable with: Phase 3 (zero file overlap; both can develop in parallel after Phase 1 lands)

Problem

The /reports page is a single screen with one filter, not a real analytics dashboard. The principal (who shares the /dashboard/admin route with school_admin) needs to answer questions like:

• How many active enrollments do we have right now, and how does that compare to last term?
• What's the attendance rate for Form 4 IGCSE 2026 cohort over the last 30 days?
• Which class has the lowest pass rate this term?
• What's the fee collection rate per cohort?
• Where are the bottlenecks in leave / payroll approval?

The current page renders 4–5 Recharts widgets from /api/reports/summary but is effectively a static report — no time-range filter that's actually wired, no drilldowns, no role-scoped data, no export, and principal is not in the allowed-roles list in some getRoutes() arms of App.tsx. A real dashboard lets a principal open the page first thing in the morning and see the state of the school in one screen.

What this is, in one line: turn /reports into a real analytics dashboard with KPI tiles, time-series, drilldowns, role-scoped tabs, and CSV export.

Scope (in / out)

In this plan:

• New GET /api/reports/kpis, /timeseries, /cohorts, /export endpoints in server/src/controllers/reports.controller.js (extend the existing controller, no new file)
• Full rewrite of client/src/pages/admin/Reports.tsx — header filters, KPI tile row, tabbed sections, drilldown modals, CSV export
• Add principal (and bursar / hr where appropriate) to the allowed roles for /reports in every getRoutes() arm of App.tsx
• Wire the useEducationTerms() locale helpers and the existing date helpers — no hardcoded strings
• Reuse Recharts, lucide, axios, AuditService — no new libraries

Out of this plan (explicitly):

• A second BI tool or external integration (Metabase, etc.)
• PDF export — CSV only this round
• Real-time streaming updates (websockets / SSE) — ship a refresh button + 5-minute in-memory cache
• Replacing the existing /api/reports/summary and /api/reports/dashboard endpoints — they stay; they're consumed elsewhere
• Tightening JWT secret, CORS, request-size limit — separate open follow-ups
• Auto-creating the missing demo sysadmin@school.com seed — carry-over from the 2026-07-19 portal-e2e plan, do it elsewhere if at all

Approach

Backend — extend server/src/controllers/reports.controller.js

The existing controller has summary, dashboard, enrollments, financial, attendance, grades, grades/subject-performance, engagement, export/:type. Keep all of them. Add four new endpoints and one new middleware-helper:

GET /api/reports/kpis?range=6m

Returns:

{
  enrollments: { total, active, newThisTerm, byProgramme: { zimsec: N, igcse: N, ... } },
  attendance: { rate30d, rate7d, trend: 'up'|'down'|'flat' },
  academics:  { avgMark, passRate, topSubject, bottomSubject },
  finance:    { collected30d, outstanding, overdueCount },   // role-gated to finance roles
  staff:      { activeTeachers, onLeave, pendingApprovals }   // role-gated to hr/principal
}

The query plan for each KPI is a single SQL statement with one index scan. If a query takes >500ms on the seed, add a covering index; don't introduce a cache this round.

GET /api/reports/timeseries?metric=enrollments|attendance|marks|attendance&range=6m&granularity=week|month

Returns { points: [{ date, value, secondary? }] }. Granularity week returns ~26 points for 6m; month returns ~6. Use the existing attendance and enrollments tables — both have indexed date / created_at columns. Marks are bucketed by graded_at.

GET /api/reports/cohorts

Returns per-cohort performance:

[
  { cohortId, name, programme, level, academicYear, studentCount,
    avgMark, passRate, attendanceRate, feeCollectionRate }
]

This endpoint assumes student_cohorts + cohort_students exist (Phase 1). If Phase 1 is delayed, gate this endpoint behind a feature flag and ship it later — the rest of the dashboard works without it.

GET /api/reports/export?type=overview|academics|cohorts|finance&format=csv&range=6m

Server-side CSV streaming using the same queries as the JSON endpoints. The existing GET /api/reports/export/:type route already does CSV for one report type — generalize it to accept type as a query param, support the four new types, and stream the response (don't buffer).

Auth / RBAC:

• Read paths: school_admin | systems_admin | principal | bursar | hr (each role sees a different subset of tabs)
• Use the hasRole helper from Phase 1 if it has landed; otherwise use the existing req.user.role checks
• The role-scoped data filtering happens server-side in the controller, not the client. The client just receives what its role is allowed to see.

Performance:

• Each KPI = one SQL statement, one index scan. Verify with EXPLAIN QUERY PLAN on the seed.
• Time-series = one SQL statement with GROUP BY date(?, 'week'|'month').
• Per-cohort table = one SQL with LEFT JOIN student_cohorts + cohort_students + attendance + grades + invoices. If the JOIN is slow at the school's data size, denormalize later — but ship the JOIN first.
• No new indexes this round. Add them only if a query is actually slow.

Frontend — full rewrite of client/src/pages/admin/Reports.tsx

The existing page already uses Recharts and lucide. Keep the import set; replace the layout.

Layout:

┌─────────────────────────────────────────────────────────────┐
│ Header: time-range picker | cohort filter | programme |     │
│         class filter              [Refresh] [Export ▼]      │
├─────────────────────────────────────────────────────────────┤
│ KPI tile row (5 tiles) with sparklines                      │
│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐                    │
│ │Enrol│ │Atten│ │Acad │ │Finan│ │Staff│                    │
│ └─────┘ └─────┘ └─────┘ └─────┘ └─────┘                    │
├─────────────────────────────────────────────────────────────┤
│ Tabs: [Overview] [Academics] [Cohorts] [Finance] [Staff]   │
├─────────────────────────────────────────────────────────────┤
│ <Active tab content>                                        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Tab 1 — Overview (default)

• Enrollment area chart (line + area fill) from /api/reports/timeseries?metric=enrollments
• Attendance line from /api/reports/timeseries?metric=attendance
• Fee collection stacked bar (only visible to finance roles; backend returns 403 for others)

Tab 2 — Academics

• Grade distribution bar chart (A/B/C/D/F) — data already in /api/reports/grades
• Subject performance horizontal bar — data already in /api/reports/grades/subject-performance
• Top/bottom classes table (computed server-side from the same query)
• Click a class row → drilldown modal with that class's students ranked

Tab 3 — Cohorts (requires Phase 1)

• Table of cohorts from /api/reports/cohorts with sortable columns
• Click a row → drilldown modal with that cohort's students ranked
• If student_cohorts doesn't exist (Phase 1 hasn't shipped), this tab is hidden, not broken

Tab 4 — Finance (only visible to bursar | school_admin | systems_admin | principal)

• Reuse the existing /api/reports/financial endpoint
• Add a "fee collection rate per cohort" stacked bar (uses /api/reports/cohorts data)

Tab 5 — Staff (only visible to hr | school_admin | systems_admin | principal)

• Teachers on leave (count + list)
• Pending leave / payroll approvals (count + drilldown)
• Staff count by department (pie chart)

Filters (header):

• Time range: 7d / 30d / 90d / 6m / YTD / custom (date range picker)
• Cohort: dropdown populated from /api/cohorts?is_active=1
• Programme: ZIMSEC / IGCSE / AS / A2 / primary / other (only enabled when cohorts are present)
• Class: dropdown populated from /api/classes

Export button: top-right, dropdown of Overview / Academics / Cohorts / Finance. Each option hits /api/reports/export?type=...&format=csv&range=...&cohort=.... Browser downloads the file.

Routing fix (in client/src/App.tsx):

• In every getRoutes() arm that lists /reports as a route, add principal to the roles: [] array. Also add bursar and hr where the role is allowed to see the relevant tab.
• Verify with git grep -n "path: '/reports'" client/src/App.tsx before merging.

Existing components to keep:

• CustomTooltip (already in Reports.tsx) — reuse, no change
• The 4–5 existing chart widgets — delete; they're replaced by the new tabs

New components:

• KpiTile.tsx — small tile with a value, a label, a sparkline, a trend arrow
• DrilldownModal.tsx — generic modal that takes a title + a child for the body
• FilterBar.tsx — the header filter row, with the four filter controls + refresh + export

Sync / offline

This plan adds no new tables. Zero sync impact. All queries are read-only against existing tables. The dashboard works offline if the data was already synced (the existing data layer handles that), but we don't add new offline-first behavior for it this round.

Files to add / change

Backend (modify only)

• server/src/controllers/reports.controller.js — add kpis, timeseries, cohorts, generalize export; add a roleScopedKpis(user) helper that filters the response by role
• server/src/index.js — no change (the new routes are added inside the existing controller)

Frontend (new)

• client/src/components/KpiTile.tsx
• client/src/components/DrilldownModal.tsx
• client/src/components/FilterBar.tsx

Frontend (modify)

• client/src/pages/admin/Reports.tsx — full rewrite (existing component, same path, mostly new JSX)
• client/src/App.tsx — add principal, bursar, hr to the /reports allowed roles in every getRoutes() arm
• client/src/components/Nav.tsx — no change (the Reports nav entry already exists for the relevant roles)

Docs / changelogs

• .harness/changelogs/2026-07-22-analytics-dashboard.md
• .harness/docs/conventions.md — only if a new analytics pattern is introduced (likely not)

Constraints / decisions baked in

• Reuse, don't replace. Keep the existing /api/reports/summary and /api/reports/dashboard endpoints — other pages consume them. Only add the four new endpoints.
• Role-scoped data, server-side. The backend filters by role; the client just renders what it gets. A 403 from a role-gated section is silent (the tab doesn't render), not a toast.
• Cohorts tab is conditional. If Phase 1 hasn't shipped when Phase 2 merges, the Cohorts tab is hidden, the rest of the dashboard works. Use a feature flag if needed; we don't need a real one if we're disciplined about merging order.
• No new libraries. Recharts, lucide, axios are all already in the bundle.
• No real-time updates. Refresh button + 5-minute in-memory cache (useState + setInterval, not Redux or any new state lib). On ?range= change, refetch.
• No new tables, no sync impact. Read-only against existing schema.
• CSV only, no PDF. PDF export is a follow-up.
• The "73 TypeScript warnings" issue raised in the earlier critique isn't a blocker for this plan — we're not adding enough new TS to make the warnings materially worse, and the project compiles with strict: false already.
• No principal-specific portal. Principal shares the school_admin portal with role-scoped data, per existing convention.

Verification (acceptance bar)

This plan is "done" when all of the following pass:

1. Migrations are not needed — this plan adds no new tables. (Confirm: git diff --stat on server/src/database/ is empty for this PR.)
2. Endpoints work:
   • GET /api/reports/kpis?range=6m returns the documented shape with non-zero values from the seed.
   • GET /api/reports/timeseries?metric=...&range=...&granularity=... returns 6+ points for 6m, 26+ points for 6m+week.
   • GET /api/reports/cohorts returns one row per cohort (or [] if Phase 1 hasn't shipped — the page handles both).
   • GET /api/reports/export?type=...&format=csv downloads a non-empty CSV.
3. Dashboard renders for the right roles:
   • school_admin sees all 5 tabs.
   • principal sees all 5 tabs (after the routing fix in App.tsx).
   • bursar sees Overview, Academics, Cohorts, Finance (no Staff).
   • hr sees Overview, Academics, Cohorts, Staff (no Finance).
   • student, parent, teacher get a 403 from the route, not a half-rendered page.
4. Time-range filter actually changes the data — set range to 7d, then 6m, the values change.
5. Cohort tab drilldown — click a cohort row, the drilldown modal shows the cohort's students ranked by some metric. The modal has a close button + Esc-to-close.
6. CSV export downloads a non-empty file for each of Overview / Academics / Cohorts / Finance.
7. Performance — every KPI query and the time-series query is <500ms on the seed. Use EXPLAIN QUERY PLAN to verify. If anything is slow, add a covering index and document the choice.
8. E2E (Playwright): one spec loads /reports as school_admin, verifies all 5 tabs render, switches time range, drills into a cohort, exports a CSV. Existing portal-smoke tests still pass.
9. No regression: existing admin/principal dashboards, the exam-review module, and the finance module all still load and respond.

Execution order (this plan only)

PR 1: Backend endpoints (kpis, timeseries, cohorts, export)
PR 2: Frontend rewrite (Reports.tsx + KpiTile + DrilldownModal + FilterBar)
PR 3: Routing fix (App.tsx — add principal/bursar/hr to /reports) + E2E spec

PR 1 and PR 2 can be developed in parallel (different files). PR 3 is small and merges last because it depends on both the new endpoints and the new frontend.

If Phase 1 hasn't landed yet, PR 1's /cohorts endpoint is added with a featureFlagCohorts query param (default false) and the Cohorts tab in PR 2 is hidden until the flag is on. The rest of the dashboard ships.

Out of scope (explicitly)

• A second BI tool or external integration
• PDF export this round
• Real-time streaming updates
• Replacing the existing /api/reports/summary and /api/reports/dashboard endpoints
• Auto-creating the missing demo sysadmin@school.com seed
• Migrating the existing adminOrTeacher middleware to the hasRole helper from Phase 1 (the migration sweep is a separate follow-up)
• Tightening JWT secret, CORS, request-size limit — separate open follow-ups

Parallelizability with Phase 3

This plan and Phase 3 (offboarding) share zero files:

• Different controllers (reports.controller.js vs offboarding.controller.js)
• Different frontend pages (Reports.tsx vs Offboarding.tsx / OffboardingWizard.tsx / Alumni.tsx)
• Different tables (no new tables here, vs offboarding_records + offboarding_actions + users.archive_status ALTER in Phase 3)
• Different sync impact (none here, vs 2 new tables in Phase 3)

After Phase 1 lands, a dev can pick up Phase 2 in one worktree and another dev can pick up Phase 3 in another. The only coordination is the merge order: either order is fine, since the two PRs don't touch the same code.