USP Aberta

1Context and problem

Brazil’s research CVs live in the Lattes platform^[1], a rich but closed-shaped source: every curriculum is a large XML document and the data on USP’s faculty sat inside a 9 GB database view. Anyone asking simple questions (who works on this topic, which units collaborate, how is production distributed) had to go around the platform instead of through it.

Problem. Make the university’s research output searchable and explorable by the public, without paid search APIs, with personal data kept out, and on infrastructure small enough for one engineer to operate. Early versions crashed from running out of memory under crawler and analytics load.

Decision. Treat it as a data product: a reproducible pipeline from the source view to purpose-built stores (Elasticsearch for search and statistics, Neo4j for graph queries), a thin Django API, and a Next.js front end that renders everything server-side and caches aggressively.

2Architecture

One public entry point, one server-side path to the data. The browser never talks to Django directly: Next.js calls the API server-side, so the API surface can stay private and cache policy lives in one place.

Application

Next.js 15 · React 19 · TypeScript 5 · Tailwind v4 · Django 5.2 · DRF · Go

Data

PostgreSQL · Elasticsearch 8.12 · Neo4j 5 · Apache Airflow · Apache Jena Fuseki

Delivery

Docker Compose · GitLab CI · Terraform · Ansible · systemd

Quality

pytest-bdd · Vitest · Playwright + axe · Cypress · k6 · OWASP ZAP

3Data engineering

Problem. The source is a database view of raw curricula that includes personal data, duplicates and uneven quality; the product needs clean, deduplicated, privacy-safe documents, refreshed repeatedly without downtime.

Decision. A dedicated read-only role reads the view; the ETL strips 13 private tags (tax ID, address, phones, e-mail, birth data and others) before anything is stored, and the newest curriculum version wins. A single command chain refreshes everything downstream.

• ETL run of 2026-10-05: 436,480 source rows read, 347,554 upserted, 168 unparseable, 0 errors; the rest had no Lattes ID and were skipped.
• Record linkage: works are matched by normalized DOI, otherwise by title, year and category. That turns 1,514,901 author-works into 1,339,162 unique works (88.4%); 132,098 of them have more than one USP author. The same idea, applied to events with a Fellegi–Sunter^[2] style blocking rule, powers the Ágora calendar.
• Data quality: measured on the seven dimensions of ISO/IEC 25012^[3], stored as snapshots, served by an API and exported to Prometheus as a ratio.
• Gender inference: an IBGE first-name table first, then an SVM that abstains when unsure: 98.0% accuracy at 94% coverage on 3,000 names, with 4.7% of faculty reported as “undetermined” instead of guessed.

Mnemosine bots. External sources (university news, innovation hub, patents, course catalogues, open-data providers) are collected by eight Airflow-based bots. Each bot owns its schema; raw responses are stored compressed in a bronze layer, deduplicated by SHA-256 and versioned append-only, so a parser fix can reprocess history without re-fetching. One example: the news archive holds 52,606 posts and 6,552 podcast episodes.

• Polite by design. signed user agent with contact header, robots.txt honored^[4], at least 3 s between requests to university hosts with jitter, one shared Airflow pool slot, backoff that honors Retry-After, a circuit breaker whose cooldown doubles up to 24 h, and heavy work only off-peak.
• Free and open only. a standing rule: every source must be open-licensed, free and approval-free, and shows a citation badge on the site. 41 providers feed the events calendar’s enrichment.

4Search and information retrieval

Problem. Names, research areas and biographies behave differently, and a single ranking function treats them all alike.

Decision. Field-specific similarity: language-model smoothing with Dirichlet priors (μ = 2000)^[5] on names and biographies, divergence-from-independence on areas of expertise, BM25 on production titles, and an edge-n-gram analyzer for autocomplete. There is deliberately no stemmer, stop-word list or synonym table: the corpus is names and technical terms.

Evaluation. An offline harness runs 1,200 queries (12 query classes × 100) and scores nDCG@k^[6] with k = 3. The gating rule is that no query class may get worse. The BM25 versus Dirichlet comparison is the subject of a conference submission; I am not quoting nDCG values here until that paper is public.

• Telemetry for click models. queries and interactions (impressions, ranked clicks, dwell, returns) are logged against a pseudonymous 30-minute session id, exportable for unbiased learning-to-rank, and summarized in a Grafana dashboard that tracks mean reciprocal rank.
• Input limits. queries up to 1,000 characters, validated in the browser with an inline counter so a user never triggers a backend error; queries over eight words skip fuzzy matching.

5Graph benchmark

Co-author networks are the one workload where an inverted index is the wrong tool. Rather than argue, I benchmarked it.

Result. Neo4j serves only the collaboration graph, through eight Cypher strategies. Only the traversal depth is interpolated into a query, from an allow-list of 1, 2 and 3; everything else is a bound parameter. Graph builds stream results instead of materializing them after an out-of-memory incident.

6Caching and performance

Problem. Statistics pages aggregate over a million works, and public traffic is bursty. Cold requests were slow, and under load the host ran out of memory.

Decision. Three cache layers, each with its own invalidation story:

• Elasticsearch: the shard request cache raised from the 1% default to 10% of heap.
• Django: cached responses for 24 h (pure aggregations), 1 h (admin-editable data) and 30 min (trending). A database-backed generation counter invalidates statistics, so every worker drops them within about a minute.
• Next.js: incremental static regeneration with a one-hour shared-cache lifetime, tag-based revalidation from an internal endpoint, and a start-up warm-up that waits for the API to answer before it purges.

• Memory. the search heap went from 256 MB to 768 MB after OutOfMemory errors; the works-index build used to abort after 1,040 of 5,653 faculty and now spools to disk and finishes 1.34M works in about three minutes; co-author builds stream with a resident-memory circuit breaker at 1 GiB.
• Reindexing. batches run at the quietest hour predicted from 14 days of Prometheus request rates, instead of on a manual button.

7DevOps

Decision. Production is a single Docker Compose project. Deploys are deliberately simple and reversible: every release is an immutable image tag, the previous one is kept as a rollback tag, and images are shipped with a resumable, checksummed transfer so an interrupted upload resumes instead of restarting.

• Pipeline. GitLab CI defines build, test (SAST, SEO budgets, k6 load, OWASP ZAP, network-chaos with netem), deploy and metrics stages. Pushes are guarded by a pre-push hook that runs the suites.
• Infrastructure as code. Terraform and nine Ansible roles (secrets in Vault) describe the earlier cloud setup. A k3s and ArgoCD migration path was built, evaluated and deliberately deferred in August 2026, recorded as a decision rather than left as drift.
• Self-healing. systemd timers and watchdogs restart what matters and protect the remote-access daemons from the OOM killer; scheduled jobs are timers because the host has no cron.

8Observability and SRE

Prometheus (15-day retention), Grafana, Alertmanager and blackbox probes cover the stack, organized around the eight DORA-style measures: MTTR, lead time, deploy frequency, change-failure rate, availability against a 99.5% SLO, deploy duration, time to detect and response-time histograms^[7].

• Argos status page. a Go service probes 11 components every 60 seconds, only flips a state after three consecutive checks (a flap guard), draws 90-day uptime bars, opens and resolves incidents automatically and e-mails on both events. It is public at /status.
• Runbooks. four written runbooks (backend 5xx, Elasticsearch degraded, latency, search incident); search failures return a localized problem+json body with a request id.

9Security

Decision. Identity is a small Go service (auth-server): RS256 JSON Web Tokens with a 15-minute lifetime, a JWKS endpoint with RFC 7638^[8] key identifiers, Google OAuth as the only sign-in, signed CSRF state, and deny-by-default role claims (user, admin, super admin). It also acts as the single-sign-on provider for Grafana and Airflow.

• Password login was removed after bots brute-forced it. The upstream proxy does not forward client IP addresses, so IP-based throttling is ineffective by construction; defenses are identity-based and route-based instead: nginx rate-limit zones (alerts, graph, auth, known bots) and DRF throttles on graph queries.
• Tamper-evident audit trail. an insert-only database role, a hash chain computed by a database trigger under an advisory lock, a command that verifies the chain, monthly partitions and two-year retention.
• Injection. bound parameters everywhere; the single interpolated value in Cypher comes from an allow-list.
• Scanning. bandit, pip-audit, pnpm audit, gitleaks, gosec, golangci-lint and an OWASP ZAP baseline run in CI.

10Privacy and compliance

The platform publishes academic data about named people, so privacy is a design input rather than a checklist. A record of processing activities under Brazil’s LGPD^[9] states a purpose and legal basis per activity (faculty data, audit, mailing lists, news, search logs).

• Minimization. personal data is stripped twice, at ETL time and again at index time; the internal university-number field was deleted from every layer in June 2026.
• Retention. search logs 13 months, application logs 30 days, audit records two years.
• Telemetry. honors Global Privacy Control and Do Not Track, scrubs e-mails, tax IDs and phone numbers from queries, and never stores IP address or user agent.
• Mailing lists. double opt-in, addresses encrypted with AES-256-GCM plus an HMAC blind index for lookup, one-click unsubscribe^[10], no tracking pixels, unconfirmed sign-ups purged after 48 hours.
• Open licensing and attribution. site content is CC BY-NC-SA 4.0^[11] and every external source carries a visible citation badge.
• Accessibility. automated axe-core scans against WCAG 2.1 A and AA fail the build on serious or critical issues.

11SEO and LLM discoverability

Problem. A public-interest site that search engines and AI assistants cannot find does not serve the public. The sitemap held three URLs.

• Sitemap. now an index of child sitemaps (up to 10,000 faculty URLs each) with real last-modified dates taken from the curricula: more than 6,200 URLs, including 5,552 faculty profiles.
• Canonical hygiene. duplicates redirect with 308 to the canonical URL, and soft 404s became real 404s.
• Structured data. JSON-LD for WebSite with SearchAction, Organization, Person, CollegeOrUniversity, Museum, ResearchOrganization and BreadcrumbList.
• For language models. /llms.txt and /llms-full.txt, a compact Markdown version of every faculty profile (a few kilobytes against 1.1–1.6 MB of HTML), an A–Z directory, and an explicit robots.txt group for 16 AI crawlers.
• Regression gates. render, bundle and Lighthouse budgets plus a Playwright SEO spec; on the first run against the old site, 15 Lighthouse CI assertions failed.

12Testing and quality

Behavior-driven tests describe the product in the vocabulary of the people who use it, and a pre-push hook runs the suites before code leaves the machine.

36

pytest-bdd feature files

208

Vitest test files

110

Go test files

15

Playwright specs

SEO, accessibility, visual

The backend coverage gate is set at 90%; that is a threshold, not a measured claim. Non-functional checks include k6 load tests, OWASP ZAP, network chaos and memory profiling. A behavioral code analysis in the style of Tornhill^[12] over 656 commits guides where to pay down technical debt first.

13Product surface

What the platform offers today, all generated from live data instead of hand-written copy:

• Faculty profiles, units, departments and museums, with spreadsheet-style in-place editing for curators (undo, redo and a revertible change log).
• Statistics: production over time, areas of knowledge, publication countries, gender distribution.
• A collaboration network with a graph-or-table toggle.
• Ágora: an all-USP events calendar with an .ics export and a REST API, a sub-project with its own page.
• Innovation: patents, technology-transfer data and an eligibility checker that runs entirely in the browser.
• Courses and curricula, and a graduation checker (/formatura) that parses a student’s transcript PDF client-side.
• A press archive, e-mail newsletters and the public status page.

14What’s next

The roadmap is research as much as engineering. Embeddings for faculty and works are already computed with an open model; the LangChain orchestration and the generation layer of a retrieval-augmented pipeline are next, gated on data quality, which is why the quality report comes first. Further out: formal verification of the collection pipelines with Petri nets, a lexical-versus-semantic comparison on real usage logs, and widening the corpus to artistic events and Brazilian music.

This case study is the basis of a master’s research plan; see the Research page.