xnths.

Sub-project · Live · Part of USP Aberta

Ágora: the all-USP events calendar

A calendar of lectures, symposia, concerts and exhibitions across the University of São Paulo, assembled from announcements and enriched from open sources.

Engineered by Xnths as part of USP Aberta. Formerly called Orfeu.

At a glance

Role
Sole engineer
Links
Stack
Go · Apache Airflow · PostgreSQL · REST · iCalendar
41
open enrichment providers
83
events recovered from a replay of 233 sheet rows
after fixing the load step; 49 rows quarantined

1Problem

Events at a university of this size are announced everywhere and collected nowhere. The proof of concept started from announcement e-mails and a spreadsheet.

2Pipeline

  1. 01Extractpublished sheet · adapters
  2. 02Normalizedates · units
  3. 03Validatequarantine
  4. 04Deduplicateblocking + similarity
  5. 05Loadidempotent
Fig. 1Events pipeline. Rows that fail validation go to quarantine instead of being dropped or loaded.

Deduplication. Duplicates are merged by blocking on title and date, then comparing start times (within 30 minutes) and venue similarity (at least 0.8), in the spirit of Fellegi and Sunter[1].

  • Source adapters follow the adapter pattern, so a new source is a new file and not a change to the core.
  • Enrichment crosswalks identifiers with union-find over equivalences (at most three rounds) and caches for seven days, serving stale data on upstream errors[2].

References

  1. [1]Fellegi, I. P., & Sunter, A. B.. A theory for record linkage. Journal of the American Statistical Association 64(328) (1969).
  2. [2]Nottingham, M.. HTTP Cache-Control Extensions for Stale Content (RFC 5861) (2010).