How this site works
WhatsUp Lisbon is a fully automated agenda. No editor picks the events: a pipeline finds them, cleans them, translates them and publishes them. This page explains each stage, honestly, including the parts still being built.
- 1 · Find
Sources are scraped every two hours
Around twenty independent source adapters each know how to read one publisher of events: municipal agendas (Agenda Cultural de Lisboa, CM Sintra, Cascais 360), ticket vendors (Ticketline, BOL), venues' own websites (Gulbenkian, ZDB, Village Underground, São Carlos, Hot Clube, Quinta da Regaleira and more), and aggregators (Viral Agenda, Resident Advisor). A scheduled job runs them all every two hours. Each adapter fails independently: one broken website never blocks the others, and a source's old events are only removed once that source has scraped successfully again.
- 2 · Identify
Events get a stable identity, venues get merged
An event's identity is a hash of its venue, title and start time, so re-scraping the same event updates it instead of duplicating it. Venue names arrive spelled a dozen ways ("MAAT", "MAAT — Museu de Arte, Arquitetura e Tecnologia"), so every listing is mapped onto one canonical venue: identical names collapse automatically, and the harder cases are written down by hand rather than guessed at, because a wrong merge quietly moves events to the wrong building.
- 3 · Dedup
The same show from two sources becomes one card
Big shows appear in several sources at once, usually with different titles (the venue announces "Evita", the ticket vendor sells "EVITA"). After every scrape a dedup pass compares events that share a venue and a date: titles are normalized and matched by word overlap, and near-identical pairs are collapsed. One version is kept as canonical — preferring the listing with a showtime, a description and an image — and the others are hidden but keep contributing their links, so a card can point to both the venue's page and the ticket shop.
- 4 · Enrich
A language model writes the trilingual summaries
Scraped descriptions are messy, Portuguese-only, or missing. After each scrape, new events are sent to a small language model that writes a short factual summary and title in English, Spanish and Portuguese. The original Portuguese title is kept and shown as a tooltip. The same pass tags whether an event is free and whether it is family-friendly when the source doesn't say so outright.
- 5 · Illustrate
Images are normalized, and sometimes repainted
Posters come in every shape; cards use a fixed 3:4 frame. Most images are shown as-is. For events with poor or missing artwork there is an opt-in image pipeline: a cheap image model produces a uniform watercolor version (about $0.03 per image), and recurring formats like weekly jam sessions will draw from a small pool of reusable generated artwork instead of generating one image per occurrence.
- 6 · Publish
Deploys are boring on purpose
The site is a SvelteKit app on the Bun runtime with a single SQLite file as its database. A deploy builds the site locally, syncs it to a small VPS and restarts one systemd service. The scraper runs inside the same app, triggered by a secret-guarded endpoint on a timer.
Where hosting is heading
The current single-server setup has two weaknesses: deploys can briefly mix old and new assets, and one machine is one point of failure. Because the whole site is read-only for visitors, the plan is to go fully static: a scheduled job (GitHub Actions) runs the scrape + enrichment, builds the site as plain files, and pushes them to a CDN host (Cloudflare Pages). That gives atomic deploys, effectively unlimited uptime, no server to patch, and near-zero cost — the "server" then only exists for a few minutes every two hours, inside the build job. The feedback form stays as one tiny dynamic endpoint.