The most authentic local experience you can possibly have in Reykjavík is visit a public geothermal pool to get a glimpse of everyday local culture.