Light combines the ability to carry quantum information in ambient conditions with a large information capacity, making it ideal for building quantum networks. However, due to the probabilistic nature of linear-optical entangling operations, it remains an outstanding challenge to grow such networks. Historically, the goal of swapping entanglement over large scale networks motivated the development of "quantum repeaters'', based on quantum memories that can trap and release photons on demand to synchronise entangling operations. In contrast to long-distance communications networks, local networks can operate fast and with low latency. For such local quantum networks, the key figures of merit are low noise and the time-bandwidth product, i.e. the number of clock-cycles over which a memory allows synchronisation. While quantum-limited performance, large time-bandwidth products, and operation in ambient conditions have been shown separately, to date no system satisfies all these desiderata simultaneously. Here we introduce and demonstrate a new memory protocol -- the off-resonant cascaded absorption (ORCA) memory  -- that is optimised for the aforementioned low-latency applications. Based on off-resonant stimulated two-photon absorption into a doubly excited electronic state in warm atomic vapour, ORCA is the first broadband and intrinsically noise-free room-temperature quantum memory. We use the ORCA scheme to store, and retrieve on-demand, GHz-bandwidth heralded single photons with 15% efficiency, measuring a heralded autocorrelation g(2)=0.028(9), the most statistically significant anti-bunching recorded from any quantum memory . These results open a route to arbitrarily scalable local photonic quantum networks.