Catalog: parallelize all-sky tile prefetch

This commit is contained in:
wyj committed 2026-08-27 03:53:45 -04:00
1 parent d9f2c83b9f
commit 457c0728b4
10 files changed
+371 -55

No files matched your search

+38 -1
View File
@@ -46,7 +46,44 @@ make ENABLE_PNG=1 SPACETIME=minkowski all
```
The renderer lazily loads only tiles touched by source triangles in the frame.
It prefetches them serially, so OpenMP splat workers read immutable tile data.
It first collects and de-duplicates the frame's touched tile IDs on the main
thread. Newly needed CSVs are then read by a bounded OpenMP loader group
(`--catalog-load-workers N`, default `4`); each worker owns distinct tile
files. The main thread commits their results to the cache only after that group
finishes, so OpenMP splat workers always read immutable tile data and never
race catalog I/O. A tile already loaded or previously found absent is not read
again during the same renderer run. The cache deliberately remains live for
the entire movie: final numerical-BBH renders commonly sweep across the full
sky, so tile eviction would trade the normal case for repeated filesystem I/O.
Each frame reports the number of requested/newly loaded/unavailable tiles,
newly loaded stars, and CSV-load wall time. The loader count is deliberately
independent of the splat worker count: start at the default and benchmark on
the target storage before raising it.
Warm-cache reference measurement, 2026-08-27, on the available partial
all-sky directory (18 CSV tiles, 12,917 stars, 128x72, 2-degree field):
| Loader workers | CSV load time |
| ---: | ---: |
| 1 | 5 ms |
| 2 | 3 ms |
| 4 | 2 ms |
| 8 | 3 ms |
This confirms the bounded-loader path and motivates the default of four; it is
not a cold-cache throughput claim for the final full catalog. Reproduce it
with the rotating observer fixture:
```sh
./build/minkowski_sky --all-sky-catalog assets/2mass/processed/all_sky \
--observer-track tests/data/minkowski_rotation_2s.csv \
--frames-dir output/imgs/catalog_prefetch_bench/w4 --frames-prefix frame \
--start-time 0 --duration 0 --fps 1 --width 128 --height 72 \
--coarse-cell-pixels 8 --fov-deg 2 --exposure 1e13 \
--catalog-load-workers 4
```
Missing tiles are skipped until the downloader has produced them. Fully
contained tiles skip per-star source-triangle containment; partial tiles retain
that exact check. For 2MASS apparent-solid-angle amplitudes, start with