GR-raytracing/assets/2mass/processed/all_sky/README.md

Full-sky 2MASS PSC output

This directory is generated, not versioned. Create it with:

python3 scripts/download_2mass_psc_all_sky.py --plan
python3 scripts/download_2mass_psc_all_sky.py --download --workers 2

The acquisition uses fixed final integer indices: ra_index = 0..359 and dec_index = 0..179. Each ownership cell is [ra_index, ra_index + 1) x [dec_index - 90, dec_index - 89) in ICRS degrees, except that dec_index=179 also owns Dec +90. To avoid polar over-fetch, each latitude band is acquired through wider integer-RA cover sectors. A cover owns a disjoint run of final tiles and its one-degree cone contains every point it owns; cone overlap is therefore discarded by ownership rather than a whole-sky de-duplication table. Raw responses and cleaning intermediates are in /tmp/2mass_psc_all_sky/ and are deleted after each successful cover. .done/ makes reruns resume after completed final tiles.

Downloads stay serial to be considerate of IRSA, while --workers N cleans up to N already-downloaded covers in parallel. The queue is bounded to N, so temporary raw/intermediate storage cannot grow without bound. Start with --workers 2; increase it only if CPU and /tmp headroom remain available.

Each atomic output CSV uses the renderer's four-column CSV format and has a name such as tile_ra129_dec109.csv, which means RA [129, 130) and Dec [19, 20) degrees. For any coordinate, use ra_index = floor(RA_deg mod 360) and dec_index = min(179, floor(Dec_deg + 90)) to select the filename directly. It applies the same three-band rd_flg/photometry selection and blackbody fit as the sample processor. The downloader fails if an IRSA response reaches --outrows, so crowded fields cannot be silently truncated. It deletes raw tile tables after successful cleaning unless --keep-raw is passed.

Renderer use

Use the directory mode rather than --catalog for this partitioned dataset:

make ENABLE_PNG=1 SPACETIME=minkowski all
./build/minkowski_sky --all-sky-catalog assets/2mass/processed/all_sky \
  --fov-deg 2 --look-ra-deg 180 --look-dec-deg -75 \
  --exposure 1e15 \
  --output output/imgs/2mass_all_sky_flat_2deg.png

The renderer lazily loads only tiles touched by source triangles in the frame. It first collects and de-duplicates the frame's touched tile IDs on the main thread. Newly needed CSVs are then read by a bounded OpenMP loader group (--catalog-load-workers N, default 4); each worker owns distinct tile files. The main thread commits their results to the cache only after that group finishes, so OpenMP splat workers always read immutable tile data and never race catalog I/O. A tile already loaded or previously found absent is not read again during the same renderer run. The cache deliberately remains live for the entire movie: final numerical-BBH renders commonly sweep across the full sky, so tile eviction would trade the normal case for repeated filesystem I/O.

Each frame reports the number of requested/newly loaded/unavailable tiles, newly loaded stars, and CSV-load wall time. The loader count is deliberately independent of the splat worker count: start at the default and benchmark on the target storage before raising it.

Warm-cache reference measurement, 2026-08-27, on the available partial all-sky directory (18 CSV tiles, 12,917 stars, 128x72, 2-degree field):

Loader workers	CSV load time
1	5 ms
2	3 ms
4	2 ms
8	3 ms

This confirms the bounded-loader path and motivates the default of four; it is not a cold-cache throughput claim for the final full catalog. Reproduce it with the rotating observer fixture:

./build/minkowski_sky --all-sky-catalog assets/2mass/processed/all_sky \
  --observer-track tests/data/minkowski_rotation_2s.csv \
  --frames-dir output/imgs/catalog_prefetch_bench/w4 --frames-prefix frame \
  --start-time 0 --duration 0 --fps 1 --width 128 --height 72 \
  --coarse-cell-pixels 8 --fov-deg 2 --exposure 1e13 \
  --catalog-load-workers 4

Missing tiles are skipped until the downloader has produced them. Fully contained tiles skip per-star source-triangle containment; partial tiles retain that exact check. For 2MASS apparent-solid-angle amplitudes, start with --exposure 1e15; use PNG (make ENABLE_PNG=1) for substantially smaller single-frame output than binary PPM.

The two existing fields imply roughly 15--17 GiB of cleaned CSV for the full PSC. Their raw response rows imply 70--75 GiB if every source appeared once. The cover grid uses substantially fewer cone areas than the old fixed-grid fetcher, but retaining every overlapping raw response would still need roughly 190--210 GiB. The default bounded pipeline only needs up to --workers raw responses and intermediates in addition to final output.