diff --git a/CLAUDE.md b/CLAUDE.md index 8ee90c2cc..cae6d99eb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -80,19 +80,36 @@ Full detail: `docs/source/installation.md` and `docs/source/container.md`. - `src/shapepipe/` — the package (src-layout). `modules/` holds the pipeline modules and their `*_runner.py` wrappers; `pipeline/` is execution and file - I/O; `utilities/`; `canfar/` is CANFAR/cluster job orchestration. Console entry - points (`shapepipe_run`, `summary_run`, `canfar_*`) are defined under + I/O; `utilities/`; `canfar/` monitors CANFAR sessions. Console entry + points (`shapepipe_run`, `canfar_*`, `plot_coverage_map`) are defined under `[project.scripts]`. - `tests/` — the whole test suite, one discovery root: `module/` (per-module unit/property/integration tests), `unit/` (structural), `science/` (fast guardrails), `cluster/` (candide-only), `helpers/` (shared library code). See `tests/README.md`. +- `workflow/` — **the production orchestration**: a Snakemake workflow over the + same `shapepipe_run` calls. `bin/sp` is the entry point (`run`, `report`, + `container`, `cancel`); `Snakefile` plus `rules/{prepare,exposure,tile, + coverage}.smk` are the rule graph; `scripts/` holds the plain Python each rule + shells out to; `config.yaml` declares one run (tile list, the scratch and + persistent roots, the container); `config/cfis/` holds the committed ini + chain; `profiles/nibi/config.yaml` is the SLURM executor profile. Deep + reference: `workflow/README.md`. User-facing: `docs/source/workflow.md`. - `example/` — a runnable example pipeline (`example/config.ini`) on a single CFIS tile; doubles as the CI smoke test. - `scripts/` — shell / Python / notebook helpers (`sh/`, `python/`, `jupyter/`), symlinked onto `$PATH` inside the image. - `docs/` — Sphinx sources; the API docs are generated from docstrings. +**There is no concept of a catalogue version in the code.** Nothing branches on +`v1.3`..`v1.6` or `v2.0`, and there are no sky patches (`P1`..`P9`): a campaign +is a tile list, and the version of a catalogue is the git tag of the code that +produced it. The CANFAR layers and the summary scrape that carried those +constructs were retired, last carried at `2ef07e45`. The pre-Snakemake bash job +layer (`scripts/sh/run_job_sp_canfar_v2.0.bash`, `job_sp_canfar_v2.0.bash`, +`job_list_help.bash`, `functions.sh`) stays — it is version-free, and +sp_validation's image-simulation workflow calls it. + ## Development workflow - **`develop` is the integration branch** — open PRs against it. `main` / diff --git a/Dockerfile b/Dockerfile index bd8a07d49..bbf63c334 100644 --- a/Dockerfile +++ b/Dockerfile @@ -118,6 +118,7 @@ RUN chmod -R go+rwX /app && \ uv pip install --no-deps -e . && \ for ext in .py .sh .bash; do \ for script in /app/scripts/*/*$ext; do \ + [ -e "$script" ] || continue; \ link_name=$(basename $script $ext); \ ln -s $script /usr/local/bin/$link_name; \ done; \ @@ -169,6 +170,7 @@ RUN chmod -R go+rwX /app && \ uv pip install --no-deps -e . && \ for ext in .py .sh .bash; do \ for script in /app/scripts/*/*$ext; do \ + [ -e "$script" ] || continue; \ link_name=$(basename $script $ext); \ ln -s $script /usr/local/bin/$link_name; \ done; \ diff --git a/README.rst b/README.rst index e27b4028d..01303fe4e 100644 --- a/README.rst +++ b/README.rst @@ -50,7 +50,8 @@ to start: - `Installation `_ — getting ShapePipe onto your machine or cluster. - `Basic execution `_ and `configuration `_ — running ``shapepipe_run`` and writing pipeline configs. - `Container workflow `_ — the two image targets and the ``pyproject.toml`` / ``uv.lock`` / ``Dockerfile`` layers. -- `Running on a cluster `_ — pulling the image and submitting jobs, with worked candide (SLURM) and CANFAR examples. +- `Running on a cluster `_ — pulling the image and submitting jobs, with a worked candide (SLURM) example. +- `The Snakemake workflow `_ — the production orchestration for a whole tile list, driven by ``workflow/bin/sp``. If you use ShapePipe in academic work, please cite Guinot et al. (2022) and Farrens et al. (2022). diff --git a/bin/canfar_submit_job.py b/bin/canfar_submit_job.py deleted file mode 100755 index 202e9d274..000000000 --- a/bin/canfar_submit_job.py +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env python -from shapepipe.canfar_run import run_job -run_job() diff --git a/bin/summary_run.py b/bin/summary_run.py deleted file mode 100755 index a59689e41..000000000 --- a/bin/summary_run.py +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env python -from shapepipe.summary_run import main -main() diff --git a/docs/source/clusters.md b/docs/source/clusters.md index 41d32a0bf..521e51bb3 100644 --- a/docs/source/clusters.md +++ b/docs/source/clusters.md @@ -2,9 +2,9 @@ ShapePipe runs the same way on every cluster: **through the container**. You pull the slim `runtime` image once, bind-mount your clone, and run -`shapepipe_run` (or the CANFAR submission tooling) inside it — there is no -environment to install or activate on the host. This page covers the shared -pattern, then the specifics for each supported machine. +`shapepipe_run` inside it — there is no environment to install or activate on +the host. This page covers the shared pattern, then the specifics for each +supported machine. For what is *inside* the image and how it is built, see [Container Workflow](container.md). @@ -67,32 +67,17 @@ over unchanged. ## CANFAR -CANFAR submission does not go through a batch scheduler. Instead you submit -container jobs to CANFAR's headless system with the `canfar_submit_job` console -script (backed by the `canfar` library), and watch them with `canfar_monitor` / -`canfar_monitor_log`. Pipeline steps are **bit-coded** through `-j` (the same -scheme as `scripts/sh/job_sp_canfar.bash`), the PSF model is chosen with -`-p psfex|mccd`, and `-V` selects the image version: +CANFAR submission has been retired. The `canfar_submit_job` console script and +the bash layers underneath it (`curl_canfar_local.sh`, +`init_run_exclusive_canfar.sh`, `job_sp_canfar.bash`) were last carried at +`2ef07e45`; the bit-coded `-j` production walkthrough went with them. -```bash -# Submit pipeline step(s) for the configured tiles (bit-coded -j). -canfar_submit_job -j 1 -p psfex -V 1.1 - -# Monitor sessions/jobs and stream logs. -canfar_monitor -canfar_monitor_log -``` +Production orchestration is now the Snakemake workflow in `workflow/` — see +`workflow/README.md`. It is SLURM-only and has no CANFAR execution mode, so a +CANFAR campaign means moving to a SLURM site (nibi, candide). -The full production run — input preparation, the per-step `-j` table, and -post-processing — is documented in the -[CANFAR production walkthrough](pipeline_canfar.md). - -```{note} -The CANFAR production submission scripts (`scripts/sh/job_sp_canfar*.bash`) still -run under the pre-container environment and are slated for the same -container-first cleanup the candide scripts received. Treat the walkthrough as -the current-but-evolving production procedure. -``` +`canfar_monitor` and `canfar_monitor_log` stay: they list, filter and destroy +CANFAR sessions, independently of how those sessions were started. ## ccin2p3 diff --git a/docs/source/conf.py b/docs/source/conf.py index eea9b8f43..d52836d43 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -104,6 +104,10 @@ myst_enable_extensions = ["html_image"] +# Generate slug anchors for h1-h3 so in-page links like +# `[Mask images](#mask-images)` resolve instead of warning. +myst_heading_anchors = 3 + # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for diff --git a/docs/source/dependencies.md b/docs/source/dependencies.md index b907a452d..e8722584f 100644 --- a/docs/source/dependencies.md +++ b/docs/source/dependencies.md @@ -35,8 +35,7 @@ Data access & infrastructure (CANFAR / UNIONS): | Package | Purpose | |---------|---------| | [vos](https://github.com/opencadc/vostools) | CADC / CANFAR VOSpace access | -| [skaha](https://github.com/shinybrar/skaha) | CANFAR Science Platform sessions | -| canfar | CANFAR container-job submission | +| canfar | CANFAR session monitoring | | [astroquery](https://astroquery.readthedocs.io/) | external catalogue queries | | [cs_util](https://github.com/CosmoStat/cs_util) | shared CosmoStat utilities | | [sqlitedict](https://github.com/RaRe-Technologies/sqlitedict) | on-disk pipeline state | diff --git a/docs/source/pipeline_canfar.md b/docs/source/pipeline_canfar.md deleted file mode 100644 index 41e2c1301..000000000 --- a/docs/source/pipeline_canfar.md +++ /dev/null @@ -1,545 +0,0 @@ -# Running `ShapePipe` processing and post-processing pipelines on CANFAR - -Documentation to create ShapePipe output products for catalogues v1.x. - -## Initial Setup - -### CANFAR Login - -Login to the canfar system with - -```bash -canfar auth login -``` - -This can be done from at notebook or terminal within the canfar science portal, -or any remote terminal that has the canfar library installed. - -Check authentication status with - -```bash -canfar auth list -``` - -If not on "default", run - -```bash -canfar auth switch default -``` - -### Set variables (optional) - -Set the current patch in the shell as - -```bash -patch=P[1-9] -``` - -For convenience, the current PSF model can be set as environment variable, e.g.: - -```bash -psf="psfex" -``` - -Allowed are `psfex` and `mccd`. - -Setting the terminal title to display the patch can be useful for long jobs, to keep track of which terminal -runs which patch: - -```bash -echo -ne "\033]0;$patch\007" -``` - -### Prepare run directory - -First, go to the dedicated directory with - -```bash -cd /path/to/version/$patch -``` - -Next, set links to the tile number list and configuration directory: - -```bash -ln -s ~/shapepipe/auxdir/CFIS/tiles_202106/tiles_$patch.txt tile_numbers.txt -ln -s ~/shapepipe/example/cfis -``` - -Create output and debug log directories - -```bash -mkdir -p output -mkdir -p debug -``` - -Finally, create and link to central image storage directories for tiles and exposures: - -```bash -mkdir -p ~/cosmostat/v2/data_tiles/$patch -ln -s ~/cosmostat/v2/data_tiles/$patch data_tiles -mkdir -p ~/cosmostat/v2/data_exp/$patch -ln -s ~/cosmostat/v2/data_tiles/$patch data_exp -``` - -## `ShapePipe` processing - -Now, everything should be ready to start running `ShapePipe` for the weak lensing processing. The following -details all necessary steps. - -### Get Images - -We first download images, and in a second run create symbolic links with the proper pipeine naming scheme. - -#### Download and move tiles - -When running the main `ShapePipe` script `shapepipe_run`, the following env variable needs to point -to the current working directory - -```bash -export SP_RUN=`pwd` -``` - -Now we run the first module (`get_images_runner`) to download the tile images together with the weight files. -This run can get interrupted by VOSpace I/O or connection errors. In that case, -we move new files to the image storage directory, remove the previous (now void of images) run directory, -and update the run log file. We also check the number of previous and new tiles. - -```bash -shapepipe_run -c cfis/config_Git_vos.ini -ls -l data_tiles/ | wc -mv -i output/run_sp_Git_*/get_images_runner/output/CFIS.???.???.*fits* data_tiles -ls -l data_tiles/ | wc -rm -rf output/run_sp_Git_* -update_runs_log_file.py -``` - -Repeat the above block as needed. - -### Find Exposures - -With all tile images (= stacks) downloaded, we can inquire their headers to identify the exposures that were used -to create the stacks. This call to the pipeline also creates the symbolic links to the downloaded tile images. - -```bash -shapepipe_run -c cfis/config_GitFe_symlink.ini -``` - -(One could also run `Fe` alone.) - -### Download and Move Exposures - -The last module create exposure lists on output. These are now used to download all exposures. As for the tile downloads, -we have to account for VOSpace errors. - -```bash -shapepipe_run -c cfis/config_Gie_vos.ini -mv -i output/run_sp_Gie_*/get_images_runner/output/*.fits*fz data_exp -rm -rf output/run_sp_Gie_* -update_runs_log_file.py -``` - -Repeat the above by hand, or peform it in an automatic loop: - -```bash -while true; do - shapepipe_run -c cfis/config_Gie_vos.ini - ls -l data_exp/ | wc - mv -i output/run_sp_Gie_*/get_images_runner/output/*.fits*fz data_exp - ls -l data_exp/ | wc - rm -rf output/run_sp_Gie_* - update_runs_log_file.py -done -``` - -**Note:** Make sure that after all images are downloaded there is no `Gie` run in the output directory. -This would mess up later modules since `last:get_image_runner` could point to this run. - -### Create tile links again (necessary?) - -If necessary, e.g. because a previous `Git` run is no longer valid, re-create the symbolic links to the downloaded tiles with - -```bash -job_sp_canfar.bash -p $psf `cat tile_numbers.txt` -j 1 -r symlink -``` - -### Uncompress tile weights - -The downloaded tile weights are compressed. The following call uncompresses all. - -```bash -shapepipe_run -c cfis/config_tile_Uz.ini -``` - -### Masks - -There is no masking step. `ShapePipe` generates no masks: the sky-fixed -healsparse maps are queried once per object, by `mask_query` on the exposure -catalogues (`MASK_EXT`, recorded but not cut on) and by `make_cat` on the tile -catalogue (`MASK_` columns). Point the `MASK_PATHS` / `MASK_EXT_PATHS` -config entries at the maps and nothing else is needed — no star-catalogue -download, no rasterization, no `combine_runs.bash -c flag_*`. The only mask that -touches pixels is the instrument flag image shipped with each exposure, which -`split_exp` splits per CCD. - -## Tile detection - -We can finally run our first module using the canfar submission system. First, determine the number of optimal jobs such that at a given -time the allowed maximum of 512 running jobs is not exceeded. -Set as `N_PAR` (number of parallel jobs) a number between 1 and 8. - -```bash -canfar_submit_job -j 16 -f tile_numbers.txt -P N_PAR -v -s -``` - -Now, run the previous command with that number `JMAX` - -```bash -canfar_submit_job -j 16 -f tile_numbers.txt -P N_PAR -v -J JMAX -``` - -### Exposure Processing - -#### Option 0: Global split (deprecated; used for earlier v1.x patch runs) - -For this option, set `sp_local=0`. - -**TODO: ** Split Uz and SpMh - -For `sp_local=-` both `mh_local` (0, 1) are ok: - -```bash -export mh_local=0 -``` - -### Option 1: Local split exposures (recommended) - -Optional: Enable flags for local split processing and merge header runs as - -```bash -export sp_local=1 -export mh_local=1 -``` - -These flags are automatically set to 1 in the new job scripts. - - -Get single-HDU single-exposure IDs file (from missing 32 job): - -```bash -summary_run P$patch 32 -cp summary/missing_job_32_all.txt exp_shdu.txt -``` - -### Split exposures - -First, determine the number of maximum jobs with the option `-s` (see above). Then, submit with - -```bash -canfar_submit_job -j 2 -v -f exp_shdu.txt -v -P N_PAR -J JMAX -``` - -### Exposure detection - -```bash -canfar_submit_job -j 32 -f exp_shdu.txt -v -P N_PAR -J JMAX -``` - -### Tile preparation - -```bash -canfar_submit_job -j 64 -f tile_numbers.txt -``` - -### Tile shape measurement - -```bash -canfar_submit_job -j 128 -f tile_numbers.txt -``` - -### Merge sub-catalogues - -```bash -canfar_submit_job -j 256 -f tile_numbers.txt -``` - -### Create final catalogues - -```bash -canfar_submit_job -j 512 -f tile_numbers.txt -``` - -This was the last `ShapePipe` module to run for main processing. - - -### Merge all final catalogues - -The last step of `ShapePipe` processing is, per patch, to merget all final catalogues. This is done via a python script, as follows. -First, change to parent directory `/path/to/version` and run the following command for all patches - -```bash -patchnum=`tr $patch P ''` -create_final_cat.py -m final_cat_$patch.hdf5 -i . -p $patch/cfis/final_cat.param \ - -P $patchnum -o $patch/n_tiles_final.txt -v -``` - -## Additional `ShapePipe` processing - - -### Create star Catalogue - -We can additionaly create a combined star catalogue, with star shapes projecte from detector to world coordinates. -This is useful for validation and galaxy-PSF/star correlation diagnostics. - -#### Combine all PSF runs - -In each patch directory /path/to/version/$patch, run - -```bash -combine_runs.bash -p $psf -c psf -``` - -to create a single output directory of PSF files (symbolic links). - -Optionally, to create and plot results for this patch only: - -```bash -shapepipe_run -c $SP_CONFIG/config_Ms_$psf.ini -shapepipe_run -c $SP_CONFIG/config_Pl_$psf.ini -``` - -#### Collate star catalogues - -Collate all input validation PSF files into star catalogues, gathering positions -(X/Y/RA/DEC) and the MCCD CCD id. - -Note: HSM shapes are no longer rotated into world coordinates at this step. The -PSF/star ellipticities and sizes are now measured directly in sky coordinates -during PSF interpolation (galsim `FindAdaptiveMom(use_sky_coords=True)`), so -`collate_star_cat.py` only collates and passes the shapes through. - -> **v2.0 is patch-less.** Runs up to `v1.6` are organised in sky patches -> `P1`..`P`, each patch a run directory. `v2.0` removes the patch concept: a -> single run root, outputs directly under it. `v2.0` is the default; select an -> older layout with `-V` (e.g. `-V v1.6`). For `v2.0` the patch loop and the -> `-P` option no longer apply, and the patch token drops from the output -> filename (`validation_psf_conv-.fits` instead of -> `validation_psf_conv--.fits`). - -```bash -cd /path/to/version -mkdir star_cat -cd star_cat -``` - -For `v2.0` (the default), run once against the patch-less run root, producing -files `validation_psf_conv-.fits`: - -```bash -collate_star_cat.py -i .. -v -``` - -For `v1.x`, pass the version explicitly and run once per patch, creating a -directory per patch `P?` and producing files -`validation_psf_conv--.fits` (for the v1.4 setup only one file): - -```bash -collate_star_cat.py -i .. -V v1.6 -P $patchnum -v -``` - -Combine previously created files as links within one ShapePipe run directory (for the v1.4 setup only one link). -First (and optiohnal), create a subdir for a run and link to the input patches: - -```bash -cd /path/to/version/star_cat -mkdir v1.6 -ln -s ../P1 -ln -s ../P2 -... -``` - -Next, create links to all `validation_conv` runs: - -```bash -combine_runs.bash -p psfex -c psf_conv -``` - -Merge all converted star catalogues and create `final-starcat.fits`: - -```bash -export SP_RUN=`pwd` -shapepipe_run -c ~/shapepipe/example/cfis/config_Ms_psfex_conv.ini -``` - -Rename to general PSF and star catalogue used for all ("a") sub-versions: - - -```bash -cp output/run_sp_tile_Ms/merge_starcat_runner/output/full_starcat-0000000.fits \ - unions_shapepipe_psf_2024_v1.6.a.fits -``` - -The FITS file `CATTYPE` (newer version) should be `validation_psf_conf`. - -## Post-processing - -The following post-processing steps are performed with the library `sp_validation`. - -### Extract Information - -First, we extract all information from the final catalogue, per patch. We copy -the parameter file and set links to the catalogues and `ShapePipe` config directory. - -```bash -cd /path/to/version/$patch -cp ~/astro/repositories/github/sp_validation/notebooks/params.py . -ln -s /path/to/final_cat_$patchnum.hdf5 # not relative path ../final_cat_P$patchnum.hdf5 ! -ln -s output/run_sp_MsPl/mccd_merge_starcat_runner/output/full_starcat-0000000.fits -ln -s ~/astro/repositories/github/shapepipe/example/cfis -``` - -Then edit `params.py`: Set patch name; set `wrap_ra` for P2. - -Now we can run the script, recommended via job submission on candide. For large patches, -this requies a job with a large memory, e.g. with `mem=380000` - - -```bash -[squeue] python ~/astro/repositories/github/sp_validation/notebooks/extract_info.py -``` - -This creates a patch-wise comprehensive catalogue. - -### Create global comprehensive catalogues - -```bash -cd /patch/to/version -[squeue] python ~/astro/repositories/github/sp_validation/scripts/create_joint_comprehensive_cat.py \ - -v v1.6.c -v -p P1+P2+P3+P4+P5+P6+P7+P8+P9 -``` - -This creates the file `unions_shapepipe_comprehensive_2024_v1.6.c.hdf5`. - - -### Apply structural masks - -First, edit the Python script `~/astro/repositories/github/sp_validation/notebooks/demo_apply_hsp_masks.py` -to match catalogue name. Check the coverage mask input file (see below). -Run the script to apply the healsparse structural masks: - -```bash -[squeue] python ~/astro/repositories/github/sp_validation/notebooks/demo_apply_hsp_masks.py -``` - -This creates the file `unions_shapepipe_comprehensive_struct_2024_v1.6.c.hdf5`. - - -### Define sample, calibrate catalogue - -We are close to finally perform the last post-processing step, which is the calibration. First, the final galaxy sample -in question needs to be defined, with masks and cuts to apply from a `yaml` config file. A number of pre-defined files -can be found in `~/astro/repositories/github/sp_validation/calibration`. - -For example, to create `v1.6.6`, the steps are: - -```bash -cd /path/to/version -mkdir -p v1.6.6 -cd v1.6.6 -ln -s ~/astro/repositories/github/sp_validation/calibration/mask_v1.X.6.yaml config_mask.yaml -ln -s ..//unions_shapepipe_comprehensive_struct_2024_v1.6.c.hdf5 unions_shapepipe_comprehensive_struct_2024_v1.X.c.hdf5 -[squeue] python ~/astro/repositories/github/sp_validation/calibrate_comprehensive_cat.py -``` - -calibrate_comprehensive - - -```{note} -The matched-star-catalogue diagnostic (`merge_psf_cat.py`) has moved into -[`sp_validation`](https://github.com/CosmoStat/sp_validation) / `cosmo_val`; -see that repository for its current equivalent. The coverage-mask helpers -(`get_ccds_with_psf`, `download_headers`, `extract_field_corners`, -`build_coverage_map`, `plot_coverage_map`, and `summary_run`) are shipped by -ShapePipe and documented below. -``` - -### Create matched star catalogue - -For diagnostics, a catalogue with multi-epoch shapes measured by ngmix matched with the validation star catalogue is used. -This is created as follows: - -```bash -cd /path/to/version -merge_psf_cat.py [-V v1.6|-P P1+P2+...] -v -``` - -This creates the joint catalogue unions_shapepipe_star_2024_v1.6.a.fits . - -### Create coverage mask - -First, on canfar, move to the directory that has the patch subdirectories. - -```bash -cd /path/to/version -``` - -#### Get exposure numbers - -If the file `$patch/exp_numbers.txt` does not exist for a given patch, create it with the summary program - -```bash -summary_run $patch 1 -``` - -Now, create the list of single-CCD footprints that have a valid PSF model. -Only these CCDs are stamped into the mask, so the accumulated map counts, per -sky pixel, the number of exposures with a valid PSF. The list is written to -`ccds_with_psf_v1.6.txt`: - -```bash -get_ccds_with_psf -v -V v1.6 -o ccds_with_psf_v1.6.txt -``` - -Next, download the exposure headers; indicate (with `-d`) a directory of -already downloaded headers, which are linked so duplicate downloads are -skipped: - -```bash -download_headers -i ccds_with_psf_v1.6.txt -o headers_v1.6 -d headers_v1.3 -v -``` - -From the headers, the per-CCD corner coordinates are extracted. Pass the -CCD list with `-l` so only CCDs with a valid PSF are written: - -```bash -extract_field_corners -i headers_v1.6 -l ccds_with_psf_v1.6.txt -o exp_ra_dec_v1.6.txt -v -``` - -Then, build the healsparse coverage mask file as -```bash -build_coverage_map -i exp_ra_dec_v1.6.txt -o coverage_v1.6.x.hsp -c 128 -n 131072 -v -``` - -The healsparse resolutions (128, 131072) match the bit masks. - - -Use `plot_coverage_map` to plot a region of the coverage mask. For example, -the SGC region with a colorbar clipped to the 1–5 exposure range: - -```bash -plot_coverage_map -i coverage_v1.6.x.hsp -o coverage_v1.6_SGC.png \ - -C -R -20 -r 45 -D 18 -d 40 -m 1 -M 5 -v -``` - -Here `-R`/`-r` and `-D`/`-d` set the RA and Dec plot limits, `-m`/`-M` the -colorbar range, and `-C` adds the colorbar. Building and plotting for a range -of versions (SGC and NGC regions) is automated by -`build_and_plot_coverage_maps.sh`. - - -## Extra Utilities - -### Run in Terminal in Parallel - -```bash -cat IDs.txt | xargs -I {} -P 16 bash -c 'init_run_exclusive_canfar.sh -j 512 -e {}' -``` diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index b7ae7985c..d7add0d91 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -2,13 +2,17 @@ ## Quick start -Run the entire pipeline on a single example CFIS image with tile ID 246.290: -1. [Install](installation.md) `ShapePipe` — the recommended path is the container image, which bundles everything needed to run the pipeline. -3. Run the job script +Production runs go through the [Snakemake workflow](workflow.md). Put the tile +IDs in the file named by `tile_list` in `workflow/config.yaml`, one +`IDra.IDdec` per line, and run + ```bash -job_sp 246.290 -j 127 +workflow/bin/sp run ``` +A single tile is a legal tile list and is the smallest useful end-to-end run. +The sections below describe what that produces, stage by stage. + ## Introduction The `ShapePipe` pipeline processes single-exposure images and stacked images. Input images have to be calibrated beforehand for astrometry and photometry. This tutorial of an entire `ShapePipe` run covers specifically images from CFIS, the Canada-France Imaging Survey. CFIS stacks are so-called tiles, which are the co-adds of on average three exposures in the r-band. @@ -69,21 +73,21 @@ Naming and numbering of the input files can closely follow the original image na sheared images. This information is used in post-processing to compute calibrated shear estimates via metacalibration. - Summary statistic files - The `SETools` module that creates samples of objects according to some user-defined selection criteria (see [Select stars](#select-stars)) also outputs ASCII + The `SETools` module that creates samples of objects according to some user-defined selection criteria (see [Detect objects](#detect-objects-on-tiles-and-process-stars-on-single-exposures)) also outputs ASCII files with user-defined summary statistics for each CCD, for example the number of selected stars, or mean and standard deviation of their FWHM. Example: `star_stat-2366993-18.txt` - Tile ID list - ASCII file with a tile number on each line. Used for the `get_image_runner` module to download CFIS images (see [Download tiles](#download-tiles)). + ASCII file with a tile number on each line. Used for the `get_image_runner` module to download CFIS images (see [Retrieve input images](#retrieve-input-images)). - Single-exposure name list ASCII file with a single-exposure name on each line. Produced by the `find_exposure_runner` module to identify single exposures that were used to create - a given tile. See [Find exposures](#find-exposures)). + a given tile. See [Retrieve input images](#retrieve-input-images)). - Plots The `SETools` module can also produce plots of the objects properties that were selected for a given CCD. The type of plot (histogram, scatter plot, ...) and quantities to plot as well as plot decorations can be specified in the - selection criteria config file (see [Select stars](#select-stars)). + selection criteria config file (see [Detect objects](#detect-objects-on-tiles-and-process-stars-on-single-exposures)). Example: `hist_mag_stars-2104133-5.png` - Log files @@ -94,7 +98,7 @@ Naming and numbering of the input files can closely follow the original image na `ShapePipe` splits the processing of CFIS images into several parts: These are the retrieval and preparation of input images, processing of single exposures, -processing of tile images, creation and upload (optional) of _final_ shape catalogues. +processing of tile images, and creation of _final_ shape catalogues. The following flowchart visualised the processing parts and steps. @@ -104,44 +108,58 @@ Below, the individual processing steps are described in detail. ### Input and output paths -All required paths are automatically set in the job script `job_sp`. +The workflow's rules export every path a config interpolates before each +`shapepipe_run` call, so nothing here needs setting by hand for a normal run. -If an example config file is run outside this script, -the following path variables might need to be defined. +If a config file is run outside the workflow, the following variables need to be +defined. - `$SP_RUN`: Run directory of `ShapePipe`. In general this is just `pwd`, and can be set via ```bash export SP_RUN=`pwd` ``` but on a cluster this directory might be different. - `$SP_CONFIG`: Path to configuration files. In our example this is `$SP_BASE/example/cfis`. +- `$SP_UNIT_NUM`: the tile or exposure this run is restricted to, in ShapePipe's + image-number convention — a leading dash, and dots replaced by dashes (tile + `210.282` becomes `-210-282`, exposure `2605805` becomes `-2605805`). The + configs interpolate it into `NUMBER_LIST`. In addition, the output path `$SP_RUN/output` needs to be created by the user before running `ShapePipe`. -### Job and pipeline scripts +### The rule chain -The job script to run the pipeline in its entity or in parts is `job_sp[.bash]`. Type -```bash -job_sp -h -``` -for all options. +There is no monolithic job script. Every processing step below is a **rule**, and +every rule instance is one `shapepipe_run` call on one unit — one tile, or one +single exposure — inside that unit's own directory: -This script creates the subdirectory `$SP_RUN/output` to store all pipeline outputs -(log files, diagnostics, statistics, output images, catalogues, single-exposure headers with WCS information). +| level | rule chain | +| --- | --- | +| tile, prepare phase | `tile_get_images` → `tile_uncompress` → `tile_find_exposures` | +| exposure | `exp_get_images` → `exp_split` → `exp_psf` → `exp_persist` → `exp_footprint` → `clean_exposure` | +| tile, compute phase | `tile_exp_forest` → `tile_merge_headers` → `tile_detect` → `tile_vignets` → `tile_ngmix` → `tile_merge_cats` → `tile_make_cat` → `clean_tile` | + +Each rule calls -The job script automaticall performs a number of subsequent calls to the `ShapePipe` executable `shapepipe_run`, as ```bash shapepipe_run -c $SP_CONFIG/.ini ``` -The config file `.ini` contains the configuration for one or more modules. -See the main `ShapePipe` readme for more details. -The user specifies which steps are run with the command line option `-j JOB`. The integer value `JOB` -is bit-coded such that arbitrary combinations of steps can be run with a single call to `job_sp`. For -example, to run steps #1 and #2, type `job_sp -j 3`. +on a config file committed under `workflow/config/cfis/`, containing the +configuration for one or more modules. See [Configuration](configuration.md) for +what goes in one. To run a single rule and its prerequisites and nothing else, +name it as a snakemake target: `sp exp_psf ...`. + +Each unit directory holds an `output/` subdirectory with all pipeline outputs +(log files, diagnostics, statistics, output images, catalogues, single-exposure +headers with WCS information), beside the `manifests/` and `logs/` that are the +workflow's own bookkeeping. ### Select tiles -To run the job script, one or more CFIS tiles need to be chosen. If the tile IDs are known, they are provided to `job_sp` on the command line. +The tiles to process are the lines of the tile list. The campaign grows by +appending to that file: the tile-to-exposure index accumulates across +invocations, so adding tiles later changes which jobs exist without invalidating +completed work. If the tile IDs are not known a priori, they can be selected via sky coordinates, with the script `cfis_field_select`. For example, to find the tile number for a Planck cluster at R.A.=213.68 deg, dec=57.79 deg, run: @@ -151,41 +169,41 @@ cfis_field_select -i /path/to/shapepipe/auxdir/CFIS/tiles_202007/tiles_all_order The input text file (provide via the flag `-i`) contains a list of CFIS tiles, this can also be directory containing the tile FITS files. -The following sections describe the different steps that are performed with `job_sp`. +The following sections describe the steps the rule chain performs. ## Run the pipeline ### Retrieve input images -The command -```bash -job_sp TILE_ID -j 1 -``` -retrieves the image and weight corresponding to TILE_ID using the module `get_images`. -It then identifies the exposures that were used to create the tile image via the `find_exposures` runner. -Finally, another call to `get_images` retrieves the exposure images, weights, and flag files. +`tile_get_images` retrieves the image and weight corresponding to a tile ID +using the module `get_images`. `tile_find_exposures` then identifies the +exposures that were used to create the tile image via the `find_exposures` +runner, and `exp_get_images` retrieves each of those exposures' image, weight +and flag files. -For the retrieval method the user can choose betwen -- download from VOspace (`-r vos`); -- create symbolic link to existing file on disk (`-r symlink`). +For the retrieval method the user can choose between +- download from VOspace (`RETRIEVE = vos` in the config); +- create symbolic link to existing file on disk (`RETRIEVE = symlink`). Note that internet access is required for this step if the download method is `vos`. -An output directory `run_sp_GitFeGie` (in `output`) is created containing the results of `get_images` for tiles (`Git`), -`find_exposures` (`Fe`), and `get_images` for exposures (`Gie`). +These three rules are the workflow's PREPARE phase. The exposure half of the run +cannot be scheduled before them, because which exposures exist is something +`find_exposures` has to say first — which is why one run is two snakemake +invocations over one Snakefile (see [the workflow page](workflow.md)). ## Prepare input images -With -```bash -job_sp TILE_ID -j 2 -``` -the compressed tile weight image is uncompressed via the `uncompress_fits` module. Then, the single-exposure images, weight, and flags are split into single-exposure single-CCD file -(one FITS file per CCD) with `split_exp`. -Finally, the headers of all single-exposure single-CCD files are merged into a single `sqlite` file, to store the WCS information of the input exposures. +`tile_uncompress` uncompresses the compressed tile weight image via the +`uncompress_fits` module. + +`exp_split` splits each single-exposure image, weight and flag into +single-exposure single-CCD files (one FITS file per CCD) with `split_exp`, and +writes that exposure's WCS information to `headers-.npy`. -Two output directories are created, `run_sp_Uz` for `uncompress_fits`, and `run_sp_exp_SpMh` for the output of the modules -`split_exp` (`Sp`) and `merge_headers` (`Mh`). +`tile_merge_headers` then merges the headers of the exposures overlapping one +tile into a single `sqlite` file, so the tile-side modules can look up the WCS of +any of them. ## Masks @@ -233,23 +251,23 @@ with `FLAG_IMAGE = False`. ## Detect objects on tiles and process stars on single exposures -The call -```bash -job_sp TILE_ID -j 8 -``` -performs a number of steps. First, objects on the tiles are deteced with the `sextractor` runner. -Next, the following tasks are run on the single-exposure single-CCD images: -- Objects are deteced with `sextractor`. +`tile_detect` detects objects on the tile with the `sextractor` runner. + +`exp_psf` runs the single-exposure single-CCD chain: +- Objects are detected with `sextractor`. - Star candidates are selected via `setools`. -- The PSF model is created, either with `psfex` for PSFex, or - with `mccd_preprocessing` and `mccd_fit_val` for MCCD. -- The PSF model is interpolated to star positions for validation. For the PSFEx model, this is done - via a call to `psfex_interp`. For MCCD, the modules `merge_starcat`, `mccd_plots`, and - `mccd_interp` are called. +- The PSF model is created with `psfex`. +- The PSF model is interpolated to star positions for validation, via a call to + `psfex_interp`. -The output directory for both the `mccd` and `psfex` options is `run_sp_tile_Sx_exp_SxSePsf`. -This stores the output of SExtractor on the tiles (`tile_Sx`), on the exposures (`exp_Sx`), -`setools` (`Se`), and the Psf model (`Psf`). +The output directory is `run_sp_exp_SxSePsfPi`, holding the output of SExtractor +on the exposures (`Sx`), `setools` (`Se`), the PSF model (`Psf`) and the +validation interpolation (`Pi`). + +`exp_persist` then copies the products named by `persist_exp:` in +`workflow/config.yaml` — by default the `validation_psf-*.fits` that the rho and +tau statistics are computed from — off scratch onto the persistent root, before +the purge or `clean_exposure` can take them. The following plots show an example of a single CCD, in the center of the focal plane. @@ -297,60 +315,40 @@ several runs. ## Galaxy selection -The focus of the next step, -```bash -job_sp TILE_ID -j 16 -``` -is the selection of galaxies as extended objects compared to the PSF. -First, the PSF model is interpolated to galaxy positions, according to the PSF model -with `psfex_interp` or `mccd_interp`. Next, postage stamps around galaxies -of the weights maps are created via `vignetmaker`. Then, the spread model -is computed by the `spread_model` module. Finally, postage stamps -around galaxies of single-exposure data is extracted with another call -to `vignetmaker`. - -The output directory is -- `run_sp_MiViSmVi` if the PSF model is `mccd`; -- `run_sp_tile_PsViSmVi` for the `PSFEx` PSF model. +`tile_vignets` selects galaxies as extended objects compared to the PSF. +First, the PSF model is interpolated to galaxy positions with `psfex_interp`. +Next, postage stamps around galaxies of the weight maps are created via +`vignetmaker`. Then the spread model is computed by the `spread_model` module. +Finally, postage stamps around galaxies of single-exposure data are extracted +with another call to `vignetmaker` — `Pi`, `Vi`, `Sm`, `Vi`, hence the config +name `config_tile_PiViVi.ini`. -This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`), `spread_model` (`Sm`), and the -second call to `vignetmaker` (`Vi`). +`tile_vignets` reads the exposure products through the symlink forest that +`tile_exp_forest` built for this tile, and is the last stage that touches them: +once every consuming tile has its vignets, an exposure store becomes +reclaimable. ## Shape measurement -The call -```bash -job_sp TILE_ID -j 32 -``` -computes galaxy shapes using the multi-epoch model-fitting method `ngmix`. At the same time, -shapes of artifically sheared galaxies are obtained for metacalibration. +`tile_ngmix` computes galaxy shapes using the multi-epoch model-fitting method +`ngmix`. At the same time, shapes of artificially sheared galaxies are obtained +for metacalibration. -Shape measurement is performed in parallel for each tile, the number of processes can be specified -by the user with the option `--nsh_jobs NJOB`. This creates `NJOB` output directories `run_sp_tile_ngmix_Ngu`. -with `X` = 1 ... `NJOB` containing the result of `ngmix`. +Shape measurement is split into parallel chunks: `ngmix_chunks:` in +`workflow/config.yaml` sets how many, and each chunk is its own rule instance +writing its own `run_sp_tile_ngmix_Ngu` output directory. ## Paste catalogues -The last real processing step is -```bash -job_sp TILE_ID -j 64 -``` -This task first merges the `NJOB` parallel `ngmix` output files from the previous step into -one output file. Then, previously obtained information are pasted into a _final_ shape catalogue via `make_cat`. -Included are galaxy detection and basic measurement parameters, the PSF model at -galaxy positions, the spread-model classification, and the shape measurement. - -Two output directories are created. -The first one is `run_sp_tile_Ms` for the `merge_sep` run. -The second is `run_sp_tile_Mc` for the `make_cat` task; the name is the same for both the `MCCD` and `PSFEx` PSF model. - - -## Upload results - -Optionally, after the pipeline is finished, results can be uploaded to VOspace via -```bash -job_sp TILE_ID -j 128 -``` +`tile_merge_cats` merges the parallel `ngmix` output files from the previous +step into one file with `merge_sep`, in `run_sp_Ms`. +`tile_make_cat` then pastes the previously obtained information into a _final_ +shape catalogue via `make_cat`, in `run_sp_Mc`. Included are galaxy detection +and basic measurement parameters, the PSF model at galaxy positions, the +spread-model classification, and the shape measurement. The rule publishes that +catalogue onto the persistent root as +`/tiles///final_cat-.fits` — the campaign's +product, and also the marker that says this tile is finished. diff --git a/docs/source/post_processing.md b/docs/source/post_processing.md deleted file mode 100644 index f6af7f8e2..000000000 --- a/docs/source/post_processing.md +++ /dev/null @@ -1,76 +0,0 @@ -# Post-processing - -This page shows all required steps of post-processing the results from one or -more `ShapePipe` runs. Post-processing combines various individual `ShapePipe` -output files, and creates joint results, for example combining individual tile -catalogues into a large sky area. The output of post-processing is a joint _shape -catalogue_, containing all required information to create a calibrated shear -catalogue via _metacalibration_), a joint star catalogue, and PSF diagnostic plots. - - - ---- - -```{note} -This page documents the **legacy** post-processing used for pre-v1.4 runs on the -canfar VM system. PSF validation and the scale-dependent diagnostics -(ρ-statistics) now live in -[`sp_validation`](https://github.com/CosmoStat/sp_validation) rather than in -ShapePipe; the steps below are retained for reference and for reproducing older -runs. -``` - -The following steps were used for pre-v1.4 runs performed on the canfar VM system. - -1. Optional: Split output into sub-samples - - An optional intermediate step is to create directories for sub-samples, for example one directory - for each patch on the sky. This will create symbolic links to the results `.tgz` files downloaded in - the previous step. For example, to create the subdir `tiles_W3` with links to result files to `all` for - those tiles contained in the list `tiles_W3.txt`, do: - ```bash - create_sample_results --input_IDs tiles_W3.txt -i . all -o tiles_W3 -v - ``` - The following steps will then be done in the directory `tiles_W3`. - -2. Run PSF diagnostics, create merged catalogue - - Type - ```bash - post_proc_sp -p PSF - ``` - to automatically perform a number of post-processing steps. Choose the PSF model with the option - `-p psfex|mccd`. In detail, these are (and can also be done individually - by hand): - - 1. Analyse psf validation files - - ```bash - combine_runs -t psf -p PSF - ``` - with options as for `post_proc_sp`. - This script creates a new combined psf run in the ShapePipe `output` directory, by identifying all psf validation files - and creating symbolic links. The run log file is updated. - - 3. Merge the individual PSF validation files into one catalogue, and create - plots of the PSF and its residuals in the focal plane as a diagnostic of - the overall PSF model: - ```bash - shapepipe_run -c /path/to/shapepipe/example/cfis/config_MsPl_PSF.ini - ``` - The scale-dependent PSF diagnostics that propagate to the shear correlation - function (the ρ-statistics, {cite:p}`rowe:10`, {cite:p}`jarvis:16`) are no - longer computed here — they have moved to - [`sp_validation`](https://github.com/CosmoStat/sp_validation). - - 4. Merge final output files - - Create a single main shape catalogue: - ```bash - merge_final_cat -i -p -v - ``` - Choose as input directory `input_dir` the `make_cat` output of the runs - being combined. A default parameter file `` is - `/path/to/shapepipe/workflow/config/cfis/final_cat.param`. - On success, the file `./final_cat.npy` is created. Depending on the number of - input tiles, this file can be several tens of Gb large. diff --git a/docs/source/toc.rst b/docs/source/toc.rst index 01c5a1596..5e4113728 100644 --- a/docs/source/toc.rst +++ b/docs/source/toc.rst @@ -34,14 +34,7 @@ :caption: Running on a cluster clusters - pipeline_canfar - -.. toctree:: - :hidden: - :titlesonly: - :caption: Miscellaneous - - post_processing + workflow .. toctree:: :hidden: diff --git a/docs/source/workflow.md b/docs/source/workflow.md index 624a9819d..49175beed 100644 --- a/docs/source/workflow.md +++ b/docs/source/workflow.md @@ -1,24 +1,191 @@ -# Snakemake Workflow +# The Snakemake workflow -The end-to-end CFIS production run is driven by a -[Snakemake](https://snakemake.readthedocs.io) workflow that lives in the -`workflow/` directory of the repository. It replaces the older bit-coded bash -job chain (`job_sp`) that this documentation previously described. +Production ShapePipe runs — a tile list, not a tile — are orchestrated by the +Snakemake workflow in `workflow/`. It is the current and only orchestration for +real data: the CANFAR submission layers that preceded it were retired at +`2ef07e45`, while the bit-coded bash job scripts they drove +(`run_job_sp_canfar_v2.0.bash`, `job_sp_canfar_v2.0.bash`) survive only as the +entry point of sp_validation's image-simulation chain. -The workflow owns: +**Module code is untouched by it.** Every rule ultimately calls +`shapepipe_run -c ` on the config chains described in +[Configuration](configuration.md); the workflow supplies the scheduling, the +per-unit isolation, the bookkeeping and the reclamation around them. Reading +[Basic execution](basic_execution.md) first is still the right order. -- the per-module ShapePipe configuration files, in `workflow/config/cfis/`, - which is what `$SP_CONFIG` points at; -- the rule graph and job dispatch, in `workflow/Snakefile` and - `workflow/rules/`; -- the run-level settings, in `workflow/config.yaml`. +This page is the orientation. The deep reference — design rationale, every +rule, the profile, the two-roots argument — is +[`workflow/README.md`](https://github.com/CosmoStat/shapepipe/blob/develop/workflow/README.md) +in the repository, and the living design record is +[issue #848](https://github.com/CosmoStat/shapepipe/issues/848). -Full usage — dependencies, profiles, how to launch a run, and how the rules map -onto the ShapePipe modules — is documented in `workflow/README.md` in the -repository. +## Quick start -```{note} -The PSF-validation and post-processing configurations that the workflow does -not yet cover are kept separately in `example/cfis/`; see the README there and -[Post-processing](post_processing.md). +```bash +# One-time, on a SHARED filesystem: the SLURM executor re-invokes this python +# inside every job, so a node-local path (/tmp) will not do. +uv venv /project///snakemake-env --python 3.12 +source /project///snakemake-env/bin/activate +uv pip install 'snakemake>=9,<10' 'snakemake-executor-plugin-slurm>=2.7,<3' + +# Edit workflow/config.yaml: tile_list, run_dir, products_dir, container, +# star_cats. + +workflow/bin/sp run # bring products on disk up to date with the tile list +workflow/bin/sp report # the success/failure tables, any time, mid-run is fine +workflow/bin/sp container status # which image the jobs will run inside +workflow/bin/sp cancel # scancel this workflow's jobs +``` + +`workflow/bin/sp` is the entry point for everything. It loads the apptainer +module and the venv, snapshots the code it is about to launch, and sets the +state directory, the SLURM profile and `SP_PHASE`. **Bare `snakemake` outside +`sp` is unsupported** — it skips all of that, including the phase variable the +Snakefile needs to build its index at parse time. Any verb `sp` does not +recognise passes straight through to snakemake with the right profile and state +directory, which is the escape hatch for `sp --unlock`, `sp --dag` or +`sp exp_psf ...`. + +## The two static invocations + +The exposure job set is *derived from data the run itself produces* — a tile's +`find_exposures` output — so it cannot live in the same static DAG that produces +it. `sp run` is therefore two snakemake invocations over one Snakefile: + +1. **PREPARE** (`snakemake prepare_all_tiles`) — the per-tile static chain + `tile_get_images → tile_uncompress → tile_find_exposures`, with `keep-going` + so tile failures stay independent. A nonzero exit is a warning, not a fatal + error: tiles that lost their exposure list are dropped at the compute parse, + and `SP_MISSING_THRESHOLD` is the real gate on how many may be lost. +2. **COMPUTE** (`snakemake all`) — this invocation's *parse* builds the + tile↔exposure index (`run_index.sqlite`) and the DAG then runs the full + exposure and tile chains: + + | level | chain | + | --- | --- | + | exposure | `exp_get_images → exp_star_cat → exp_split → exp_mask → exp_psf → exp_persist → exp_footprint → clean_exposure` | + | tile | `tile_exp_forest → tile_merge_headers → tile_detect → tile_vignets → tile_ngmix → tile_merge_cats → tile_make_cat → clean_tile` | + | campaign | `star_catalogue`, `coverage_map` | + +`sp run` chains both and fails if either phase did. The index **accumulates** +across invocations, so appending tiles to `tile_list` later changes which jobs +exist without invalidating completed work. + +Every rule instance is its own SLURM job (`executor: slurm` in +`profiles/nibi/config.yaml`) carrying that rule's own attempt-scaled resources, +and runs inside the container via snakemake's software-deployment method — the +workflow never calls apptainer itself. + +## Manifests are the currency + +The atom is **one rule == one `shapepipe_run` on one unit** (one tile, or one +exposure), in its own sharded store: `/tiles/<2-char prefix>//` or +`/exp///`. + +A rule's single declared output is not its product files but that unit's +**manifest**, `/manifests/.json`. Per-CCD declaration would +mean millions of paths at DR6 scale, and a missing CCD is frequently +legitimate. + +Manifests are *success-only*. After each run, `completeness.py check` writes its +full verdict — per-runner counts against floors, scraped failure reasons, the +`shapepipe_run` exit status — to the rule's `log:` +(`/logs/.json`) **every** time, and additionally to the +declared manifest **only** when that verdict is a success. So a manifest on disk +means "this stage succeeded", and a resume after an unclean death can never +schedule downstream work on top of a failure. + +Completeness is a **count floor, not a taxonomy**: a stage is a real failure iff +a mandatory runner produced fewer products than its floor. Per-CCD attrition +between the floor and the expected count is tolerated and recorded. There is no +error-signature whitelist. + +## `sp report` + +`sp report` reads the index and those two directories — the manifest for +success, the log for failure, a unit with neither ran nothing — and prints the +per-stage tables plus a `run_report.json` beside the index. It is deliberately +**not** a DAG node: a report rule declaring every tile's output would be a +descendant of every job, so one hard failure under `--keep-going` would poison +its cone and the report would never run in exactly the situation it exists for. +It is a plain script, runnable at any time including mid-run, and the COMPUTE +invocation's `onsuccess`/`onerror` hooks fire it automatically so every run ends +with one. + +## `sp container` + +`sp container` owns which image the jobs run inside, in one resolution order +shared by the CLI and the workflow: **sandbox → your cached SIF → the +`container:` path in `config.yaml`**. + +```bash +sp container status # layers present, active one, revision vs HEAD +sp container pull # ghcr.io/cosmostat/shapepipe:develop-runtime +sp container sandbox # unpack the SIF writable (opt-in) +sp container exec --writable pip install +sp container resolve # just the path the workflow will run ``` + +The cached SIF is a private pull, so nobody else's refresh moves the ground +under your running jobs; the sandbox is the escape hatch for work needing a +package the image does not carry yet. `status` places the image's +`org.opencontainers.image.revision` label against this checkout's HEAD. +`pull` needs the network — run it on a login node, never from a batch job. + +See [Container workflow](container.md) for how the images are built. + +## Durable products + +Two roots. `run_dir` is scratch: the bulk per-unit stores, sized to finish +inside the purge window. `products_dir` is persistent and holds the low-volume +things worth keeping: + +- **Final catalogues** — `/tiles///final_cat-.fits`, + mirroring the scratch shard structure. This file is also the *tile-finished + marker* that lets a completed tile stop pinning its exposures, which is the + second reason it cannot live on scratch. +- **PSF products** — `exp_persist` packs the files named by `persist_exp:` in + `config.yaml` (by default the psfex_interp `validation_psf-*.fits`, the rho/tau + statistics input) into one uncompressed tar per exposure, plus a manifest + recording the members. It answers the scratch purge, not the workflow's own + reclamation, so it runs whether or not `clean:` is on. +- **The index and the report** — `run_index.sqlite`, `missing.json`, + `run_report.json`. +- **Coverage** — see below. + +Exposure stores are reclaimed by the in-DAG `clean_exposure` rule rather than by +`temp()`: exposures overlap tiles, so `temp()` would cascade destructive reruns +whenever a tile is appended. Reclamation leaves a `cleaned.json` tombstone +holding the absorbed manifests, so a reclaimed exposure still reports its +per-runner counts and blocks no tile. + +## Coverage + +Sky coverage is a workflow product, built from records the DAG already writes +rather than by scraping a finished campaign. + +`exp_footprint` writes one JSON per exposure to +`/exp///manifests/exp_footprint.json` giving the +four sky corners of every CCD that got a PSF model. The valid-PSF CCD set comes +off `exp_persist.json`'s tar members — exact, because `psfex_interp` returns +*without* writing `validation_psf-*.fits` on NOT_ENOUGH_STARS, BAD_CHI2 or +FILE_NOT_FOUND — and the WCS off the `headers-.npy` written by `exp_split`. +That makes `validation_psf-*` in `persist_exp:` a precondition of the chain, not +a preference. + +Set `coverage: {enabled: true}` and one further job, `coverage_map`, stamps every +footprint into `/coverage/coverage.hsp` — a HealSparse map counting, +per sky pixel, the exposures with a valid PSF there. It is **campaign-cumulative**: +its declared inputs are the in-scope footprints, but the script reads every record +on the products root, reclaimed exposures included, so appending tiles grows the map +instead of replacing it. `nside` is set in `config.yaml` to the production +128/131072 pair, chosen to align pixel-wise with the UNIONS bit masks — nothing +defaults to it, and a coarser map would look plausible and silently fail to align. + +Plotting stays out of the DAG, as a human act on a durable product: + +```bash +plot_coverage_map -i /coverage/coverage.hsp ... +``` + +with the sky windows under `coverage.plot` in `config.yaml`. diff --git a/example/cfis/config_valjoint_Pl_mccd.ini b/example/cfis/config_valjoint_Pl_mccd.ini index 086690815..81c444b55 100644 --- a/example/cfis/config_valjoint_Pl_mccd.ini +++ b/example/cfis/config_valjoint_Pl_mccd.ini @@ -39,7 +39,7 @@ INPUT_DIR = ./SP OUTPUT_DIR = ./output # FILE_PATTERN (opional) list of string patterns to identify input files -FILE_PATTERN = unions_shapepipe_psf_2022_v1.0.2 +FILE_PATTERN = unions_shapepipe_psf_2022 # FILE_EXT (opional) list of string extensions to identify input files FILE_EXT = .fits diff --git a/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini b/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini index 26564e4fa..5658df35d 100644 --- a/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini +++ b/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini @@ -120,8 +120,8 @@ STAMP_SIZE = 51 PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure -# run outputs. ME_IMAGE_EXP_DIR/ME_IMAGE_EXP_RUNNERS replace ME_IMAGE_DIR for -# the v2.0 per-exposure pipeline; output dirs are discovered by scanning $SP_EXP. +# run outputs. Output dirs are discovered by scanning $SP_EXP for the +# exposures listed in the exp_numbers input file. ME_IMAGE_EXP_DIR = $SP_EXP ME_IMAGE_EXP_RUNNERS = split_exp_runner, split_exp_runner, split_exp_runner ME_IMAGE_PATTERN = flag, image, weight diff --git a/pyproject.toml b/pyproject.toml index 74d0299fc..47ae70248 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -38,7 +38,6 @@ dependencies = [ "PyQt5", "pyqtgraph", "sf_tools>=2.0.4", - "skaha>=1.7", "skyproj", "sqlitedict>=2.0", "termcolor", @@ -82,14 +81,8 @@ dev = ["shapepipe[doc,jupyter,lint,plot,release,test,fitsio]"] [project.scripts] shapepipe_run = "shapepipe.shapepipe_run:main" -summary_run = "shapepipe.summary_run:main" -canfar_submit_job = "shapepipe.canfar_run:run_job" canfar_monitor = "shapepipe.canfar_run:run_log" canfar_monitor_log = "shapepipe.canfar_run:run_monitor_log" -get_ccds_with_psf = "shapepipe.get_ccds_run:run_ccd_psf_handler" -download_headers = "shapepipe.coverage_run:run_download_headers" -extract_field_corners = "shapepipe.coverage_run:run_extract_corners" -build_coverage_map = "shapepipe.coverage_run:run_build_coverage" plot_coverage_map = "shapepipe.coverage_run:run_plot_coverage" [tool.uv] diff --git a/scripts/README.rst b/scripts/README.rst index 2b8067cb0..482aaea78 100644 --- a/scripts/README.rst +++ b/scripts/README.rst @@ -1,19 +1,77 @@ Scripts directory ================= -This directory contain scripts that are used on pipeline outputs or to prepare -files. They are not run throught the pipeline framework. For more details on how -to run each them see below. +Helpers that operate on pipeline inputs and outputs but are **not** pipeline +modules: nothing here is run by ``shapepipe_run``, and nothing here is part of +the Snakemake workflow's rule graph (that is ``workflow/scripts/``). -Python scripts -============== +Everything under ``scripts/*/`` with a ``.py``, ``.sh`` or ``.bash`` extension is +symlinked onto ``$PATH`` inside the container image under its bare name, so +``cfis_field_select`` and ``python scripts/python/cfis_field_select.py`` are the +same command. -1. `create_log_exp_headers`_ +Each script's own ``--help`` and module docstring is the reference; this file is +only a map. Run any of them with ``-h`` first. -create_log_exp_headers -====================== +``python/`` — pipeline inputs +----------------------------- -This as to run after the module `split_exp_runner` it will create a master log -file containing all the WCS information for each CCDs of each single exposures. -To run the script : -`python create_log_exp_headers.py path/to/split_exp_runner/output path/to/srcipt/output_dir` +- ``cfis_field_select.py`` — select CFIS tiles or exposures by sky coordinates + or by name, and emit the ID list. This is how a tile list for + ``workflow/config.yaml`` is made. +- ``check_tile_coverage.py`` — flag input simulation tiles whose weight maps are + mostly empty, as a YAML exclude list. Called from sp_validation's image-sims + orchestration, not from here. + +``python/`` — pipeline outputs +------------------------------ + +- ``collate_star_cat.py`` — collate the per-exposure PSF validation catalogues + into the input ``merge_starcat`` wants for the rho/tau statistics. +- ``create_final_cat.py`` — merge per-tile final catalogues into one HDF5 file + for sp_validation. +- ``stats_global.py`` — histograms and tables from the SETools per-CCD summary + statistics of a whole run. + +``python/`` — image simulations +------------------------------- + +The image-simulation chain is still driven by the pre-Snakemake bash layer, +which sp_validation's ``image_sims.smk`` calls into. These exist for it and are +not used by real-data runs. + +- ``init_run_v2.0.py`` — lay out a run directory for that layer. +- ``update_runs_log_file.py`` — rebuild ``log_run_sp.txt`` from the run + directories on disk, which that layer needs because it deletes run dirs. +- ``test_tile_det.py`` + ``test_tile_det.cfg`` — drive tile detection over it. + +``python/`` — the ngmix status dashboard +---------------------------------------- + +Orthogonal to everything above: ``build_status.py``, ``build_history.py``, +``plot_trends.py``, ``run_breakdown_grid.py``, ``plot_breakdown_grid.py`` and +``plot_s4_ablations.py`` build the shape-measurement status page and its +trend/ablation figures. + +``sh/`` +------- + +- ``shapepipe_run_example.sh`` — run the bundled example pipeline against a + writable copy of ``/app/example``. This is the container's CI smoke test. +- ``apptainer_noslurm.sh`` — ``apptainer exec`` with the SLURM/PMI/PMIX/OMPI + environment stripped, for MPI inside a SLURM job. +- ``run_job_sp_canfar_v2.0.bash``, ``job_sp_canfar_v2.0.bash``, + ``job_list_help.bash``, ``functions.sh`` — the bit-coded bash job layer. Kept + only for image simulations (above); real-data runs go through ``workflow/``. + +``jupyter/`` +------------ + +Scratch notebooks: ``wcs.ipynb``, ``test_centroid_shift.py``. + +``validation/`` +--------------- + +``centroid/`` — the metacalibration multiplicative-bias validation of the ngmix +centroid handling: ``run_all.sh`` drives ``centroid_bias_v2.py`` over the three +cases in ``configs/``. diff --git a/scripts/jupyter/summary_run.ipynb b/scripts/jupyter/summary_run.ipynb deleted file mode 100644 index 44f2f2545..000000000 --- a/scripts/jupyter/summary_run.ipynb +++ /dev/null @@ -1,399 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "id": "130112a4-f2ca-4d26-b884-d8b054676f9c", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "%load_ext autoreload\n", - "%autoreload 3\n", - "%reload_ext autoreload" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "196beca5-10a1-4cf5-9462-be145167cc70", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "ename": "ModuleNotFoundError", - "evalue": "No module named 'shapepipe'", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", - "Cell \u001b[0;32mIn[1], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mshapepipe\u001b[39;00m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mshapepipe\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutilities\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msummary\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;241m*\u001b[39m\n", - "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'shapepipe'" - ] - } - ], - "source": [ - "import shapepipe\n", - "from shapepipe.utilities.summary import *" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "ca63c72d-212c-463e-a792-71efbac0b908", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Setting\n", - "patch = \"P7\"\n", - "\n", - "verbose = False" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "dcb5604c-d61f-4705-8295-63875455cadb", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Load parameters\n", - "%run ~/shapepipe/scripts/python/summary_params_pre_v2" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "e69b7dab-1fea-4fcc-a8d9-0720e1d628c3", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Checking main directory = /arc/home/kilbinger/cosmostat/v2/pre_v2/psfex/P7\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Set job info for patch P7\n" - ] - } - ], - "source": [ - "jobs, list_tile_IDs = set_jobs_v2_pre_v2(patch, verbose)" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "2c3a9dde-cf88-493f-926e-7ae7e8e10916", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Initialize runtime dicionary.\n", - "par_runtime = init_par_runtime(list_tile_IDs)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "1e9c1487-3cec-4394-9fcf-c12e92a0f984", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# No effect in notebook\n", - "#print_par_runtime(par_runtime, verbose=verbose)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "b7c63a22-ead1-4d6a-b081-a74ade515439", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "module expected found miss_expl missing uniq_miss fr_found\n", - "====================================================================================================\n" - ] - } - ], - "source": [ - "# Start program\n", - "job_data.print_stats_header()" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "4720ae18-0633-4646-b392-b1b24e0294c3", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " (Job 1)\n", - "get_images_runner_run_1 462 462 0 0 0.0 100.0%\n", - "find_exposures_runner 231 231 0 0 0.0 100.0%\n", - "get_images_runner_run_2 537 0 0 537 179.0 0.0%\n" - ] - } - ], - "source": [ - "for key in \"1\":\n", - " jobs[key].print_intro()\n", - " jobs[key].check_numbers(par_runtime=par_runtime, indices=[0, 1])\n", - "\n", - " all_exposures = get_all_exposures(jobs[key]._paths_in_dir[1], verbose=verbose)\n", - " par_runtime[\"n_exposures\"] = len(all_exposures)\n", - " par_runtime[\"list_exposures\"] = all_exposures\n", - "\n", - " jobs[key].check_numbers(par_runtime, indices=[2])" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "f149f404-64e7-4d92-8f54-f300ed620130", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Update runtime dictionary with numbers of exposures\n", - "par_runtime = update_par_runtime_after_find_exp(par_runtime, all_exposures)" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "78a9065f-8983-41cf-a34c-21892fc52dd2", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Get all keys after \"1\"\n", - "keys = sorted(jobs.keys(), key=int)\n", - "_ = keys.pop(0)" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "79e39954-1155-4ca3-b0b2-64bc5670db53", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " (Job 2)\n", - "uncompress_fits_runner 1268 1268 0 0 0.0 100.0%\n", - "merge_headers_runner 0 0 0 0 0.0 100.0%\n", - "split_exp_runner 137940 137940 0 0 0.0 100.0%\n", - " (Job 4)\n", - "mask_runner 1268 1268 0 0 0.0 100.0%\n", - " (Job 8)\n", - "mask_runner 45600 45600 0 0 0.0 100.0%\n", - " (Job 16)\n", - "sextractor_runner 2536 2536 0 0 0.0 100.0%\n", - " (Job 32)\n", - "sextractor_runner 91200 91200 0 0 0.0 100.0%\n", - "setools_runner 91200 91032 0 168 84.0 99.8%\n", - "psfex_runner 91200 91032 0 168 84.0 99.8%\n", - " (Job 64)\n", - "psfex_interp_runner 1268 1268 0 0 0.0 100.0%\n", - "vignetmaker_runner_run_1 1268 1268 0 0 0.0 100.0%\n", - "spread_model_runner 1268 1268 0 0 0.0 100.0%\n", - "vignetmaker_runner_run_2 5072 5072 0 0 0.0 100.0%\n", - " (Job 128)\n", - "ngmix_runner 1268 1225 0 43 43.0 96.6%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - "ngmix_runner 1268 1217 0 51 51.0 96.0%\n", - "ngmix_runner 1268 1228 0 40 40.0 96.8%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - " (Job 256)\n", - "merge_sep_cats_runner 1268 0 0 1268 1268.0 0.0%\n", - "make_cat_runner 1268 0 0 1268 1268.0 0.0%\n", - " (Job 1024)\n", - "psfex_interp_runner 45600 41132 0 4468 4468.0 90.2%\n" - ] - } - ], - "source": [ - "for key in keys:\n", - " jobs[key].print_intro()\n", - " jobs[key].check_numbers(par_runtime=par_runtime)" - ] - }, - { - "cell_type": "code", - "execution_count": 65, - "id": "b3d51a05-ecca-420b-b8b3-1fb2b1ec9fe3", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " (Job 128)\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1265 0 3 3.0 99.8%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1266 0 2 2.0 99.8%\n", - "ngmix_runner 1268 1268 0 0 0.0 100.0%\n", - "ngmix_runner 1268 1266 0 2 2.0 99.8%\n" - ] - } - ], - "source": [ - "## Update some runs\n", - "for key in [\"128\"]:\n", - " jobs[key].print_intro()\n", - " jobs[key].check_numbers(par_runtime=par_runtime)" - ] - }, - { - "cell_type": "code", - "execution_count": 64, - "id": "67b50a61-e3cc-4559-941d-f39c6a200294", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " (Job 128)\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1265 0 3 3.0 99.8%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1266 0 2 2.0 99.8%\n", - "ngmix_runner 1268 1268 0 0 0.0 100.0%\n", - "ngmix_runner 1268 1266 0 2 2.0 99.8%\n" - ] - } - ], - "source": [ - "for key in [\"128\"]:\n", - " jobs[key].print_intro()\n", - " \n", - " \n", - " jobs[key].check_numbers(par_runtime=par_runtime)" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "affa8293-daf9-4d2b-9215-fe19f8e2c1e2", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "session = Session()" - ] - }, - { - "cell_type": "code", - "execution_count": 36, - "id": "269496d1-cd89-4d13-a5e4-41b897669e22", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "ids = [session[\"id\"] for session in session.fetch(kind=\"headless\")]" - ] - }, - { - "cell_type": "code", - "execution_count": 38, - "id": "80af8dff-98c7-4db4-8bcc-06936e1875cf", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "ename": "RuntimeError", - "evalue": "This event loop is already running", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m/tmp/ipykernel_69/559116804.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdestroy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mids\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", - "\u001b[0;32m~/.local/lib/python3.7/site-packages/skaha/session.py\u001b[0m in \u001b[0;36mdestroy\u001b[0;34m(self, id)\u001b[0m\n\u001b[1;32m 264\u001b[0m \u001b[0marguments\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m\"url\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserver\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\"/\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 265\u001b[0m \u001b[0mloop\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_event_loop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 266\u001b[0;31m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_until_complete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mscale\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdelete\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marguments\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 267\u001b[0m \u001b[0mresponses\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mDict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 268\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midentity\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/conda/lib/python3.7/asyncio/base_events.py\u001b[0m in \u001b[0;36mrun_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m 561\u001b[0m \"\"\"\n\u001b[1;32m 562\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_closed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 563\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_runnung\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 564\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 565\u001b[0m \u001b[0mnew_task\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mfutures\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0misfuture\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfuture\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/conda/lib/python3.7/asyncio/base_events.py\u001b[0m in \u001b[0;36m_check_runnung\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 521\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_check_runnung\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 522\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_running\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 523\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'This event loop is already running'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 524\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mevents\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_running_loop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 525\u001b[0m raise RuntimeError(\n", - "\u001b[0;31mRuntimeError\u001b[0m: This event loop is already running" - ] - } - ], - "source": [ - "session.destroy(ids[0])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "66a3ed14-8aaf-4028-b933-10ecb7376d68", - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.9" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/scripts/python/clear_ngmix_prev.py b/scripts/python/clear_ngmix_prev.py deleted file mode 100644 index a1a7e802c..000000000 --- a/scripts/python/clear_ngmix_prev.py +++ /dev/null @@ -1,155 +0,0 @@ -#!/usr/bin/env python3 -""" -Clean up previous ngmix run directories for completed tiles. -Removes *_prev directories when the current run has finished. -""" - -import argparse -import os -import re -import shutil -import glob -from pathlib import Path -from astropy.io import fits - - -def get_fits_info(fits_file): - """Get number of objects (NAXIS2 in HDU 1) and number of HDUs.""" - try: - with fits.open(fits_file) as hdul: - n_hdus = len(hdul) - # Get NAXIS2 from HDU 1 (second HDU, index 1) - if len(hdul) > 1: - n_objects = hdul[1].header.get("NAXIS2", "N/A") - else: - n_objects = "N/A (no HDU 1)" - return n_objects, n_hdus - except Exception as e: - return f"Error: {e}", "N/A" - - -def check_finished(log_file): - """Check if 'finished' appears in the log file.""" - try: - with open(log_file, "r") as f: - content = f.read() - return "finished" in content - except Exception as e: - print(f"Warning: Could not read {log_file}: {e}") - return False - - -def main(): - parser = argparse.ArgumentParser( - description="Clean up previous ngmix run directories for completed tiles." - ) - parser.add_argument( - "-n", "--dry-run", - action="store_true", - help="Show what would be removed without actually removing files" - ) - args = parser.parse_args() - - if args.dry_run: - print("dry run mode: No files will be removed\n") - - base_dir = Path("tile_runs") - - # Pattern to match tile IDs like 123.456 - tile_id_pattern = re.compile(r"^\d+\.\d+$") - - # Find all tile ID directories - if not base_dir.exists(): - print(f"Error: {base_dir} does not exist") - return - - tile_dirs = [ - d - for d in base_dir.iterdir() - if d.is_dir() and tile_id_pattern.match(d.name) - ] - - print(f"Found {len(tile_dirs)} tile directories") - - removed_count = 0 - finished_count = 0 - - for tile_dir in sorted(tile_dirs): - tile_id = tile_dir.name - - # Construct path to ngmix_runner directory - ngmix_dir = ( - tile_dir / "output" / "run_sp_tile_ngmix_Ng1u" / "ngmix_runner" - ) - - if not ngmix_dir.exists(): - continue - - # Check for process-*.log files - log_pattern = str(ngmix_dir / "logs" / "process-*.log") - log_files = glob.glob(log_pattern) - - if not log_files: - continue - - # Check if any log file contains "finished" - is_finished = any(check_finished(log_file) for log_file in log_files) - - if is_finished: - finished_count += 1 - - # Path to the _prev directory to remove - prev_dir = tile_dir / "output" / "run_sp_tile_ngmix_Ng1u_prev" - - if prev_dir.exists(): - - # File info of new - ngmix_out_dir = ngmix_dir / "output" - fits_files = list(ngmix_out_dir.glob("ngmix-*.fits")) - if fits_files: - for fits_file in fits_files: - n_objects, n_hdus = get_fits_info(fits_file) - else: - n_objects, n_hdus = (-1, -1) - - # File info of prev - prev_out_dir = prev_dir / "output" - fits_files = list(prev_out_dir.glob("ngmix-*.fits")) - if fits_files: - for fits_file in fits_files: - n_objects_prev, n_hdus_prev = get_fits_info(fits_file) - else: - n_objects_prev, n_hdus_prev = (-1, -1) - - action = "Would remove" if args.dry_run else "Removing" - print( - f"Tile {tile_id}: {action} {prev_dir} (new: {n_objects}" - + f" {n_hdus}) (prev: {n_objects_prev} {n_hdus_prev})" - ) - - if n_objects < n_objects_prev: - print(f"Warning: {tile_id}") - - if not args.dry_run: - try: - shutil.rmtree(prev_dir) - removed_count += 1 - except Exception as e: - print(f"Error removing {prev_dir}: {e}") - else: - removed_count += 1 - else: - pass - else: - print(f"Tile {tile_id}: Not finished") - - print(f"\nSummary:") - print(f" Tiles with finished runs: {finished_count}") - if args.dry_run: - print(f" Previous directories that would be removed: {removed_count}") - else: - print(f" Previous directories removed: {removed_count}") - - -if __name__ == "__main__": - main() diff --git a/scripts/python/collate_star_cat.py b/scripts/python/collate_star_cat.py index f7ab25b41..a3ccc2a2e 100755 --- a/scripts/python/collate_star_cat.py +++ b/scripts/python/collate_star_cat.py @@ -2,29 +2,25 @@ """COLLATE STAR CATALOGUES. -Collate the per-exposure PSF validation catalogues into star catalogues: gather -positions (X/Y/RA/DEC), assign the MCCD focal-plane CCD id, and merge into -``validation_psf_conv`` FITS files. - -Catalogue layout depends on the major version (``-V``). Runs up to ``v1.6`` are -organised in sky patches (``P1``..``P``): input runs live under -``/P/output`` and outputs are named -``validation_psf_conv--.fits``. ``v2.0`` removes the patch concept: -input runs live under a single ``/output`` root and outputs drop the patch -token (``validation_psf_conv-.fits``). - -HSM ellipticities and sizes are no longer rotated here. PSFEx and the in-repo -MCCD interpolation now measure adaptive moments directly in world coordinates -(galsim ``FindAdaptiveMom(use_sky_coords=True)``), so the WCS-Jacobian shape -rotation this script used to perform is redundant and has been removed. - -Caveat: the MCCD ``PSF_MOM_LIST``/``STAR_MOM_LIST`` columns are produced by the -external ``mccd`` fit-validation code (``mccd.auxiliary_fun.mccd_validation``), -which still measures HSM moments in the pixel frame. Those shapes are therefore -still rotated into world coordinates here, via the WCS-Jacobian rotation, until -``mccd`` itself adopts ``use_sky_coords``. This is the one branch that keeps the -rotation; the in-repo PSFEx and MCCD-interpolation paths measure adaptive -moments directly in world coordinates upstream and pass them straight through. +Collate the per-exposure PSFEx validation catalogues into star catalogues: +gather positions (X/Y/RA/DEC) and the HSM PSF and star shapes, tag each row +with its CCD number, and merge one ``validation_psf_conv-.fits`` per +single-exposure run. + +HSM ellipticities and sizes are not rotated here. ``psfex_interp`` measures +adaptive moments directly in world coordinates (galsim +``FindAdaptiveMom(use_sky_coords=True)``), so the WCS-Jacobian shape rotation +this script used to perform is redundant and has been removed; only positions +are collated. + +Caveat -- no workflow rule produces these catalogues yet. The input layout +below is the pre-workflow one: a link farm of ``psfex_interp`` run directories +under a single input root. The Snakemake workflow instead persists +``validation_psf--.fits`` into the per-exposure ``exp_persist`` +tars on the products root, and ``merge_starcat_runner`` still wants the +collated ``validation_psf_conv-*`` files for the rho/tau statistics. +Repointing this script at the persist tars (or replacing it with a rule) is a +follow-up, and a bigger change than this strip. """ import sys @@ -37,51 +33,27 @@ import numpy as np from astropy.io import fits -import galsim from cs_util import args as cs_args from cs_util import logging -def collate_paths(input_base_dir, output_base_dir, patch): - """Collate Paths. +# Input layout: single-exposure run directories matching ``EXP_RUN_GLOB`` under +# ``/output``, each holding the PSF interpolation output in +# ``PSF_INTERP_SUBDIR``. +EXP_RUN_GLOB = "run_sp_combined_psf*" +PSF_INTERP_SUBDIR = "psfex_interp_runner/output" - Return the ``(input run dir, output dir)`` for a patch. ``patch`` is None - for the patch-less v2.0 layout, which drops the ``P`` token; v1.x - passes the patch number. - Parameters - ---------- - input_base_dir : str - input base directory - output_base_dir : str - output base directory - patch : str or None - patch number, or None for the patch-less v2.0 layout - - Returns - ------- - tuple - input run directory and output directory - - """ - if patch is None: - return f"{input_base_dir}/output/", output_base_dir - return f"{input_base_dir}/P{patch}/output/", f"{output_base_dir}/P{patch}" - - -def output_filename(file_pattern, patch, idx): +def output_filename(file_pattern, idx): """Output Filename. - Build the collated catalogue filename. ``patch`` is None for the patch-less - v2.0 layout, which drops the patch token. + Build the collated catalogue filename. Parameters ---------- file_pattern : str input file pattern (e.g. ``validation_psf``) - patch : str or None - patch number, or None for the patch-less v2.0 layout idx : int exposure run index @@ -91,340 +63,7 @@ def output_filename(file_pattern, patch, idx): output catalogue file name """ - patch_token = "" if patch is None else f"{patch}-" - return f"{file_pattern}_conv-{patch_token}{idx}.fits" - - -def transform_shape(mom_list, jac): - """Transform Shape. - - Transform shape (ellipticity and size) using a Jacobian. - - Parameters - ---------- - mom_list : list - input moment measurements; each list element contains - first and second ellipticity component and size - jac : galsim.JacobianWCS - Jacobian transformation matrix information - - Returns - ------- - list - transformed shape parameters, which are - first and second ellipticity component and size - - """ - scale, shear, theta, flip = jac.getDecomposition() - - sig_tmp = mom_list[2] * scale - shape = galsim.Shear(g1=mom_list[0], g2=mom_list[1]) - if flip: - # The following output is not observed - print("FLIP!") - shape = galsim.Shear(g1=-shape.g1, g2=shape.g2) - shape = galsim.Shear(g=shape.g, beta=shape.beta + theta) - shape = shear + shape - - return shape.g1, shape.g2, sig_tmp - - -class Loc2Glob(object): - r"""Change from local to global coordinates. - - Class to pass from local coordinates to global coordinates under - CFIS (CFHT) MegaCam instrument. The geometrical informcation of the - instrument is encoded in this function. - - Parameters - ---------- - x_gap : int - Gap between the CCDs along the horizontal direction; - default is ``70`` (MegaCam value) - y_gap : int - Gap between the CCDs along the vertical direction; - Default is ``425`` (MegaCam value) - x_npix : int - Number of pixels per CCD along the horizontal direction; - default is ``2048`` (MegaCam value) - y_npix : int - Number of pixels per CCD along the vertical direction; - default to ``4612`` (MegaCam value) - ccd_tot : int - Total number of CCDs; - default to ``40`` (MegaCam value) - - Notes - ----- - This is the geometry of MegaCam. Watch out with the conventions ba,ab that means where - is the local coordinate system origin for each CCD. - For more info check out MegaCam's instrument webpage. - - Examples - -------- - 'COMMENT (North on top, East to the left)', - 'COMMENT --------------------------', - 'COMMENT ba ba ba ba ba ba ba ba ba', - 'COMMENT 00 01 02 03 04 05 06 07 08', - 'COMMENT --------------------------------', - 'COMMENT ba ba ba ba ba ba ba ba ba ba ba', - 'COMMENT 36 09 10 11 12 13 14 15 16 17 37', - 'COMMENT --------------*-----------------', - 'COMMENT 38 18 19 20 21 22 23 24 25 26 39', - 'COMMENT ab ab ab ab ab ab ab ab ab ab ab', - 'COMMENT --------------------------------', - 'COMMENT 27 28 29 30 31 32 33 34 35', - 'COMMENT ab ab ab ab ab ab ab ab ab', - 'COMMENT __________________________' - """ - - def __init__( - self, x_gap=70, y_gap=425, x_npix=2048, y_npix=4612, ccd_tot=40 - ): - r"""Initialize with instrument geometry.""" - self.x_gap = x_gap - self.y_gap = y_gap - self.x_npix = x_npix - self.y_npix = y_npix - self.ccd_tot = ccd_tot - - def loc2glob_img_coord(self, ccd_n, x_coor, y_coor): - """loc2glob Img Coord. - - Go from the local to the global img (pixel) coordinate system. - - Global system with (0,0) in the intersection of ccds [12,13,21,22]. - - Parameters - ---------- - ccd_n: int - CCD number of the considered positions - x_coor: float - Local coordinate system hotizontal value - y_coor: float - Local coordinate system vertical value - - Returns - ------- - glob_x_coor: float - Horizontal position in global coordinate system - glob_y_coor: float - Vertical position in global coordinate system - - """ - # Flip axes - x_coor, y_coor = self.flip_coord(ccd_n, x_coor, y_coor) - - # Calculate the shift - x_shift, y_shift = self.shift_coord(ccd_n) - - # Return new coordinates - return x_coor + x_shift, y_coor + y_shift - - def flip_coord(self, ccd_n, x_coor, y_coor): - r"""Change of coordinate convention. - - So that all of them are coherent on the global coordinate system. - So that the origin is on the south-west corner. - Positive: South to North ; West to East. - """ - if ccd_n < 18 or ccd_n in [36, 37]: - x_coor = self.x_npix - x_coor + 1 - y_coor = self.y_npix - y_coor + 1 - else: - pass - - return x_coor, y_coor - - def x_coord_range(self): - r"""Return range of the x coordinate.""" - max_x = self.x_npix * 6 + self.x_gap * 5 - min_x = self.x_npix * (-5) + self.x_gap * (-5) - return min_x, max_x - - def y_coord_range(self): - r"""Return range of the y coordinate.""" - max_y = self.y_npix * 2 + self.y_gap * 1 - min_y = self.y_npix * (-2) + self.y_gap * (-2) - return min_y, max_y - - def shift_coord(self, ccd_n): - r"""Provide the shifting. - - It is needed to go from the local coordinate - system origin to the global coordinate system origin. - """ - if ccd_n < 9: - # first row - x_shift = (ccd_n - 4) * (self.x_gap + self.x_npix) - y_shift = self.y_gap + self.y_npix - return x_shift, y_shift - - elif ccd_n < 18: - # second row, non-ears - x_shift = (ccd_n - 13) * (self.x_gap + self.x_npix) - y_shift = 0.0 - return x_shift, y_shift - - elif ccd_n < 27: - # third row non-ears - x_shift = (ccd_n - 22) * (self.x_gap + self.x_npix) - y_shift = -1.0 * (self.y_gap + self.y_npix) - return x_shift, y_shift - - elif ccd_n < 36: - # fourth row - x_shift = (ccd_n - 31) * (self.x_gap + self.x_npix) - y_shift = -2.0 * (self.y_gap + self.y_npix) - return x_shift, y_shift - - elif ccd_n < 37: - # ccd= 36 ears, second row - x_shift = (-5.0) * (self.x_gap + self.x_npix) - y_shift = 0.0 - return x_shift, y_shift - - elif ccd_n < 38: - # ccd= 37 ears, second row - x_shift = 5.0 * (self.x_gap + self.x_npix) - y_shift = 0.0 - return x_shift, y_shift - - elif ccd_n < 39: - # ccd= 38 ears, third row - x_shift = (-5.0) * (self.x_gap + self.x_npix) - y_shift = -1.0 * (self.y_gap + self.y_npix) - return x_shift, y_shift - - elif ccd_n < 40: - # ccd= 39 ears, third row - x_shift = 5.0 * (self.x_gap + self.x_npix) - y_shift = -1.0 * (self.y_gap + self.y_npix) - return x_shift, y_shift - - -class Glob2CCD(object): - r"""Get the CCD ID number from the global coordinate position. - - The Loc2Glob() object as input is the one that defines the instrument's - geometry. - - Parameters - ---------- - loc2glob: Loc2Glob object - Object with the desired focal plane geometry. - with_gaps: bool - If add the gaps to the CCD area. - """ - - def __init__(self, loc2glob, with_gaps=True): - # Save loc2glob object - self.loc2glob = loc2glob - self.with_gaps = with_gaps - self.ccd_list = np.arange(self.loc2glob.ccd_tot) - # Init edges defininf the CCDs - self.edge_x_list, self.edge_y_list = self.build_all_edges() - - def build_all_edges(self): - """Build the edges for all the CCDs in the focal plane.""" - edge_xy_list = [] - for idx in (0, 1): - edge_list = np.array( - [self.build_edge(ccd_n)[idx] for ccd_n in self.ccd_list] - ) - edge_xy_list.append(edge_list) - - return edge_xy_list - - def build_edge(self, ccd_n): - """Build the edges of the `ccd_n` in global coordinates.""" - if self.with_gaps: - corners = np.array( - [ - [-self.loc2glob.x_gap / 2, -self.loc2glob.y_gap / 2], - [ - self.loc2glob.x_npix + self.loc2glob.x_gap / 2, - -self.loc2glob.y_gap / 2, - ], - [ - -self.loc2glob.x_gap / 2, - self.loc2glob.y_npix + self.loc2glob.y_gap / 2, - ], - [ - self.loc2glob.x_npix + self.loc2glob.x_gap / 2, - self.loc2glob.y_npix + self.loc2glob.y_gap / 2, - ], - ] - ) - else: - corners = np.array( - [ - [0, 0], - [self.loc2glob.x_npix, 0], - [0, self.loc2glob.y_npix], - [self.loc2glob.x_npix, self.loc2glob.y_npix], - ] - ) - - glob_corners = np.array( - [ - self.loc2glob.loc2glob_img_coord(ccd_n, pos[0], pos[1]) - for pos in corners - ] - ) - - edge_xy = [] - for idx in (0, 1): - edge = np.array( - [np.min(glob_corners[:, idx]), np.max(glob_corners[:, idx])] - ) - edge_xy.append(edge) - - return edge_xy - - def is_inside(self, x, y, edge_x, edge_y): - """Is the position inside the edges. - - Return True if the position is within the rectangle - defined by the edges. - - Parameters - ---------- - x: float - Horizontal position in global coordinate system. - y: float - Vertical position in global coordinate system. - edge_x: np.ndarray - Edge defined as `np.array([min_x, max_x])`. - edge_y: np.ndarray - Edge defined as `np.array([min_y, max_y])`. - """ - if ( - (x > edge_x[0]) - and (x < edge_x[1]) - and (y > edge_y[0]) - and (y < edge_y[1]) - ): - return True - else: - return False - - def get_ccd_n(self, x, y): - """Returns the CCD number from the position `(x, y)`. - - Returns `None` if the position is not found. - """ - bool_list = np.array( - [ - self.is_inside(x, y, edge_x, edge_y) - for edge_x, edge_y in zip(self.edge_x_list, self.edge_y_list) - ] - ) - - try: - return self.ccd_list[bool_list][0] - except Exception: - return None + return f"{file_pattern}_conv-{idx}.fits" class Convert(object): @@ -457,38 +96,25 @@ def params_default(self): self._params = { "input_base_dir": ".", "output_base_dir": ".", - "version_cat": "v2.0", "mode": "merge", - "patches": "", - "psf": "psfex", "file_pattern_psfint": "validation_psf", } self._short_options = { "input_base_dir": "-i", - "version_cat": "-V", "mode": "-m", - "psf": "-p", - "patches": "-P", } self._types = {} self._help_strings = { "input_base_dir": ( - "input base dir; for v1.x runs are expected in" - + " /P/output, for v2.0 (patch-less) in" + "input base dir; single-exposure runs are expected in" + " /output; default is {}" ), - "version_cat": ( - "catalogue major version, allowed are v1.3, v1.4, v1.5, v1.6," - + " v2.0; v2.0 is patch-less; default is {}" - ), "mode": ( "run mode, allowed are 'merge', 'test'; default is" + " '{}'" ), - "psf": "PSF model, allowed are 'psfex' and 'mccd'; default is {}", - "patches": "(list of) input patches; ignored for v2.0", } # Output column names with types @@ -508,127 +134,67 @@ def params_default(self): ("CCD_NB", int), ] - # Extra columns for MCCD:737 - self._dt_mccd = self._dt.copy() - self._dt_mccd.append(("GLOB_X", float)) - self._dt_mccd.append(("GLOB_Y", float)) - - def update_params(self): - """Update Params. - - Update parameters. - - """ - if self._params["psf"] == "psfex": - #self._params["sub_dir_pattern"] = "run_sp_exp_202" - self._params["sub_dir_pattern"] = "run_sp_combined_psf" - self._params["sub_dir_psfint"] = "psfex_interp_runner" - elif self._params["psf"] == "mccd": - self._params["sub_dir_pattern"] = "run_sp_exp_SxSePsf_202" - self._params["sub_dir_psfint"] = "mccd_fit_val_runner" - self._params["sub_dir_setools"] = "setools_runner/output/mask" - else: - raise ValueError(f"Invalid PSF model {self._params['psf']}") - self._params["sub_dir_psfint"] = ( - f"{self._params['sub_dir_psfint']}/output" - ) - def run(self): """Run. Main processing function. """ - # Guard against a mistyped version silently falling through to the - # v1.x patch loop (e.g. ``-V v2`` or ``-V 2.0``). - allowed_versions = ("v1.3", "v1.4", "v1.5", "v1.6", "v2.0") - if self._params["version_cat"] not in allowed_versions: - raise ValueError( - f"Invalid version {self._params['version_cat']}; allowed are" - + f" {', '.join(allowed_versions)}" - ) - - # v2.0 removes the patch concept: a single patch-less run root. For - # v1.x, iterate over the requested sky patches as before. ``patch`` is - # None in the patch-less case, which drops the patch token from the - # input path and the output filename. - if self._params["version_cat"] == "v2.0": - patch_nums = [None] - elif self._params["mode"] == "test": - patch_nums = ["3", "4"] - else: - patch_nums = cs_args.my_string_split(self._params["patches"]) - do_parallel = True - # Loop over patches - for patch in patch_nums: + output_dir = self._params["output_base_dir"] + if not os.path.isdir(output_dir): + os.makedirs(output_dir, exist_ok=True) - patch_dir, output_dir = collate_paths( - self._params["input_base_dir"], - self._params["output_base_dir"], - patch, - ) - print("Running patch-less (v2.0)" if patch is None else f"Running patch: {patch}") - - if not os.path.isdir(output_dir): - os.makedirs(output_dir, exist_ok=True) + subdirs = f"{self._params['input_base_dir']}/output/{EXP_RUN_GLOB}" + exp_run_dirs = glob.glob(subdirs) + n_exp_runs = len(exp_run_dirs) + print(f"Found {n_exp_runs} input single-exposure run(s) ({subdirs})") - subdirs = f"{patch_dir}/{self._params['sub_dir_pattern']}*" - exp_run_dirs = glob.glob(subdirs) + if self._params["mode"] == "test": + exp_run_dirs = exp_run_dirs[:2] n_exp_runs = len(exp_run_dirs) print( - f"Found {n_exp_runs} input single-exposure run(s) for patch" - + f" {patch_dir} ({subdirs})" + f"test mode: only using {n_exp_runs} input single-exposure" + + f" runs" ) - if self._params["mode"] == "test": - exp_run_dirs = exp_run_dirs[:2] - n_exp_runs = len(exp_run_dirs) - print( - f"test mode: only using {n_exp_runs} input single-exposure" - + f" runs" + # Loop over exposure runs + if not do_parallel: + for idx_exp, exp_run_dir in tqdm( + enumerate(exp_run_dirs), + total=n_exp_runs, + disable=self._params["verbose"], + ): + self.transform_exposures(output_dir, idx_exp, exp_run_dir) + else: + res = Parallel(n_jobs=-1, backend="loky")( + delayed(self.transform_exposures)( + output_dir, idx_exp, exp_run_dir ) - - # Loop over exposure runs - if not do_parallel: for idx_exp, exp_run_dir in tqdm( enumerate(exp_run_dirs), total=n_exp_runs, disable=self._params["verbose"], - ): - self.transform_exposures( - output_dir, patch, idx_exp, exp_run_dir - ) - else: - res = Parallel(n_jobs=-1, backend="loky")( - delayed(self.transform_exposures)( - output_dir, patch, idx_exp, exp_run_dir - ) - for idx_exp, exp_run_dir in tqdm( - enumerate(exp_run_dirs), - total=n_exp_runs, - disable=self._params["verbose"], - ) ) + ) - def transform_exposures(self, output_dir, patch, idx, exp_run_dir): - """Transform exposures. + def transform_exposures(self, output_dir, idx, exp_run_dir): + """Transform Exposures. - Transform shapes for exposure for a given run (input exp run dir). + Collate the PSF validation catalogues of one single-exposure run + (input exp run dir) into one output catalogue. """ output_path = ( f"{output_dir}/" - + output_filename( - self._params["file_pattern_psfint"], patch, idx - ) + + output_filename(self._params["file_pattern_psfint"], idx) ) if os.path.exists(output_path): print(f"Skipping transform_exposures, file {output_path} exists") return - psf_dir = f"{exp_run_dir}/{self._params['sub_dir_psfint']}" + psf_dir = f"{exp_run_dir}/{PSF_INTERP_SUBDIR}" try: all_files = os.listdir(psf_dir) if self._params["verbose"]: @@ -645,11 +211,7 @@ def transform_exposures(self, output_dir, patch, idx, exp_run_dir): tmp = re.findall(r"\d+", file_name) - if self._params["psf"] == "psfex": - exp_name, ccd_id = int(tmp[0]), int(tmp[1]) - elif self._params["psf"] == "mccd": - exp_name = int(tmp[0]) - ccd_id = -1 + exp_name, ccd_id = int(tmp[0]), int(tmp[1]) if self._params["verbose"]: print("Match found ", exp_name, ccd_id) @@ -657,194 +219,42 @@ def transform_exposures(self, output_dir, patch, idx, exp_run_dir): psf_file_path = f"{psf_dir}/{file_name}" try: - if self._params["psf"] == "psfex": - psf_file_hdus = fits.open(psf_file_path, memmap=False) - psf_file = psf_file_hdus[2].data - psf_file_hdus.close() - mod = "RA" - else: - psf_file = fits.getdata(psf_file_path, 1, memmap=True) - mod = "RA_LIST" + psf_file_hdus = fits.open(psf_file_path, memmap=False) + psf_file = psf_file_hdus[2].data + psf_file_hdus.close() except Exception: continue - if self._params["psf"] == "psfex": - # HSM ellipticities and sizes are measured directly in world - # coordinates upstream (FindAdaptiveMom use_sky_coords=True), so - # they are passed straight through; only positions are collated. - exp_cat = np.array( - list( - map( - tuple, - np.array( - [ - psf_file["X"], - psf_file["Y"], - psf_file["RA"], - psf_file["DEC"], - psf_file["E1_PSF_HSM"], - psf_file["E2_PSF_HSM"], - psf_file["SIGMA_PSF_HSM"], - psf_file["FLAG_PSF_HSM"], - psf_file["E1_STAR_HSM"], - psf_file["E2_STAR_HSM"], - psf_file["SIGMA_STAR_HSM"], - psf_file["FLAG_STAR_HSM"], - np.ones_like(psf_file["RA"], dtype=int) - * ccd_id, - ] - ).T.tolist(), - ) - ), - dtype=self._dt, - ) - cat_list.append(exp_cat) - - else: - l2g = Loc2Glob() - g2c = Glob2CCD(l2g) - new_ccd_id = np.array( - [ - int( - g2c.get_ccd_n( - psf_file["GLOB_POSITION_IMG_LIST"][ii, 0], - psf_file["GLOB_POSITION_IMG_LIST"][ii, 1], - ) - ) - for ii in range(len(psf_file)) - ] - ) - - # Local-to-CCD position: subtract each CCD's focal-plane shift. - new_x = np.zeros_like(psf_file[mod]) - new_y = np.zeros_like(psf_file[mod]) - - # The MCCD PSF_MOM_LIST/STAR_MOM_LIST columns come from the - # external mccd fit-validation code, which still measures HSM - # moments in the pixel frame; rotate them into world coordinates - # via the per-CCD WCS Jacobian. This rotation stays until mccd - # itself adopts use_sky_coords (see the module docstring). The - # in-repo PSFEx / MCCD-interpolation paths are already in world - # coordinates and are passed through unrotated. - new_e1_psf = np.zeros_like(psf_file[mod]) - new_e2_psf = np.zeros_like(psf_file[mod]) - new_sig_psf = np.zeros_like(psf_file[mod]) - new_e1_star = np.zeros_like(psf_file[mod]) - new_e2_star = np.zeros_like(psf_file[mod]) - new_sig_star = np.zeros_like(psf_file[mod]) - new_flag_psf = np.zeros_like(psf_file[mod]) - new_flag_star = np.zeros_like(psf_file[mod]) - for ccd_id in range(40): - m_ccd_id = new_ccd_id == ccd_id - if sum(m_ccd_id) == 0: - continue - - x_shift, y_shift = l2g.shift_coord(ccd_id) - - new_x[m_ccd_id] = ( - psf_file["GLOB_POSITION_IMG_LIST"][:, 0][m_ccd_id] - - x_shift + # HSM ellipticities and sizes are measured directly in world + # coordinates upstream (FindAdaptiveMom use_sky_coords=True), so + # they are passed straight through; only positions are collated. + exp_cat = np.array( + list( + map( + tuple, + np.array( + [ + psf_file["X"], + psf_file["Y"], + psf_file["RA"], + psf_file["DEC"], + psf_file["E1_PSF_HSM"], + psf_file["E2_PSF_HSM"], + psf_file["SIGMA_PSF_HSM"], + psf_file["FLAG_PSF_HSM"], + psf_file["E1_STAR_HSM"], + psf_file["E2_STAR_HSM"], + psf_file["SIGMA_STAR_HSM"], + psf_file["FLAG_STAR_HSM"], + np.ones_like(psf_file["RA"], dtype=int) + * ccd_id, + ] + ).T.tolist(), ) - new_y[m_ccd_id] = ( - psf_file["GLOB_POSITION_IMG_LIST"][:, 1][m_ccd_id] - - y_shift - ) - - header_file_path = ( - self._params["sub_dir_setools"] - + self._params["file_pattern_psfint"] - + f"{exp_name}-{ccd_id}.fits" - ) - try: - header_file = fits.getdata(header_file_path, 1) - except Exception: - continue - header = fits.Header.fromstring( - "\n".join(header_file[0][0]), sep="\n" - ) - wcs = galsim.AstropyWCS(header=header) - - g1_psf_tmp_l = [] - g2_psf_tmp_l = [] - sig_psf_tmp_l = [] - g1_star_tmp_l = [] - g2_star_tmp_l = [] - sig_star_tmp_l = [] - flag_psf_tmp_l = [] - flag_star_tmp_l = [] - - for obj in psf_file[m_ccd_id]: - try: - jac = wcs.jacobian( - world_pos=galsim.CelestialCoord( - ra=obj["RA_LIST"] * galsim.degrees, - dec=obj["DEC_LIST"] * galsim.degrees, - ) - ) - except Exception: - flag_star_tmp_l.append(16) - flag_psf_tmp_l.append(16) - g1_psf_tmp_l.append(0) - g2_psf_tmp_l.append(0) - sig_psf_tmp_l.append(0) - g1_star_tmp_l.append(0) - g2_star_tmp_l.append(0) - sig_star_tmp_l.append(0) - continue - g1_psf_tmp, g2_psf_tmp, sig_psf_tmp = transform_shape( - obj["PSF_MOM_LIST"], jac - ) - - g1_psf_tmp_l.append(g1_psf_tmp) - g2_psf_tmp_l.append(g2_psf_tmp) - sig_psf_tmp_l.append(sig_psf_tmp) - flag_psf_tmp_l.append(obj["PSF_MOM_LIST"][3]) - - g1_star_tmp, g2_star_tmp, sig_star_tmp = ( - transform_shape(obj["STAR_MOM_LIST"], jac) - ) - g1_star_tmp_l.append(g1_star_tmp) - g2_star_tmp_l.append(g2_star_tmp) - sig_star_tmp_l.append(sig_star_tmp) - flag_star_tmp_l.append(obj["STAR_MOM_LIST"][3]) - - new_e1_psf[m_ccd_id] = g1_psf_tmp_l - new_e2_psf[m_ccd_id] = g2_psf_tmp_l - new_sig_psf[m_ccd_id] = sig_psf_tmp_l - new_flag_psf[m_ccd_id] = flag_psf_tmp_l - new_e1_star[m_ccd_id] = g1_star_tmp_l - new_e2_star[m_ccd_id] = g2_star_tmp_l - new_sig_star[m_ccd_id] = sig_star_tmp_l - new_flag_star[m_ccd_id] = flag_star_tmp_l - - exp_cat = np.array( - list( - map( - tuple, - np.array( - [ - new_x, - new_y, - psf_file["RA_LIST"], - psf_file["DEC_LIST"], - new_e1_psf, - new_e2_psf, - new_sig_psf, - psf_file["PSF_MOM_LIST"][:, 3], - new_e1_star, - new_e2_star, - new_sig_star, - psf_file["STAR_MOM_LIST"][:, 3], - new_ccd_id, - psf_file["GLOB_POSITION_IMG_LIST"][:, 0], - psf_file["GLOB_POSITION_IMG_LIST"][:, 1], - ] - ).T.tolist(), - ) - ), - dtype=self._dt_mccd, - ) - cat_list.append(exp_cat) + ), + dtype=self._dt, + ) + cat_list.append(exp_cat) del psf_file @@ -852,10 +262,10 @@ def transform_exposures(self, output_dir, patch, idx, exp_run_dir): return # Finalize catalogue - patch_cat = np.concatenate(cat_list) + star_cat = np.concatenate(cat_list) hdul = fits.HDUList() hdul.append(fits.PrimaryHDU()) - hdul.append(fits.BinTableHDU(patch_cat)) + hdul.append(fits.BinTableHDU(star_cat)) # Write catalogue hdul.writeto( @@ -874,7 +284,6 @@ def run_convert(*args): obj = Convert() obj.set_params_from_command_line(args) - obj.update_params() obj.run() diff --git a/scripts/python/distribute_tiles.py b/scripts/python/distribute_tiles.py deleted file mode 100755 index 338286002..000000000 --- a/scripts/python/distribute_tiles.py +++ /dev/null @@ -1,289 +0,0 @@ -#!/usr/bin/env python -""" -distribute_tiles.py - -Uses canfar.helpers.distributed.chunk to automatically distribute tiles -across replicas, then processes each tile assigned to this replica. -""" - -import os -import sys -import subprocess -from multiprocessing import Pool -import fcntl -from canfar.helpers.distributed import chunk - - -def parse_arguments(args): - """ - Extract the file IDs from command line arguments. - - Args: - args (list of str): Original command line arguments (e.g., sys.argv[1:]) - - Returns: - str or None: The value passed with -f, or None if not present - """ - # Default values - parsed = {"dry_run": 0, "parallel_jobs": 1} - - i = 0 - while i < len(args): - if args[i] == '-f' and i + 1 < len(args): - parsed['file_ids'] = args[i + 1] - i += 2 - elif args[i] == '--batch_num' and i + 1 < len(args): - parsed['batch_num'] = int(args[i + 1]) - i += 2 - elif args[i] == '--batch_tot' and i + 1 < len(args): - parsed['batch_tot'] = int(args[i + 1]) - i += 2 - elif args[i] == '--batch_size' and i + 1 < len(args): - parsed['batch_size'] = int(args[i + 1]) - i += 2 - elif args[i] in ('-n', '--dry_run') and i + 1 < len(args): - parsed['dry_run'] = int(args[i + 1]) - i += 2 - elif args[i] == '--parallel_jobs' and i + 1 < len(args): - parsed['parallel_jobs'] = int(args[i + 1]) - i += 2 - elif args[i] == "--debug_out" and i + 1 < len(args): - parsed["debug_out"] = args[i + 1] - i += 2 - else: - i += 1 - - return parsed - - -def get_my_tiles(all_tiles, batch_num, batch_tot, batch_size): - """ - Calculate which tiles this replica should process based on global position. - Args: - all_tiles: List of all tile IDs - batch_num: Current batch number (1-indexed) - batch_tot: Total number of batches - batch_size: Number of replicas per batch - Returns: - list: Tiles assigned to this replica - - """ - # Get local replica info from environment - local_replica_id = int(os.environ.get('REPLICA_ID', 1)) - - start_idx = (batch_num - 1) * batch_size - end_idx = min(batch_num * batch_size, len(all_tiles)) - - batch_tiles = all_tiles[start_idx:end_idx] - print(f"Batch {batch_num}/{batch_tot}, local replica {local_replica_id}/{batch_size}") - print(f"Batch processes tiles {start_idx + 1} to {end_idx} ({len(batch_tiles)} tiles)") - - # Use chunk() to distribute this batch's tiles among local replicas - # chunk() will use REPLICA_ID and REPLICA_COUNT automatically - my_tiles = list(chunk(batch_tiles)) - - return my_tiles - -def get_tile_list(file_ids): - """Read all tile IDs from file. - - Args: - file_ids: Path to file containing tile IDs - - Returns: - list: List of tile ID strings - """ - with open(file_ids, 'r') as f: - tiles = [line.strip() for line in f if line.strip()] - return tiles - - -def build_process_command(tile_id, original_args): - """ - Build command for a single tile by replacing -f with -e . - - Args: - tile_id (str): Tile ID to process - original_args (list of str): Original CLI arguments - - Returns: - list of str: Command ready for subprocess - """ - cmd = [f"{os.environ['HOME']}/shapepipe/scripts/sh/init_run_exclusive_canfar.sh"] - - # Transform arguments: replace -f with -e , - # skip batch arguments, parallel_jobs, and dry_run if not 0, 1 - i = 0 - while i < len(original_args): - if original_args[i] == '-f' and i + 1 < len(original_args): - cmd.extend(['-e', tile_id]) - i += 2 - elif original_args[i] in ('--batch_num', '--batch_tot', '--batch_size', '--parallel_jobs'): - i += 2 - elif original_args[i] in ('-n', '--dry_run'): - if i + 1 < len(original_args) and original_args[i + 1] in ("0", "1"): - cmd.extend(["-n", original_args[i + 1]]) - i += 2 - else: - cmd.append(original_args[i]) - i += 1 - - return cmd - - -def process_single_tile(args_tuple): - """ - Process a single tile (designed for multiprocessing). - - Args: - args_tuple: Tuple of (tile_id, tile_num, total_tiles, original_args, dry_run, msg_batch) - - Returns: - tuple: (tile_id, success, error_message) - """ - tile_id, tile_num, total_tiles, original_args, dry_run, msg_batch = args_tuple - - print(f"{'='*5} Processing {msg_batch}tile num/total={tile_num}/{total_tiles}: ID={tile_id} {'='*5}") - - # Build command - cmd = build_process_command(tile_id, original_args) - print(f"Command: {' '.join(cmd)}") - - # Execute command - try: - if dry_run != 2: - result = subprocess.run(cmd, check=True, capture_output=True, text=True) - print(f"✓ Successfully processed tile {tile_id}") - return (tile_id, True, None) - else: - print(f"dry_run=2") - return (tile_id, True, None) - except subprocess.CalledProcessError as e: - error_msg = f"Failed to process tile {tile_id}: {e}" - print(f"✗ {error_msg}", file=sys.stderr) - return (tile_id, False, error_msg) - except Exception as e: - error_msg = f"Unexpected error processing tile {tile_id}: {e}" - print(f"✗ {error_msg}", file=sys.stderr) - return (tile_id, False, error_msg) - - -def print_debug(pat, tile_list, out_path, verbose=False): - - local_replica_id = int(os.environ.get('REPLICA_ID', 1)) - - with open(out_path, "a") as f: - # Exclusive lock for save parallel use - fcntl.flock(f.fileno(), fcntl.LOCK_EX) - try: - # Build output string - output = f"{pat} distribute_tiles REPLICA_ID={local_replica_id}, tiles=" - output += " ".join(tile_list) + "\n" - - # Write to file - f.write(output) - - # Optionally write to stdout - if verbose: - print(output, end="") - finally: - # Unlock - fcntl.flock(f.fileno(), fcntl.LOCK_UN) - - -def main(): - - # Debug file line pattern - pat = "- " - - # Parse arguments - args = parse_arguments(sys.argv[1:]) - - if not "file_ids" in args: - print("Error: -f must be provided", file=sys.stderr) - sys.exit(1) - else: - file_ids = args["file_ids"] - - # Read all tiles - print(f"Reading tile list from {file_ids}") - all_tiles = get_tile_list(file_ids) - print(f"Total tiles in file: {len(all_tiles)}") - - print( - "REPLICA_ID, REPLICA_COUNT=", - os.environ.get('REPLICA_ID'), - os.environ.get('REPLICA_COUNT') - ) - - - # Check if we're in multi-batch mode - if "batch_num" in args and "batch_tot" in args and "batch_size" in args: - # Multi-batch mode: calculate global distribution - my_tile_list = get_my_tiles( - all_tiles, - args['batch_num'], - args['batch_tot'], - args['batch_size'] - ) - msg_batch = f"batch {args['batch_num']}/{args['batch_tot']} " - - else: - # Use chunk() to get tiles for this replica - print("Using chunk() to determine tiles for this replica...") - my_tile_list = list(chunk(all_tiles)) - msg_batch = "" - - if "debug_out" in args: - print_debug(f"{pat}{msg_batch}", my_tile_list, args["debug_out"], verbose=True) - - print(f"This replica assigned {len(my_tile_list)} tiles") - print(f"Parallel jobs: {args['parallel_jobs']}") - - if len(my_tile_list) > 0: - print(f"First tile: {my_tile_list[0]}") - if len(my_tile_list) > 1: - print(f"Last tile: {my_tile_list[-1]}") - else: - print("Only one tile") - - # Process each tile assigned to this replica - success_count = 0 - failure_count = 0 - - # Prepare arguments for parallel processing - process_args = [ - (tile_id, i, len(my_tile_list), sys.argv[1:], args["dry_run"], msg_batch) - for i, tile_id in enumerate(my_tile_list, 1) - ] - - if args['parallel_jobs'] > 1: - # Parallel processing - print(f"Processing tiles in parallel with {args['parallel_jobs']} workers") - with Pool(processes=args['parallel_jobs']) as pool: - results = pool.map(process_single_tile, process_args) - else: - # Sequential processing (original behavior) - print("Processing tiles sequentially") - results = [process_single_tile(arg) for arg in process_args] - - # Count successes and failures - for tile_id, success, error_msg in results: - if success: - success_count += 1 - else: - failure_count += 1 - - # Summary - print( - f"{'='*5} Processing completed tot/suc/fail=" - + f"{len(my_tile_list)}/{success_count}/{failure_count}" - ) - - # Exit with error if any failures - if failure_count > 0: - sys.exit(1) - - -if __name__ == '__main__': - main() diff --git a/scripts/python/get_ccds_with_psf.py b/scripts/python/get_ccds_with_psf.py deleted file mode 100755 index e284e2640..000000000 --- a/scripts/python/get_ccds_with_psf.py +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env python3 - -"""GET_CCDS_WITH_PSF - -Obtain list of CCDs (single-exposure single-HDU files) for which valid PSF information -is available. This can serve to create a footprint coverage mask. - -Author: Martin Kilbinger - -""" - -import sys - -from shapepipe.utilities.ccd_psf_handler import CcdPsfHandler - - -def run_ccd_psf_handler(args=None): - """Run CCD PSF Handler. - - Create instance and run the CCD PSF handler. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - # Create instance - obj = CcdPsfHandler() - - return obj.run(args=args) - - -def main(argv=None): - """Main. - - Main program. - - Parameters - ---------- - argv : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - return run_ccd_psf_handler(args=argv) - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/python/get_number_objects.py b/scripts/python/get_number_objects.py deleted file mode 100755 index 98422457c..000000000 --- a/scripts/python/get_number_objects.py +++ /dev/null @@ -1,243 +0,0 @@ -#!/usr/bin/env python - -# -*- coding: utf-8 -*- - -"""Script get_number_objects.py - -Get number of objects in a (last-run SExtractor) catalogue. - -:Author: Martin Kilbinger - -""" - -import sys -import copy -import glob - -from optparse import OptionParser -from astropy.io import fits - -from shapepipe.pipeline.run_log import get_last_dir, get_all_dirs -from shapepipe.utilities import cfis - - -class param: - """General class to store (default) variables""" - - def __init__(self, **kwds): - self.__dict__.update(kwds) - - def print(self, **kwds): - print(self.__dict__) - - def var_list(self, **kwds): - return vars(self) - - -def params_default(): - """Set default parameter values. - - Returns - ------- - class param - parameter values - - """ - p_def = param( - input_path=".", - input_name_base="final_cat", - hdu_num=1, - ) - - return p_def - - -def parse_options(p_def): - """Parse command line options. - - Parameters - ---------- - p_def: class param - parameter values - - Returns - ------- - list - command line options - command line str - - """ - usage = "%prog [OPTIONS]" - parser = OptionParser(usage=usage) - - # IO - parser.add_option( - "-i", - "--input_path", - dest="input_path", - type="string", - default=p_def.input_path, - help=f"input path, default='{p_def.input_path}'", - ) - parser.add_option( - "-n", - "--input_name_base", - dest="input_name_base", - type="string", - default=p_def.input_name_base, - help=f"input name base, default='{p_def.input_name_base}'", - ) - parser.add_option( - "-l", - "--list_tile_ID_path", - dest="tile_ID_list_path", - type="string", - default=None, - help=f"tile ID list, default: Use all data in input files", - ) - - # Control - parser.add_option( - "-p", - "--param_path", - dest="param_path", - type="string", - default=None, - help="parameter file path, default=None", - ) - - parser.add_option( - "", - "--hdu_num", - dest="hdu_num", - type="int", - default=p_def.hdu_num, - help=f"input HDU number, default='{p_def.hdu_num}'", - ) - - parser.add_option( - "-v", - "--verbose", - dest="verbose", - action="store_true", - help="verbose output", - ) - - options, args = parser.parse_args() - - return options, args - - -def check_options(options): - """Check command line options. - - Parameters - ---------- - options: tuple - Command line options - - Returns - ------- - bool - Result of option check. False if invalid option value. - - """ - return True - - -def update_param(p_def, options): - """Return default parameter, updated and complemented according to options. - - Parameters - ---------- - p_def: class param - parameter values - optiosn: tuple - command line options - - Returns - ------- - class param - updated paramter values - - """ - param = copy.copy(p_def) - - # Update keys in param according to options values - for key in vars(param): - if key in vars(options): - setattr(param, key, getattr(options, key)) - - # Add remaining keys from options to param - for key in vars(options): - if not key in vars(param): - setattr(param, key, getattr(options, key)) - - # Do extra stuff if necessary - - return param - - -def main(argv=None): - - # Set default parameters - p_def = params_default() - - # Command line options - options, args = parse_options(p_def) - - if check_options(options) is False: - return 1 - - param = update_param(p_def, options) - - # Save command line arguments to log file - f_log = cfis.log_command(argv, close_no_return=False) - - pattern = "sexcat" - run_log_file = "output/log_run_sp.txt" - - # For v1 - # module = 'sextractor_runner_run_1' - - # For v2 - module = "sextractor_runner" - - # Get all sextractor output directories - all_dir = get_all_dirs(run_log_file, module) - paths = [] - - # Find tile runs - for path in all_dir: - if "run_sp_tile_Sx" in path: - paths.append(path) - paths = sorted(paths) - - if len(paths) == 0: - # No previous tile SExtractor run found - return -1 - - # Get latest run - last_dir = paths[-1] - - # Get all output SExtractor catalogues - file_list = glob.glob(f"{last_dir}/{pattern}*.fits") - if len(file_list) == 0: - raise ValueError(f"No files {last_dir}/{pattern}*.fits found") - - # Add up number of objects over all catalogues - n_obj = 0 - hdu_no = -1 - for fpath in file_list: - hdu_list = fits.open(fpath) - header = hdu_list[-1].header - n_obj += int(header["NAXIS2"]) - - # Compute average - n_obj = int(n_obj / len(file_list)) - - print(n_obj) - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/python/link_to_exp_for_tile.py b/scripts/python/link_to_exp_for_tile.py deleted file mode 100755 index 5b2a5f0e2..000000000 --- a/scripts/python/link_to_exp_for_tile.py +++ /dev/null @@ -1,467 +0,0 @@ -#!/usr/bin/env python - -# -*- coding: utf-8 -*- - -"""Script link_to_exp_for_tile.py - -:Description: Link to exposure and PSF catalogue - for a given tile. - -:Author: Martin Kilbinger - -""" - -import os -import sys -import re -import copy - -from optparse import OptionParser - - -class param: - """General class to store (default) variables""" - - def __init__(self, **kwds): - self.__dict__.update(kwds) - - def print(self, **kwds): - print(self.__dict__) - - def var_list(self, **kwds): - return vars(self) - - -def params_default(): - """Params Default. - - Set default parameter values. - - Returns - ------- - class param - parameter values - - """ - p_def = param( - tile_base_dir=".", - exp_base_dir=".", - sp_local=0, - ) - - return p_def - - -def parse_options(p_def): - """Parse Options. - - Parse command line options. - - Parameters - ---------- - p_def: class param - parameter values - - Returns - ------- - list - command line options - command line str - - """ - usage = "%prog [OPTIONS]" - parser = OptionParser(usage=usage) - - # IO - parser.add_option( - "-i", - "--input_tile_dir", - dest="tile_base_dir", - type="string", - default=p_def.tile_base_dir, - help=f"input tile base directory, default='{p_def.tile_base_dir}'", - ) - parser.add_option( - "-t", - "--tile_ID", - dest="tile_ID", - type="string", - help=f"input tile ID", - ) - parser.add_option( - "-I", - "--input_exp_dir", - dest="exp_base_dir", - type="string", - default=p_def.exp_base_dir, - help=f"input exposure base directory, default='{p_def.exp_base_dir}'", - ) - parser.add_option( - "-s", - "--sp_local", - dest="sp_local", - type="int", - default=p_def.sp_local, - help=f"local runi of split_exposure_runner, default='{p_def.sp_local}'", - ) - parser.add_option( - "-v", - "--verbose", - dest="verbose", - action="store_true", - help="verbose output", - ) - - options, args = parser.parse_args() - - return options, args - - -def check_options(options): - """Check Options. - - Check command line options. - - Parameters - ---------- - options: tuple - Command line options - - Returns - ------- - bool - Result of option check. False if invalid option value. - - """ - return True - - -def update_param(p_def, options): - """Update Param. - - Return default parameter, updated and complemented according to options. - - Parameters - ---------- - p_def: class param - parameter values - optiosn: tuple - command line options - - Returns - ------- - class param - updated paramter values - - """ - param = copy.copy(p_def) - - # Update keys in param according to options values - for key in vars(param): - if key in vars(options): - setattr(param, key, getattr(options, key)) - - # Add remaining keys from options to param - for key in vars(options): - if not key in vars(param): - setattr(param, key, getattr(options, key)) - - # Do extra stuff if necessary - - return param - - -# TODO: move to cs_util -def matching_subdirs(base_dir, pattern): - """Matching Subdirs. - - Return all subdirectories whose name start with a given string pattern. - - Parameters - ---------- - base_dir: str - base directory - pattern: str - pattern to match beginning of subdir name - - Returns - ------- - list - matched subdirectory names - - """ - subdirs = [] - if os.path.exists(base_dir): - # Loop over directory entries - for entry in os.listdir(base_dir): - full_path = os.path.join(base_dir, entry) - # Append if match - if os.path.isdir(full_path) and entry.startswith(pattern): - subdirs.append(full_path) - else: - print(f"Warning: {base_dir} does not exist, continuing...") - - # Sort according to creation date - subdirs.sort(key=os.path.getctime) - - return subdirs - - -def get_tile_out_dir(tile_base_dir, tile_ID): - """Get Tile Out Dir. - - Return output directory path for a given tile ID. - - Parameters - ---------- - tile_base_dir: str - base directory of tile runs - tile_ID: str - tile ID - - Returns - ------- - str - path - - """ - return f"{tile_base_dir}/{tile_ID}/output" - - -def get_exp_IDs(tile_base_dir, tile_ID, verbose=False): - """Get Exp IDs. - - Return exposure IDs used for given tile. - - Parameters - ---------- - tile_base_dir: str - base directory of tile runs - tile_ID: str - tile ID - verbose: bool, optional - verbose output if ``True``; default is ``False`` - - Returns - ------- - list - exposure IDs - - """ - # Get tile output path - tile_out_dir = get_tile_out_dir(tile_base_dir, tile_ID) - - # Get subdirectories with runs "Fe" (find exposures) - pattern = "run_sp_GitFeGie" - subdirs = matching_subdirs(tile_out_dir, pattern) - - # Raise error if not exactly one run - if len(subdirs) == 0: - raise IOError( - f"No matching directory '{pattern}' in {tile_out_dir} found" - ) - if len(subdirs) != 1: - raise IOError( - f"Exactly one directory natching {pattern} in {tile_out_dir} " - + f"expected, not {len(subdirs)}" - ) - - # Replace dot with dash in tile ID - tile_ID_sp = re.sub(r"\.", "-", tile_ID) - # Get output file of find_exposure_runner for this tile ID - exp_ID_file = ( - f"{subdirs[0]}/find_exposures_runner/output/" - + f"exp_numbers-{tile_ID_sp}.txt" - ) - - exp_IDs = [] - # Read file to get exposure IDs - with open(exp_ID_file) as f_in: - for line in f_in: - name = line.strip() - # Remove any letter - ID = re.sub("[a-zA-Z]", "", name) - exp_IDs.append(ID) - - if verbose: - print("Exposures: ", exp_IDs) - return exp_IDs - - -def get_exp_single_HDU_IDs(exp_IDs, n_CPU): - """Get Exp Single HDU IDs. - - Return all single-HDU IDs for given exposures. - - Parameters - ---------- - exp_IDs: list - input exposure IDs, list of str - n_CPU: int - number of CPUs (=HDUs) of an exposure; n_CPU=40 for MegaCAM - - Returns - ------- - list - list of single-HDU exposure IDs - - """ - exp_shdu_IDs = [] - # Loop over input exposure IDs - for exp_ID in exp_IDs: - # Append int up to n_CPU to each - for idx in range(n_CPU): - ID = f"{exp_ID}-{idx}" - exp_shdu_IDs.append(ID) - - return exp_shdu_IDs - - -def get_paths(exp_base_dir, exp_shdu_IDs, pattern): - """Get Paths. - - Return (newest) subdirectory for each input single-exposure HDU ID that matches pattern. - - Parameters - ---------- - exp_base_dir: str - base directory for (single-HDU) exposure runs - exp_shdu_IDs: list - single-HDU exposure IDs; list of str - pattern: str - pattern to match beginning of subdir names - Returns - ------- - list - matching paths; list of str, one entry for each input single-exp ID - - """ - paths = [] - # Loop over single-HDU exposure IDs - for exp_shdu_ID in exp_shdu_IDs: - - # output path of runs - name = f"{exp_base_dir}/{exp_shdu_ID}/output" - path = os.path.abspath(name) - - # get matching subdirs - subdirs = matching_subdirs(path, pattern) - n_subdirs = len(subdirs) - - if n_subdirs != 1: - msg = ( - f"Exactly one directory matching {pattern} in {path} expected," - + f" not {n_subdirs}" - ) - - # If more than one found: sort by name = sort by date - subdirs = sorted(subdirs) - - # No match - if n_subdirs == 0: - continue - - # Append matching subdir; if more than one append newest - paths.append(f"{subdirs[-1]}") - - return paths - - -def create_links_paths(tile_base_dir, tile_ID, paths, verbose=False): - """Create Links Paths. - - Create links to paths. - - Parameters - ---------- - tile_base_dir: str - base directory for tile runs - tile_ID: str - tile ID - paths: list - paths; list of str - verbose: bool, optional - verbose output if ``True``; default is ``False`` - - """ - # Get tile output path - tile_out_dir = get_tile_out_dir(tile_base_dir, tile_ID) - - # Loop over paths - for path in paths: - - # Get destination = tile output dir + path tail (part of path after last slash) - head, tail = os.path.split(path) - src = path - dst = f"{tile_out_dir}/{tail}" - - if os.path.exists(dst): - - src_existing = os.readlink(dst) - if src_existing == src: - # destination already points to source: skip - if verbose: - print( - f"Warning: {src} <- {dst} already exists, no link created" - ) - continue - else: - # destination points to different source: create links with added index to distinguish - # from existing one(s) - idx = 1 - dst_orig = dst - while True: - # Find destination name with lowest appended index that does not exist - dst = f"{dst_orig}_{idx}" - if os.path.exists(dst): - idx += 1 - else: - # Found: create new link - if verbose: - print(f"link {src} <- {dst}") - os.symlink(src, dst) - break - else: - - # destination does not exist: create link - if verbose: - print(f"link {src} <- {dst}") - os.symlink(src, dst) - - -def main(argv=None): - - # Set default parameters - p_def = params_default() - - # Command line options - options, args = parse_options(p_def) - - if check_options(options) is False: - return 1 - - param = update_param(p_def, options) - - tile_base_dir = param.tile_base_dir - exp_base_dir = param.exp_base_dir - tile_ID = param.tile_ID - n_CPU = 40 - verbose = param.verbose - - exp_IDs = get_exp_IDs(tile_base_dir, tile_ID, verbose=verbose) - exp_shdu_IDs = get_exp_single_HDU_IDs(exp_IDs, n_CPU) - - # Note: psfex P3 is mostly run_sp_exp_SxSePsf - patterns = ["run_sp_exp_SxSePsfPi"] # , "run_sp_exp_Pi"] - if param.sp_local == 1: - patterns.append("run_sp_exp_Sp_shdu") - for pattern in patterns: - paths = get_paths(exp_base_dir, exp_shdu_IDs, pattern) - - create_links_paths(tile_base_dir, tile_ID, paths, verbose=verbose) - - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/python/merge_final_cat.py b/scripts/python/merge_final_cat.py deleted file mode 100755 index 9d9c10d21..000000000 --- a/scripts/python/merge_final_cat.py +++ /dev/null @@ -1,396 +0,0 @@ -#!/usr/bin/env python - -# -*- coding: utf-8 -*- - -"""Script merge_final_cat.py - -Merge all final catalogues, created by ShapePipe module -``make_catalogue_runner``, into a joined numpy binary file. - -:Authors: Axel Guinot, Martin Kilbinger - -""" - -from astropy.io import fits -import numpy as np -import os -import sys -import re -import copy - -from optparse import OptionParser - -from shapepipe.utilities import cfis - - -class param: - """General class to store (default) variables""" - - def __init__(self, **kwds): - self.__dict__.update(kwds) - - def print(self, **kwds): - print(self.__dict__) - - def var_list(self, **kwds): - return vars(self) - - -def params_default(): - """Params Default. - - Set default parameter values. - - Returns - ------- - class param - parameter values - - """ - p_def = param( - input_path=".", - input_name_base="final_cat", - hdu_num=1, - ) - - return p_def - - -def parse_options(p_def): - """Parse Options. - - Parse command line options. - - Parameters - ---------- - p_def: class param - parameter values - - Returns - ------- - list - command line options - command line str - - """ - usage = "%prog [OPTIONS]" - parser = OptionParser(usage=usage) - - # IO - parser.add_option( - "-i", - "--input_path", - dest="input_path", - type="string", - default=p_def.input_path, - help=f"input path, default='{p_def.input_path}'", - ) - parser.add_option( - "-n", - "--input_name_base", - dest="input_name_base", - type="string", - default=p_def.input_name_base, - help=f"input name base, default='{p_def.input_name_base}'", - ) - parser.add_option( - "-l", - "--list_tile_ID_path", - dest="tile_ID_list_path", - type="string", - default=None, - help=f"tile ID list, default: Use all data in input files", - ) - - # Control - parser.add_option( - "-p", - "--param_path", - dest="param_path", - type="string", - default=None, - help="parameter file path, default=None", - ) - - parser.add_option( - "", - "--hdu_num", - dest="hdu_num", - type="int", - default=p_def.hdu_num, - help=f"input HDU number, default='{p_def.hdu_num}'", - ) - - parser.add_option( - "-v", - "--verbose", - dest="verbose", - action="store_true", - help="verbose output", - ) - - options, args = parser.parse_args() - - return options, args - - -def check_options(options): - """Check Options. - - Check command line options. - - Parameters - ---------- - options: tuple - Command line options - - Returns - ------- - bool - Result of option check. False if invalid option value. - - """ - return True - - -def update_param(p_def, options): - """Update Param. - - Return default parameter, updated and complemented according to options. - - Parameters - ---------- - p_def: class param - parameter values - optiosn: tuple - command line options - - Returns - ------- - class param - updated paramter values - - """ - param = copy.copy(p_def) - - # Update keys in param according to options values - for key in vars(param): - if key in vars(options): - setattr(param, key, getattr(options, key)) - - # Add remaining keys from options to param - for key in vars(options): - if not key in vars(param): - setattr(param, key, getattr(options, key)) - - # Do extra stuff if necessary - - return param - - -# MKDEBUG TODO: remove this function, duplicate in create_final_cat.py -def read_param_file(path, verbose=False): - """Read Param File. - - Return parameter list read from file. - - Parameters - ---------- - path: str - input file name - verbose: bool, optional, default=False - verbose output if True - - Returns - ------- - list of str - parameter names - - """ - param_list = [] - - if path: - - with open(path) as f: - for line in f: - if line.startswith("#"): - continue - entry = line.rstrip() - if not entry or entry == "": - continue - param_list.append(entry) - - if verbose: - if len(param_list) > 0: - print(f"Copying {len(param_list)} columns", end="") - else: - print("Copying all columns", end="") - print(" into merged catalogue") - - # Check for multiples - multiples = [] - for param in param_list: - if param_list.count(param) > 1: - multiples.append(param) - - if len(multiples) > 0: - print( - "The following parameters are more than one times " - "in the parameter file: ", - end="", - ) - for m in multiples: - print(m, end=" ") - print() - raise ValueError("Multiple identical keys found") - - return param_list - - -def get_data(path, hdu_num, param_list): - """Get Data. - - Return data of selected columns from FITS file. - - Parameters - ---------- - path: str - input file name - hdu_num: int - HDU number - param_list: list of str - parameters to be extracted. If none, copy - all columns - - Returns - ------- - numpy array - data columns - - """ - hdu_list = fits.open(path) - hdu = hdu_list[hdu_num] - - if param_list: - cols = [] - for p in param_list: - cols.append(hdu.columns[p]) - coldefs = fits.ColDefs(cols) - hdu_new = fits.BinTableHDU.from_columns(coldefs) - d = hdu_new.data - else: - d = hdu.data - - return d - - -def main(argv=None): - - - # Set default parameters - p_def = params_default() - - # Command line options - options, args = parse_options(p_def) - - if check_options(options) is False: - return 1 - - param = update_param(p_def, options) - - if param.verbose: - print("Start") - - # Save command line arguments to log file - f_log = cfis.log_command(argv, close_no_return=False) - - path = param.input_path - - if param.verbose: - print("Read parameter file") - param.param_list = read_param_file(param.param_path, verbose=param.verbose) - - # read (optional) input tile ID file - if param.tile_ID_list_path: - if param.verbose: - print("Read tile ID list") - tile_ID_list = cfis.read_list(param.tile_ID_list_path) - - if param.verbose: - print("Find input catalogue FITS files") - l = os.listdir(path=path) - ext = "fits" - lpath = [] - for this_l in l: - - add_this_l = False - - # mark to add if correct extension, matches input pattern, not `.npy` file - if ( - this_l.endswith(ext) - and (f"{param.input_name_base}" in this_l) - and (".npy" not in this_l) - ): - add_this_l = True - - # unmark to add if no in (optional) input tile ID file - if param.tile_ID_list_path: - nix, niy = cfis.get_tile_number(this_l) - tile_ID = f"{nix}.{niy}" - if tile_ID not in tile_ID_list: - add_this_l = False - if add_this_l: - lpath.append(os.path.join(path, this_l)) - - if param.verbose: - print(f"{len(lpath)} files files to merge found") - - count = 0 - - # Determine number of columns and keys from first catalogue file - d_tmp = get_data(lpath[0], param.hdu_num, param.param_list) - d = np.zeros(d_tmp.shape, dtype=d_tmp.dtype) - for key in d_tmp.dtype.names: - d[key] = d_tmp[key] - count = count + 1 - if param.verbose: - print(f"File '{lpath[0]}' copied ({count}/{len(lpath)})") - - # merge remaining catalogue files - for fname in lpath[1:]: - - try: - d_tmp = get_data(fname, param.hdu_num, param.param_list) - dd = np.zeros(d_tmp.shape, dtype=d.dtype) - - for key in d_tmp.dtype.names: - dd[key] = d_tmp[key] - - count = count + 1 - if param.verbose: - print(f"File '{fname}' copied ({count}/{len(lpath)})") - - d = np.concatenate((d, dd)) - except: - print( - f"Error while adding file '{fname}', {len(dd)} objects" - " not in final cat" - ) - - # Save merged catalogue as numpy binary file - if param.verbose: - print("Saving merged catalogue") - np.save(f"{param.input_name_base}.npy", d) - - msg = f"{count} catalog files merged with success" - if param.verbose: - print(msg) - print(msg, file=f_log) - - f_log.close() - - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/python/summary_tiles.py b/scripts/python/summary_tiles.py deleted file mode 100755 index 9addbb017..000000000 --- a/scripts/python/summary_tiles.py +++ /dev/null @@ -1,183 +0,0 @@ -#!/usr/bin/env python - -import sys -import os -import re -import numpy as np - -from tqdm import tqdm - - -def get_input_IDs_patch(tiles, patch): - - # Read in files from summary output with missing CCDs. - # Create and update CCD name set. - fname = f"{patch}/tile_numbers.txt" - with open(fname) as f: - lines = f.readlines() - for line in lines: - ID = line.rstrip("\n") - tiles.append( - { - "ID": ID, - "patch": patch, - } - ) - - return tiles - - -def check_zero_weight(tiles, patches): - - for patch in tqdm(patches, desc="check zero weight"): - - fname = f"{patch}/summary/special_job_16_sextractor_runner_5.txt" - - if os.path.exists(fname): - - with open(fname) as f: - lines = f.readlines() - for line in lines: - m = re.search("process-(\d{3})-(\d{3})\.log", line) - if m: - ID = f"{m[1]}.{m[2]}" - my_tile = next((tile for tile in tiles if tile.get("ID") == ID), None) - my_tile["zero_weight"] = 1 - - return tiles - - -def check_exceptions(tiles, patches): - - for patch in tqdm(patches, desc="check processing exceptions"): - - fname = f"{patch}/exceptions.txt" - - if os.path.exists(fname): - - with open(fname) as f: - lines = f.readlines() - for line in lines: - m = re.search("(\d{3})\.(\d{3})\s+\t+(.*)", line) - if m: - ID = f"{m[1]}.{m[2]}" - my_tile = next((tile for tile in tiles if tile.get("ID") == ID), None) - my_tile[m[3]] = 1 - - return tiles - - -def get_final(tiles, patches): - - n_final = {} - - for patch in tqdm(patches, desc="check processing exceptions"): - - n_final[patch] = 0 - - fname = f"{patch}/n_tiles_final.txt" - if os.path.exists(fname): - with open(fname) as f: - lines = f.readlines() - for line in lines: - num = line.rstrip("\n") - n_final[patch] = int(num) - - n_final["all"] = sum(n_final.values()) - - return n_final - - -def fill_True(tiles, keys): - - - for tile in tiles: - for key in keys: - tile[key] = 0 - - return tiles - - -def summary(patches, keys): - - tiles = [] - - n_final = get_final(tiles, patches) - - # Loop over patches - for patch in tqdm(patches, desc="read input tile lists"): - tiles = get_input_IDs_patch(tiles, patch) - - tiles = fill_True(tiles, keys) - - tiles = check_zero_weight(tiles, patches) - - tiles = check_exceptions(tiles, patches) - - return tiles, n_final - -def print_tile(tile, f_out, keys): - - for key in keys: - print(f"{tile[key]}", file=f_out, end=" ") - print(file=f_out) - - -def output_tiles(tiles, path, keys): - - with open(path, "w") as f_out: - - for tile in tiles: - print_tile(tile, f_out, keys) - - -def print_summary(tiles, patches, n_final, keys, keys_sum, nform): - - for key in keys_sum: - print(key.rjust(nform, " "), end="") - print("tot_miss".rjust(nform, " ")) - - for patch in patches: - tiles_patch = [entry for entry in tiles if entry.get("patch") == patch] - print_summary_patch(tiles_patch, patch, n_final[patch], keys, nform) - print_summary_patch(tiles, "all", n_final["all"], keys, nform) - -def print_summary_patch(tiles, patch, n_final, keys, nform): - - n_tiles = len(tiles) - print(patch.rjust(nform, " "), end="") - print(str(n_tiles).rjust(nform, " "), end="") - print(str(n_final).rjust(nform, " "), end="") - n_all = 0 - for key in keys: - n = sum(v[key] for v in tiles) - print(str(n).rjust(nform, " "), end="") - n_all += n - print(str(n_all).rjust(nform, " "), f"{n_all / n_tiles:10.2%}", end="") - print() - - -def main(argv): - - n_patch = 8 - patches = [f'P{x}' for x in np.arange(n_patch) + 1] - - keys = ["zero_weight", "aborted", "large_num_det"] - - nform = 15 - - tiles, n_final = summary(patches, keys) - - output_path = "tile_summary.txt" - - keys_ext = ["ID", "patch"] + keys - output_tiles(tiles, output_path, keys_ext) - - keys_sum = ["patch", "n_tiles", "n_final"] + keys - print_summary(tiles, patches, n_final, keys, keys_sum, nform) - - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/sh/build_and_plot_coverage_maps.sh b/scripts/sh/build_and_plot_coverage_maps.sh deleted file mode 100755 index b7ea9c8b2..000000000 --- a/scripts/sh/build_and_plot_coverage_maps.sh +++ /dev/null @@ -1,95 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -# Build and plot per-CCD coverage (nexp) maps for a range of catalogue -# versions. Each map counts, per sky pixel, the number of exposures with a -# valid PSF model. The per-version flow is: -# 0. get_ccds_with_psf -> ccds_with_psf_.txt (valid CCD IDs) -# 1. download_headers -> per-exposure header text files -# 2. extract_field_corners-> exp_ra_dec_.txt (per-CCD corners) -# 3. build_coverage_map -> coverage_.x.hsp -# 4. plot_coverage_map -> SGC / NGC plots -# Steps 0-2 are expected to have been run already on canfar (they need -# VOSpace access); this script rebuilds from the extracted per-CCD corners and -# plots. Uncomment the marked lines to run the full chain. - -# Configuration variables - -## Catalogue versions -VERSIONS=("v1.3" "v1.4" "v1.5" "v1.6") - -## Output directory of plots -OUTPUT_DIR="coverage_map_plots" - -# healsparse resolution. nside=131072 gives ~0.1" per pixel, chosen to -# match the UNIONS bit-mask resolution so coverage and mask align pixel- -# wise. CoverageMapBuilder defaults to nside=2048 for lighter-weight -# offline use; override here for the production build. -BUILD_NSIDE=131072 -BUILD_CHANNELS=128 - -# Common parameters -VERBOSE="-v" - -## Colorbar -PLOT_COLORBAR="-C" -PLOT_MIN=1 -PLOT_MAX=5 - -# Plot parameters -## SGC region -SGC_RA_MIN=-20 -SGC_RA_MAX=45 -SGC_DEC_MIN=18 -SGC_DEC_MAX=40 - -## NGC region -NGC_RA_MIN=110 -NGC_RA_MAX=270 -NGC_DEC_MIN=28 -NGC_DEC_MAX=90 - -# Create output directory if it doesn't exist -mkdir -p "${OUTPUT_DIR}" - -# Loop over versions -for VERSION in "${VERSIONS[@]}"; do - echo "Processing ${VERSION}..." - - # Define file paths - CCD_LIST="ccds_with_psf_${VERSION}.txt" - HEADER_DIR="headers_${VERSION}" - INPUT_CORNERS="exp_ra_dec_${VERSION}.txt" - COVERAGE_MAP="coverage_${VERSION}.x.hsp" - - # Step 0-2: build the per-CCD corner file (canfar / VOSpace only). - # Uncomment to run the full chain from scratch. - # get_ccds_with_psf -V "${VERSION}" -o "${CCD_LIST}" ${VERBOSE} - # download_headers -i "${CCD_LIST}" -o "${HEADER_DIR}" ${VERBOSE} - # extract_field_corners -i "${HEADER_DIR}" -l "${CCD_LIST}" \ - # -o "${INPUT_CORNERS}" ${VERBOSE} - - # Build coverage map - echo " Building coverage map from ${INPUT_CORNERS}..." - CMD="build_coverage_map -i ${INPUT_CORNERS} -o ${COVERAGE_MAP} -c ${BUILD_CHANNELS} -n ${BUILD_NSIDE} ${VERBOSE}" - echo "$CMD" - $CMD - - # Plot SGC region - echo " Plotting SGC region..." - CMD="plot_coverage_map -i ${COVERAGE_MAP} -o ${OUTPUT_DIR}/coverage_${VERSION}_SGC.png ${VERBOSE} ${PLOT_COLORBAR} -R ${SGC_RA_MIN} -r ${SGC_RA_MAX} -D ${SGC_DEC_MIN} -d ${SGC_DEC_MAX} -m ${PLOT_MIN} -M ${PLOT_MAX}" - echo "$CMD" - $CMD - - # Plot NGC region - echo " Plotting NGC region..." - CMD="plot_coverage_map -i ${COVERAGE_MAP} -o ${OUTPUT_DIR}/coverage_${VERSION}_NGC.png ${VERBOSE} ${PLOT_COLORBAR} -R ${NGC_RA_MIN} -r ${NGC_RA_MAX} -D ${NGC_DEC_MIN} -d ${NGC_DEC_MAX} -m ${PLOT_MIN} -M ${PLOT_MAX}" - echo "$CMD" - $CMD - - echo " Done with ${VERSION}" - echo -done - -echo "All versions processed successfully!" diff --git a/scripts/sh/canfar_async_job.sh b/scripts/sh/canfar_async_job.sh deleted file mode 100755 index 7128e7734..000000000 --- a/scripts/sh/canfar_async_job.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash - -source $HOME/shapepipe/scripts/sh/functions.sh - - -# 1. Extract the line from -f file -# Save original arguments -ARGS=("$@") - -# Find -f argument and get its value; remove -f and -S -FILE_IDS="" -NEW_ARGS=() -i=0 -while [[ $i -lt $# ]]; do - arg="${ARGS[i]}" - case "$arg" in - -f) - ((i++)) - FILE_IDS="${ARGS[i]}" - ;; - -S) - ((i++)) # skip next argument (value of -S) - ;; - *) - NEW_ARGS+=("$arg") - ;; - esac - ((i++)) -done - -if [[ -z "$FILE_IDS" ]]; then - echo "Error: -f must be provided" - exit 1 -fi - -# Get the line for this replica -LINE=$(get_line_from_file -f "$FILE_IDS") - -# 2. Call init_run.sh with remaining args + -e line -$HOME/shapepipe/scripts/sh/init_run_exclusive_canfar.sh "${NEW_ARGS[@]}" -e "$LINE" diff --git a/scripts/sh/combine_runs.bash b/scripts/sh/combine_runs.bash deleted file mode 100755 index 91b2ca42a..000000000 --- a/scripts/sh/combine_runs.bash +++ /dev/null @@ -1,224 +0,0 @@ -#!/usr/bin/env bash - -# Name: combine_runs.bash -# Description: Create new shapepipe run directory with -# links to source files from combined existing runs -# Author: Martin Kilbinger - - -# Command line arguments - -## Default values -cat='final' -psf="mccd" - -## Help string -usage="Usage: $(basename "$0") [OPTIONS] -\n\nOptions:\n - -h\tthis message\n - -p, --psf MODEL\n - \tPSF model, allowed are 'psfex', 'mccd', 'setools', default='$psf'\n - -c, --cat TYPE\n - \tCatalogue type, allowed are 'final', 'flag_tile', 'flag_exp', \n - \t'psf', 'psf_conv', 'image', 'shdu', default='$cat'\n -" - -## Parse command line -while [ $# -gt 0 ]; do - case "$1" in - -h) - echo -ne $usage - exit 0 - ;; - -c|--cat) - cat="$2" - shift - ;; - -p|--psf) - psf="$2" - shift - ;; - *) - echo -ne $usage - exit 1 - ;; - esac - shift -done - - -## Check options -if [ "$cat" != "final" ] \ - && [ "$cat" != "flag_tile" ] \ - && [ "$cat" != "tile_detection" ] \ - && [ "$cat" != "flag_exp" ] \ - && [ "$cat" != "psf" ] \ - && [ "$cat" != "psf_conv" ] \ - && [ "$cat" != "image" ] \ - && [ "$cat" != "shdu" ]; then - echo "cat (option -c) needs to be 'final', 'tile_detection', 'flag_tile', 'flag_exp', 'psf', 'psf_conv', 'shdu', or 'image'" - exit 2 -fi - -## Check options -if [ "$psf" != "psfex" ] \ - && [ "$psf" != "mccd" ] \ - && [ "$psf" != "setools" ]; then - echo "PSF (option -p) needs to be 'psfex' or 'mccd'" - exit 2 -fi - - -## Functions -function link_s () { - target=$1 - link_name=$2 - - if [ -L "$link_name" ]; then - echo "link with name $link_name already exists, skipping..." - let "n_skipped+=1" - else - echo "create link $target <- $link_name" - ln -s $target $link_name - let "n_created+=1" - fi -} - - -# Start program - -n_skipped=0 -n_created=0 - -pwd=`pwd` -out_base="output" - -# Set paths: -## run_out: target output new run directory -## run_in: source input run base directory -## module: source input module runner sub-directory -## pattern: source file pattern - -run_out="run_sp_combined_$cat" - -if [ "$cat" == "final" ]; then - - # v1 - #run_in="$pwd/$out_base/run_sp_tile_Mc_*" - # v2 - run_in="$pwd/tile_runs/*/$out_base/run_sp_tile_Mc_*" - - module="make_catalog_runner" - pattern="final_cat-*" - -elif [ "$cat" == "tile_detection" ]; then - - run_in="$pwd/P?/tile_runs/*/$out_base/run_sp_tile_Sx_*" - module="sextractor_runner" - pattern="sexcat-*" - -elif [ "$cat" == "flag_tile" ]; then - - # v1 - #run_in="$pwd/$out_base/run_sp_MaMa_*/mask_runner_run_1" - # v2 - run_in="$pwd/$out_base/run_sp_tile_Ma_*" - run_out="run_sp_Ma_tile" - - module="mask_runner" - pattern="pipeline_flag-*" - -elif [ "$cat" == "flag_exp" ]; then - - # v1 - #run_in="$pwd/$out_base/run_sp_MaMa_*/mask_runner_run_2" - # v2 - run_in="$pwd/$out_base/run_sp_exp_Ma_*" - run_out="run_sp_Ma_exp" - - module="mask_runner" - pattern="pipeline_flag-*" - -elif [ "$cat" == "image" ]; then - - run_in="$pwd/$out_base/run_sp_Git_*" - module="get_images_runner" - pattern="CFIS_image-*" - -elif [ "$cat" == "psf" ]; then - - #MKDEBUG TODO: add option - # v1 - #run_in="$pwf/$out_base/run_sp_exp_Pi_*" - # v2 - #run_in="$pwd/exp_runs/*/$out_base/run_sp_exp_Pi_*" - run_in="$pwd/exp_runs/*/$out_base/run_sp_exp_SxSePsfPi_*" - - pattern="validation_psf-*" - if [ "$psf" == "psfex" ]; then - module="psfex_interp_runner" - elif [ "$psf" == "setools" ]; then - module="setools_runner" - else - module="mccd_interp_runner" - fi - -elif [ "$cat" == "psf_conv" ]; then - - #run_in="$pwd/../P?" - run_in="$pwd" - pattern="validation_psf_conv-*" - module="psfex_interp_runner" - -elif [ "$cat" == "shdu" ]; then - - run_in="$pwd/$out_base/run_sp_exp_Sp_shdu_*" - module="split_exp_runner" - pattern="headers-*" - -else - - echo "Invalid catalogue type $cat" - exit 2 - -fi - - -OUTPUT="$pwd/$out_base/$run_out" -mkdir -p $OUTPUT - - -# Create links - -## target directory -outdir=$OUTPUT/$module/output -mkdir -p $outdir - -## identify source files - -# The following can result in an "Argument list too long" error -#FILES=(`find $run_in -type f -name "$pattern" -print0 | xargs -0 echo`) - -i=0 -for dir in $run_in; do - FILES=(`find -L $dir -type f -name "$pattern" -print0 | xargs -0 echo`) - - #echo "$dir $pattern" - - ## Look over source files - for file in ${FILES[@]}; do - - target=$file - link_name=$outdir/`basename $file` - link_s $target $link_name - ((i=i+1)) - - done - -done - -#echo " $n_files target files, $i links created/skipped" -echo " $i total, "$n_skipped skipped, "$n_created links created" - -# Update log file -update_runs_log_file.py diff --git a/scripts/sh/functions.sh b/scripts/sh/functions.sh index f6763c5b0..777f1a557 100644 --- a/scripts/sh/functions.sh +++ b/scripts/sh/functions.sh @@ -1,97 +1,12 @@ -# Global variables -SSL=~/.ssl/cadcproxy.pem -SESSION=https://ws-uv.canfar.net/skaha/v0/session -IMAGE=images.canfar.net/unions/shapepipe -NAME=shapepipe +# Shell helper sourced by run_job_sp_canfar_v2.0.bash. +# +# command() reads $VERBOSE, $debug_out, $pat and $STOP as free variables set +# by its caller; the two defaults below cover callers that set neither. -version="2.0" -cmd_remote="$HOME/shapepipe/scripts/sh/run_job_sp_canfar_v2.0.bash" pat="---- " STOP=0 -get_line_from_file() { - local file="" - local OPTIND opt - - # Parse options: look for -f - while getopts ":f:" opt; do - case $opt in - f) - file="$OPTARG" - ;; - \?) - echo "Unknown option: -$OPTARG", ", continuing" - #return 1 - ;; - :) - echo "Option -$OPTARG requires an argument." - return 1 - ;; - esac - done - - # Check that file and REPLICA_ID are set - if [[ -z "$file" ]]; then - echo "Error: -f option not provided." - return 1 - fi - if [[ -z "$REPLICA_ID" ]]; then - echo "Error: REPLICA_ID not set." - return 1 - fi - - # Check file exists - if [[ ! -f "$file" ]]; then - echo "Error: file '$file' not found." - return 1 - fi - - # Extract the line corresponding to REPLICA_ID (1-based index) - sed -n "${REPLICA_ID}p" "$file" -} - - - -# Add session and image IDs to log files -function update_session_logs() { - echo $my_session >> session_IDs.txt - echo "$my_session $ID" >> session_image_IDs.txt -} - -function call_curl() { - my_name=$1 - my_job=$2 - my_psf=$3 - my_ID=$4 - my_N_SMP=$5 - my_dry_run=$6 - my_dir=$7 - my_debug_out=$8 - my_scratch=$9 - my_test_arg=${10} - - my_arg="-j $my_job -p $my_psf -e $my_ID -N $my_N_SMP -n $my_dry_run -d $my_dir --debug_out $my_debug_out -S $my_scratch $my_test_arg" - - if [ "$my_dry_run" == "0" ]; then - my_session=`curl -E $SSL "$SESSION?$RESOURCES" -d "image=$IMAGE:$version" -d "name=${my_name}" -d "cmd=$cmd_remote" --data-urlencode "args=${my_arg[@]}"` - fi - - cmd=("curl" "-E" "$SSL" "$SESSION?$RESOURCES" "-d" "image=$IMAGE:$version" "-d" "name=${my_name}" "-d" "cmd=$cmd_remote" "--data-urlencode" "args=\"${my_arg}\"") - - if [ -n "$my_debug_out" ]; then - echo "${pat}call_curl $my_name $my_arg" >> $my_debug_out - echo "${pat}Running ${cmd[@]} (dry_run=$my_dry_run)" >> $my_debug_out - fi - echo "${cmd[@]} (dry_run=$my_dry_run)" - - - # Running $cmd does not work due to unknown problems with passing of args - - update_session_logs -} - - ## Print string, executes command, and prints return value. function command () { cmd=$1 @@ -142,57 +57,3 @@ function command () { fi fi } - - -function get_kind_from_job() { - my_job=$1 - - job_to_test=2 - kind="none" - - # loop over possible job numbers - while [ $job_to_test -le 1024 ]; do - - (( do_job = $job & $job_to_test )) - if [[ $do_job != 0 ]]; then - - if [ $job_to_test == 32 ]; then - if [ "$kind" == "tile" ]; then - echo "Error: Invalid job $job. mixing tile and exp kinds" - exit 6 - fi - - # job=32 -> set kind to exp - kind="exp" - elif [ $job_to_test == 2 ]; then - if [ "$kind" == "tile" ]; then - echo "Error: Invalid job $job. mixing tile and exp kinds" - exit 6 - fi - - kind="exp" - elif [ $job_to_test == 8 ]; then - if [ "$kind" == "tile" ]; then - echo "Error: Invalid job $job. mixing tile and exp kinds" - exit 6 - fi - - kind="exp" - else - if [ "$kind" == "exp" ]; then - echo "Error: Invalid job $job. mixing tile and exp kinds" - exit 6 - fi - - # job != 32 -> set kind to tile - kind="tile" - fi - - fi - - # Multiply job number by two to get next bitwise number - job_to_test=$((job_to_test * 2)) - done - - echo $kind -} diff --git a/scripts/sh/init_run_exclusive_canfar.sh b/scripts/sh/init_run_exclusive_canfar.sh deleted file mode 100755 index 556153b0d..000000000 --- a/scripts/sh/init_run_exclusive_canfar.sh +++ /dev/null @@ -1,642 +0,0 @@ -#!/bin/bash - -# init_run_exclusive_canfar.sh - -# Command line arguments -## Default values -job=-1 -ID=-1 -N_SMP=1 -dry_run=0 -dir=`pwd` -debug_out=-1 -scratch=-1 -fix=0 -test_only=0 -sm=1 - -# mh_local is 0 (1) if merge_header_runner is run on all exposures, -# which is standard so far (run on exposures of given tile only; new) -mh_local=1 - -# sp_local is 0 (1) is split_headers_runner and mask_runner is run -# on all exposures (locally). Not 100% automatic yet. -sp_local=1 -VERBOSE=1 - -pat="-- " - - -## Help string -usage="Usage: $(basename "$0") -j JOB -e ID [OPTIONS] -\n\nOptions:\n - -h\tthis message\n - -j, --job JOB\tRUnning JOB, bit-coded\n - -e, --exclusive ID - \timage ID\n - -p, --psf MODEL\n - \tPSF model, one in ['psfex'|'mccd'], default='$psf'\n - -m, --mh_local MH\n - \tmerge header file local (MH=1) or global (MH=0); default is $mh_local\n - -s, --sp_local SP\n - \tsplit local run local (SP=1) or global (SP=0); default is $sp_local\n - --sm SM\n - \tWith (SM=1; default) or without (SM=0) spread model input\n - -N, --N_SMP N_SMOp\n - \tnumber of jobs (SMP mode only), default from original config files\n - -d, --directory\n - \trun directory, default is pwd ($dir)\n - -S, --scratch\n - \tprocessing scratch directory, default is None ($scratch)\n - -F, --fix FIX\n - \tfix missing data (re-download tile, unzip) for FIX=1; default is $fix\n - -n, --dry_run LEVEL\n - \tdry run (LEVEL=1), no actual processing; default is $dry_run\n - --debug_out PATH\n - \tdebug output file PATH, default not used\n - --test\n - \ttest mode, no processing\n -" - -## Help if no arguments -if [ -z $1 ]; then - echo -ne $usage - exit 1 -fi - -## Parse command line -while [ $# -gt 0 ]; do - case "$1" in - -h) - echo -ne $usage - exit 0 - ;; - -j|--job) - job="$2" - shift - ;; - -e|--exclusive) - ID="$2" - shift - ;; - -p|--psf) - psf="$2" - shift - ;; - -m|--mh_local) - mh_local="$2" - shift - ;; - -s|--sp_local) - sp_local="$2" - shift - ;; - --sm) - sm="$2" - shift - ;; - -N|--N_SMP) - N_SMP="$2" - shift - ;; - -d|--directory) - dir="$2" - shift - ;; - -S|--scratch) - scratch="$2" - shift - ;; - -n|--dry_run) - dry_run="$2" - shift - ;; - -F|--fix) - fix="$2" - shift - ;; - --debug_out) - debug_out="$2" - shift - ;; - --test) - test_only=1 - ;; - esac - shift -done - - -# functions -function message() { - msg=$1 - my_debug_out=$2 - my_exit=$3 - - echo $msg - if [ "$my_debug_out" != "-1" ]; then - echo ${pat}$msg >> $my_debug_out - fi - - if [ "$my_exit" != "-1" ]; then - echo "${pat}exiting with code $my_exit" >> $my_debug_out - exit $my_exit - fi -} - -# Write an updated copy of a shapepipe config with NUMBER_LIST set to the -# given image ID, expressed in the numbering scheme (leading dash, dots -> -# dashes). Replaces the retired shapepipe_run -e/--exclusive flag (#746). -function set_config_number_list() { - local config_orig=$1 - local config_upd=$2 - local _id=$3 - - local number="-$(echo $_id | tr '.' '-')" - local config_tmp="${config_upd}.tmp" - - if grep -q "^NUMBER_LIST" "$config_orig"; then - perl -pe 's/^NUMBER_LIST\s*=.*/NUMBER_LIST = '$number'/' "$config_orig" > "$config_tmp" - else - perl -pe 's/^\[FILE\][ \t]*$/[FILE]\nNUMBER_LIST = '$number'/' "$config_orig" > "$config_tmp" - fi - if ! grep -q "^NUMBER_LIST = $number$" "$config_tmp"; then - echo "set_config_number_list: failed to set NUMBER_LIST in $config_orig" >&2 - exit 1 - fi - mv "$config_tmp" "$config_upd" -} - - -# Init message -message "test=$test_only" $debug_out -1 -if [ "$test_only" == "1" ]; then - echo "MKDEBUG test" - msg="init_run_exclusive.py script test mode, exiting." - ex=0 -else - msg="init_run_exclusive.py script processing mode, starting." - ex=-1 -fi -message "$msg" $debug_out $ex - - -## Check options -message "checking options" $debug_out -1 - -if [ "$job" == "-1" ]; then - message "No job indicated, use option -j" $debug_out 2 -fi - -if [ "$ID" == "-1" ]; then - message "No image ID indicated, use option -e" $debug_out 3 -fi - -if [ "$psf" != "psfex" ] && [ "$psf" != "mccd" ]; then - message "PSF (option -p) needs to be 'psfex' or 'mccd'" $debug_out 4 -fi - -if [ "$mh_local" != "0" ] && [ "$mh_local" != "1" ]; then - message "mh_local (option -m) needs to be 0 or 1" $debug_out 5 -fi - -if [ "$sp_local" != "0" ] && [ "$sp_local" != "1" ]; then - message "sp_local (option -m) needs to be 0 or 1" $debug_out 6 -fi - -if [ "$dry_run" != "0" ] && [ "$dry_run" != 1 ]; then - message "dry_run must be 0 or 1, not $dry_run" $debug_out 8 -fi - - -# Start script - -source $HOME/shapepipe/scripts/sh/functions.sh - -msg="Starting $(basename "$0") `date` ID=$ID" -message "$msg" $debug_out -1 -#message "`date`" $debug_out -1 -#message "ID=$ID" $debug_out -1 - -# Set kind -kind=$(get_kind_from_job $job) - -if [ "$kind" == "none" ]; then - message "Error: invalid job $job" $debug_out 5 -fi - -message "kind=$kind" $debug_out -1 - - -if [ "$dry_run" == "1" ]; then - message "running in dry run mode" $debug_out -1 -else - message "not running in dry run mode" $debug_out -1 -fi - -CONDA_PREFIX=$HOME/.conda/envs/shapepipe -PATH=$PATH:$CONDA_PREFIX/bin - -cd $dir - -if [ ! -d ${kind}_runs ]; then - command "mkdir ${kind}_runs" $dry_run -fi - -if [ "$fix" == "1" ]; then - message "Fixing missing data" $debug_out -1 - - message "Download tile images" $debug_out -1 - cd data_tiles - file_name=CFIS.$ID.r.fits - if [ ! -e $file_name ]; then - message "Downloading $file_name" $debug_out -1 - vcp vos:cfis/tiles_DR5/$file_name . - else - message "File $file_name exists, skipping" $debug_out -1 - fi - file_name=CFIS.$ID.r.weight.fits.fz - if [ ! -e $file_name ]; then - message "Downloading $file_name" $debug_out -1 - vcp vos:cfis/tiles_DR5/$file_name . - else - message "File $file_name exists, skipping" $debug_out -1 - fi - cd .. - - message "link to tiles..." $debug_out -1 - cd output/run_sp_GitFeGie_202*/get_images_runner_run_1/output - IDt=`echo $ID | tr "." "-"` - path_tiles=/arc/home/kilbinger/cosmostat/v2/pre_v2/$psf/$patch/data_tiles - link_name=CFIS_image-$IDt.fits - if [ ! -e $link_name ]; then - message "Creating link $link_name" $debug_out -1 - # force to overwrite broken link - ln -sf $path_tiles/CFIS.$ID.r.fits $link_name - else - message "Link $link_name exists, skipping" $debug_out -1 - fi - - link_name=CFIS_weight-$IDt.fitsfz - if [ ! -e $link_name ]; then - message "Creating link $link_name" $debug_out -1 - ln -sf $path_tiles/CFIS.$ID.r.weight.fits.fz $link_name - else - message "Link $link_name exists, skipping" $debug_out -1 - fi - - command "cd ../../../.." $dry_run - - message "Unzip weight ($dry_run)" $debug_out -1 - command "cd tile_runs/$ID" $dry_run - export SP_RUN=`pwd` - command "set_config_number_list cfis/config_tile_Uz.ini config_tile_Uz_upd.ini $ID" $dry_run - command "shapepipe_run -c config_tile_Uz_upd.ini" $dry_run - - cd $dir -else - message "Not fixing missing data" $debug_out -1 -fi - - -cd ${kind}_runs - -if [ ! -d "$ID" ]; then - command "mkdir $ID" $dry_run -fi - -cd $ID -pwd - -# Point cfis to local link, to be independent of platform -ln -sf ~/shapepipe/example/cfis - - -if [ ! -d "output" ]; then - command "mkdir output" $dry_run -fi - -cd output - - -# Update links to global run directories (GiFeGie) -# New: 27/11/2024: Remove link to Uz, conflict with fix -for my_dir in $dir/output/run_sp_[G]*; do - command "ln -sf $my_dir" $dry_run -done - -# The following could be done also with fix=1 -if [ "$fix" == "0" ] && [ "$kind" == "tile" ]; then - if [ ! -e "run_sp_Uz*" ]; then - - message "previous uncompress run not found..." $debug_out -1 - - # Remove broken links - for my_dir in run_sp_Uz*; do - if [ -L $my_dir ]; then - command "rm $my_dir" $dry_run - message "remove broken link $my_dir" $debug_out -1 - fi - done - - # Create working link - for my_dir in $dir/output/run_sp_Uz*; do - command "ln -sf $my_dir" $dry_run - message "create valid link $my_dir" $debug_out -1 - done - cd .. - command "update_runs_log_file.py" $dry_run - cd output - fi -fi - - -# Combined flags - -## Tiles -command "ln -sf $dir/output/run_sp_Ma_tile" $dry_run - -if [ "$sp_local" == "0" ]; then - # Exposures - command "ln -sf $dir/output/run_sp_Ma_exp" $dry_run -#else - #command "rm -f $dir/output/run_sp_Ma_exp" $dry_run -fi - -# Check for existing exp SpMh dir -dir_SpMh="run_sp_exp_SpMh" -if [[ -d "$dir/output/$dir_SpMh" ]]; then - # create link - command "ln -sf $dir/output/$dir_SpMh" $dry_run -else - message "No global SpMh directory found" $debug_out -1 - if [ -L "$dir_SpMh" ] && [ ! -e "$dir_SpMh" ]; then - message "Remove broken link $dir_SpMh" $debug_out -1 - rm $dir_SpMh - fi -fi - -(( do_job = $job & 2 )) -if [ $do_job != 0 ] && [ "$sp_local" == "1" ]; then - message "run local sp" $debug_out -1 - command "rm -rf run_sp_Gie*" $dry_run - command "rm -rf run_sp_exp_Sp*" $dry_run - - # Create new get_image dir - new_dir="run_sp_Gie/get_images_runner/output" - command "rm -rf $new_dir" $dry_run - command "mkdir -p $new_dir" $dry_run - - # Link to image, weight, and flag file of current exposure - - # Remove HDU extension - command "cd $new_dir" $dry_run - exp_ID=$(echo "$ID" | sed 's/-[0-9]\{1,2\}//') - for file in $dir/output//run_sp_GitFeGie_20*/get_images_runner_run_2/output/*${exp_ID}.* ; do - echo $file - command "ln -s $file" $dry_run - done - command "cd ../../.." $dry_run - - # Run Sp - command "cd .." $dry_run - if [ "$scratch" != "-1" ]; then - command "cd ../.." $dry_run - command "mkdir -p $scratch/exp_runs" $dry_run - command "cp -R exp_runs/$ID $scratch/exp_runs" $dry_run - command "cd $scratch/exp_runs/$ID" $dry_run - fi - command "update_runs_log_file.py" $dry_run - export SP_RUN=`pwd` - command "set_config_number_list cfis/config_exp_Sp.ini config_exp_Sp_upd.ini $exp_ID" $dry_run - command "shapepipe_run -c config_exp_Sp_upd.ini" $dry_run - - # Only keep CCD of this ID - command "mkdir -p output/run_sp_exp_Sp_shdu/split_exp_runner/output" $dry_run - command "mv output/run_sp_exp_Sp/split_exp_runner/output/*$ID.* output/run_sp_exp_Sp_shdu/split_exp_runner/output" $dry_run - command "mv output/run_sp_exp_Sp/split_exp_runner/output/headers* output/run_sp_exp_Sp_shdu/split_exp_runner/output" $dry_run - command "rm -rf output/run_sp_exp_Sp" $dry_run - if [ "$scratch" != "-1" ]; then - command "mv output/run_sp_exp_Sp_shdu $dir/exp_runs/$ID/output" $dry_run - command "cd .." $dry_run - command "rm -rf $ID" $dry_run - command "cd $dir" $dry_run - fi - command "update_runs_log_file.py" $dry_run - cd output - - if [ "$job" == "2" ]; then - msg="Finishing $(basename "$0") after job=2 `date` ID=$ID" - message "$msg" $debug_out 0 - fi - -fi - -if [ "$kind" == "tile" ] && [ "$sp_local" == "1" ]; then - echo "New (for P9): skipping link_to_exp_for_tile.py" - cd ../../.. - #command "link_to_exp_for_tile.py -t $ID -i tile_runs -I exp_runs -s $sp_local" $dry_run - cd tile_runs/$ID - #command "combine_runs.bash -p psfex -c shdu" $dry_run - cd output -fi - - -if [ "$mh_local" == "0" ]; then - if [ ! -f log_exp_headers.sqlite ]; then - # Global Mh and file does not exist -> symlink to - # gllobal mh file - message "creating global link to exp headers (mh_local=0)" $debug_out -1 - command "ln -s $dir/output/log_exp_headers.sqlite" $dry_run - else - message "global link to exp headers (mh_local=0) exists" $debug_out -1 - fi -else - # Local Mh - message "not creating global link to exp headers (mh_local=1)" $debug_out -1 - if [ "$ID" == "-1" ]; then - message "ID needs to be given (option -e) for mh_local" $debug_out 6 - fi - - # Check and remove symbolic (global) mh file link - if [ -L log_exp_headers.sqlite ]; then - # Local Mh and symlink -> remove previous link to - # (potentially incomplete) global mh file - message "Removing previous mh sym link" $debug_out -1 - command "rm log_exp_headers.sqlite" $dry_run - else - message "no mh link found" $debug_out -1 - fi - - # Check size of existing header file - if [ -e log_exp_headers.sqlite ]; then - size=$(stat -c %s log_exp_headers.sqlite) - if (( size > 15000 )); then - message "Found valid local mh file, continuing" $debug_out -1 - else - message "Existing local mh file looks invalid, deleting" $debug_out -1 - rm -f log_exp_headers.sqlite - fi - fi - - if [ ! -e log_exp_headers.sqlite ]; then - message "Creating local mh file" $debug_out -1 - - cd .. - command "update_runs_log_file.py" $dry_run - export SP_RUN=`pwd` - command "shapepipe_run -c cfis/config_exp_Mh.ini" $dry_run - cd output - else - message "Found local mh file, continuing" $debug_out -1 - fi -fi - - -(( do_job = $job & 8 )) -if [ $do_job != 0 ] && [ "$sp_local" == "1" ]; then - # Remove previous local Ma runs - message "cdsclient = $CDSCLIENT" $debug_out -1 - command "rm -rf run_sp_exp_Ma*" $dry_run -fi - -(( do_job = $job & 16 )) -if [[ $do_job != 0 ]]; then - # Remove previous Sx runs - command "rm -rf run_sp_tile_Sx_*" $dry_run -fi - -# Update links to exposure run directories, which were created in job 32 -(( do_job = $job & 64 )) -if [[ $do_job != 0 ]]; then - if [ "$kind" == "tile" ]; then - - # Remove previous runs of this job - rm -rf run_sp_tile_PsViSmVi* - fi -fi - -(( do_job = $job & 128 )) -if [[ $do_job != 0 ]]; then - - echo - - cat_ngmix="run_sp_tile_ngmix_Ng1u/ngmix_runner/output/ngmix-*.fits" - dir_ngmix_prev="run_sp_tile_ngmix_Ng1u_prev/ngmix_runner/output" - cat_ngmix_prev="$dir_ngmix_prev/ngmix-*.fits" - - # Remove if empty - if [ ! -s $cat_ngmix ]; then - echo "Removing empty file $cat_ngmix" - rm $cat_ngmix - fi - if [ ! -s $cat_ngmix_prev ]; then - echo "Removing empty file $cat_ngmix_prev" - rm $cat_ngmix_prev - fi - - # Check whether ngmix output exists - if [ -e $cat_ngmix ]; then - message "ngmix output catalogue exists" $debug_out -1 - - # Check whether previous ngmix directory and output cat exist - exists="1" - if [ ! -d $dir_ngmix_prev ]; then - exists="0" - elif [ ! -e "$cat_ngmix_prev" ]; then - exists="1" - fi - if [ "$exists" == "0" ]; then - message "Moving to previous batch-save dir (does not exist yet)" $debug_out -1 - command "mkdir -p $dir_ngmix_prev" $dry_run - command "mv $cat_ngmix $dir_ngmix_prev" $dry_run - else - # Compare file sizes - size_cat_ngmix=$(stat -c%s $cat_ngmix) - size_cat_ngmix_prev=$(stat -c%s $cat_ngmix_prev) - if [ "$size_cat_ngmix" -gt "$size_cat_ngmix_prev" ]; then - message "Moving to batch-save dir, overwriting smaller batch-save cat" $debug_out -1 - command "mv $cat_ngmix $dir_ngmix_prev" $dry_run - else - message "Previous batch-save dir not smaller, removing ngmix output" $debug_out -1 - command "rm $cat_ngmix" $dry_run - fi - fi - else - # Whether or not previous ngmix exists, job_sp_canfar will handle it - message "No ngmix output exists, continuing..." $debug_out -1 - fi - - echo -fi - -(( do_job = $job & 256 )) -if [[ $do_job != 0 ]]; then - - # Remove previous runs of this job - rm -rf run_sp_tile_Ms_20??-* - -fi - -(( do_job = $job & 512 )) -if [[ $do_job != 0 ]]; then - - # Remove previous runs of this job - rm -rf run_sp_tile_Mc_20??-* - -fi - - -cd .. - -# Update log file -command update_runs_log_file.py $dry_run - -echo -n "pwd: " -pwd - -echo -n "environment: " -echo $CONDA_PREFIX - -# To avoid (new?) qt error with setools (-j 32) -export DISPLAY=:1.0 - - -if [ "$scratch" != "-1" ]; then - # Copy inputs to scratch - command "mkdir -p $scratch/${kind}_runs" $dry_run - cd ../.. - command "pwd" $dry_run - command "cp -R ${kind}_runs/$ID $scratch/${kind}_runs" $dry_run - command "cd $scratch/${kind}_runs/$ID" $dry_run -fi - -command "job_sp_canfar.bash -p psfex -j $job -e $ID --n_smp $N_SMP --nsh_jobs $N_SMP --debug_out $debug_out --sm $sm " $dry_run - -if [ "$scratch" != "-1" ]; then - cd ../.. - if [ "$job" == "16" ]; then - command "mv ${kind}_runs/$ID/output/run_sp_Sx_* $dir/${kind}_runs/$ID/output" $dry_run - elif [ "$job" == "32" ]; then - command "mv ${kind}_runs/$ID/output/run_sp_exp_SxSe* $dir/${kind}_runs/$ID/output" $dry_run - elif [ "$job" == "64" ]; then - command "mv ${kind}_runs/$ID/output/run_sp_tile_PsViSm** $dir/${kind}_runs/$ID/output" $dry_run - elif [ "$job" == "128" ]; then - command "mv ${kind}_runs/$ID/output/run_sp_tile_ngmix* $dir/${kind}_runs/$ID/output" $dry_run - else - echo "scratch mode with job=$job not implemented yet, exiting..." - exit 8 - fi - - command "rm -rf ${kind}_runs/$ID" $dry_run - command "cd $dir/${kind}_runs/$ID" $dry_run - command "update_runs_log_file.py" $dry_run - command "cd $dir" $dry_run -fi - -cd $dir - -#msg="End $(basename "$0")" - -msg="Finished $(basename "$0") `date` ID=$ID" -message "$msg" $debug_out -1 diff --git a/scripts/sh/job_sp_canfar.bash b/scripts/sh/job_sp_canfar.bash deleted file mode 100755 index e64ed1e5f..000000000 --- a/scripts/sh/job_sp_canfar.bash +++ /dev/null @@ -1,610 +0,0 @@ -#!/bin/bash - -# Name: job_sp_canfar.bash -# Description: General script to process one or more tiles -# with all contributing exposures. -# This works as job submission script for -# the canfar batch system. -# called in interactive mode on a virtual -# machine. -# Author: Martin Kilbinger - - -# Command line arguments -## Default values -job=255 -config_dir=$HOME/shapepipe/example/cfis -psf='mccd' -retrieve='vos' -star_cat_for_mask='onthefly' -exclusive='' -results='cosmostat/kilbinger/results_v2' -n_smp=-1 -nsh_jobs=8 -debug_out=-1 -sm=1 - -pat="--- " - -## Help string -usage="Usage: $(basename "$0") [OPTIONS] [TILE_ID] -\n\nOptions:\n - -h\tthis message\n - -j, --job JOB\tRunning JOB, bit-coded\n - \t 1: retrieve images (online if method=vos)\n - \t 2: prepare images (offline)\n - \t 4: mask tiles (online if star_cat_for_mask=onthefly)\n - \t 8: mask exposures (online if star_cat_for_mask=onthefly)\n - \t 16: detection of galaxies on tiles (offline)\n - \t 32: processing of stars on exposures (offline)\n - \t 64: galaxy selection on tiles (offline)\n - \t 128: shapes and morphology (offline)\n - \t 256: paste catalogues (offline)\n - -c, --config_dir DIR\n - \t config file directory, default='$config_dir'\n - -p, --psf MODEL\n - \tPSF model, one in ['psfex'|'mccd'], default='$psf'\n - -r, --retrieve METHOD\n - \tmethod to retrieve images, allowed are 'vos', 'symlink', default='$retrieve'\n - -s, --star_cat_for_mask\n - \tcatalogue for masking bright stars, allowed are 'onthefly', 'save',\n - \tdefault is '${star_cat_for_mask}'\n - --sm SM\n - \tWith (SM=1; default) or without (SM=0) spread model input\n - -e, --exclusive ID\n - \texclusive input filer number string ID (default: None)\n - -o, --output_dir\n - \toutput (upload) directory on vos:cfis, default='$results'\n - -n, --n_smp N_SMP\n - \tnumber of jobs (SMP mode only), default from original config files\n - --nsh_jobs NJOB\n - \tnumber of objects per parallel shape module call, \n - \tdefault: optimal number is computed\n - --debug_out PATH\n - \tdebug output file PATH, default not used\n - TILE_ID_i\n - \ttile ID(s), e.g. 283.247 214.242, only with '-j 1'\n -" - -## Help if no arguments -if [ -z $1 ]; then - echo -ne $usage - exit 1 -fi - -## Parse command line -while [ $# -gt 0 ]; do - case "$1" in - -h) - echo -ne $usage - exit 0 - ;; - -j|--job) - job="$2" - shift - ;; - -c|--config_dir) - config_dir="$2" - shift - ;; - -p|--psf) - psf="$2" - shift - ;; - -r|--retrieve) - retrieve="$2" - shift - ;; - -s|--star_cat_for_mask) - star_cat_for_mask="$2" - shift - ;; - --sm) - sm="$2" - shift - ;; - -e|--exclusive) - exclusive="$2" - shift - ;; - -o|--output_dir) - results="$2" - shift - ;; - -n|--n_smp) - n_smp="$2" - shift - ;; - --nsh_jobs) - nsh_jobs="$2" - shift - ;; - --debug_out) - debug_out="$2" - shift - ;; - esac - shift -done - -## Check options -if [ "$psf" != "psfex" ] && [ "$psf" != "mccd" ]; then - echo "PSF (option -p) needs to be 'psfex' or 'mccd'" - exit 2 -fi - -if [ "$star_cat_for_mask" != "onthefly" ] && [ "$star_cat_for_mask" != "save" ]; then - echo "Star cat for mask (option -s) needs to be 'onthefly' or 'save'" - exit 4 -fi - -if [ "$retrieve" != "vos" ] && [ "$retrieve" != "symlink" ]; then - echo "method to retrieve images (option -r) needs to be 'vos' or 'symlink'" - exit 5 -fi - -if [ "$debug_out" != "-1" ]; then - echo $pat`date` >> $debug_out - echo "${pat}Starting $(basename "$0")" >> $debug_out -fi - -CONDA_PREFIX=/arc/home/kilbinger/.conda/envs/shapepipe -PATH=$PATH:$CONDA_PREFIX/bin - -# For tar archives. TODO: Should be unique to each job -export ID="test" - -## Paths - -## Path variables used in shapepipe config files - -# Run path and location of input image directories -export SP_RUN=`pwd` - -# Config file path -export SP_CONFIG=$SP_RUN/cfis -export SP_CONFIG_MOD=$SP_RUN/cfis_mod - -## Other variables - -# Output -OUTPUT=$SP_RUN/output - -# For tar archives -output_rel=`realpath --relative-to=. $OUTPUT` - -# Stop on error, default=1 -STOP=1 - -# Verbose mode (1: verbose, 0: quiet) -VERBOSE=1 - -# VCP options -export CERTFILE=$HOME/.ssl/cadcproxy.pem -export VCP="vcp --certfile=$CERTFILE" - - -## Functions - -# Print string, executes command, and prints return value. -function command () { - cmd=$1 - str=$2 - - RED='\033[0;31m' - GREEN='\033[0;32m' - NC='\033[0m' # No Color - # Color escape characters show up in log files - #RED='' - #GREEN='' - #NC='' - - - if [ "$debug_out" != "-1" ]; then - echo "${pat}pwd = `pwd`" >> $debug_out - echo "${pat}SP_RUN = $SP_RUN" >> $debug_out - echo "${pat}SP_CONFIG = $SP_CONFIG" >> $debug_out - fi - - if [ $# == 2 ]; then - if [ $VERBOSE == 1 ]; then - echo "$str: running '$cmd'" - fi - if [ "$debug_out" != "-1" ]; then - echo "${pat}Running $cmd" >> $debug_out - fi - - $cmd - - else - if [ $VERBOSE == 1 ]; then - echo "$str: running '$cmd $4 \"$5 $6\"'" - fi - if [ "$debug_out" != "-1" ]; then - echo "${pat}Running $cmd $4 \"$5 $6\"" >> $debug_out - fi - - $cmd $4 "$5 $6" - - fi - - res=$? - - if [ "$debug_out" != "-1" ]; then - echo "${pat}exit code = $res" >> $debug_out - fi - - if [ $VERBOSE == 1 ]; then - if [ $res == 0 ]; then - echo -e "${GREEN}success, return value = $res${NC}" - else - echo -e "${RED}error, return value = $res${NC}" - if [ $STOP == 1 ]; then - echo "${RED}exiting 'canfar_sp.bash', error in command '$cmd'${NC}" - exit $res - else - echo "${RED}continuing 'canfar_sp.bash', error in command '$cmd'${NC}" - fi - fi - fi -} - -# Run shapepipe command. If error occurs, upload sp log files before stopping script. -function command_sp() { - local cmd=$1 - local str=$2 - - command "$1" "$2" -} - -# Set up config file and call shapepipe_run -function command_cfg_shapepipe() { - local config_name=$1 - local str=$2 - local _n_smp=$3 - local _exclusive=$4 - - config_upd=$(set_config_n_smp $config_name $_n_smp) - - # Run a single image ID via NUMBER_LIST in an updated config copy; - # replaces the retired shapepipe_run -e/--exclusive flag (#746) - if [ "$_exclusive" != "" ]; then - set_config_number_list "$config_upd" "$SP_CONFIG_MOD/$config_name" "$_exclusive" - config_upd="$SP_CONFIG_MOD/$config_name" - fi - - local cmd="shapepipe_run -c $config_upd" - command_sp "$cmd" "$str" -} - -# Tar and upload files to vos -function upload() { - base=$1 - shift - ID=$1 - shift - verbose=$1 - shift - upl=("$@") - - echo "Counting upload files" - n_upl=(`ls -l ${upl[@]} | wc`) - if [ $n_upl == 0 ]; then - if [ $STOP == 1 ]; then - echo "Exiting script, no file found for '$base' tar ball" - exit 3 - fi - fi - tar czf ${base}_${ID}.tgz ${upl[@]} - command "$VCP ${base}_${ID}.tgz vos:cfis/$results" "Upload tar ball" -} - -# Upload log files -function upload_logs() { - id=$1 - verbose=$2 - - upl="$output_rel/*/*/logs $output_rel/*/logs" - upload "logs" "$id" "$verbose" "${upl[@]}" -} - -function set_config_n_smp() { - local config_name=$1 - local _n_smp=$2 - - local config_orig="$SP_CONFIG/$config_name" - - if [[ $_n_smp != -1 ]]; then - # Update SMP batch size - local config_upd="$SP_CONFIG_MOD/$config_name" - update_config $config_orig $config_upd "SMP_BATCH_SIZE" $_n_smp - else - # Keep original config file - local config_upd=$config_orig - fi - - # Set "return" value (stdout) - echo "$config_upd" -} - -# Update config file -function update_config() { - local config_orig=$1 - local config_upd=$2 - local key=$3 - local val_upd=$4 - - cat $config_orig \ - | perl -ane 's/'$key'\s+=.+/'$key' = '$val_upd'/; print' > $config_upd -} - -# Write an updated copy of a shapepipe config with NUMBER_LIST set to the -# given image ID, expressed in the numbering scheme (leading dash, dots -> -# dashes). Replaces the retired shapepipe_run -e/--exclusive flag (#746). -function set_config_number_list() { - local config_orig=$1 - local config_upd=$2 - local _id=$3 - - local number="-$(echo $_id | tr '.' '-')" - local config_tmp="${config_upd}.tmp" - - if grep -q "^NUMBER_LIST" "$config_orig"; then - perl -pe 's/^NUMBER_LIST\s*=.*/NUMBER_LIST = '$number'/' "$config_orig" > "$config_tmp" - else - perl -pe 's/^\[FILE\][ \t]*$/[FILE]\nNUMBER_LIST = '$number'/' "$config_orig" > "$config_tmp" - fi - if ! grep -q "^NUMBER_LIST = $number$" "$config_tmp"; then - echo "set_config_number_list: failed to set NUMBER_LIST in $config_orig" >&2 - exit 1 - fi - mv "$config_tmp" "$config_upd" -} - -### Start ### - -echo "Start processing" - -# Create input and output directories -mkdir -p $SP_RUN -cd $SP_RUN -mkdir -p $OUTPUT -mkdir -p $SP_CONFIG_MOD - -# Processing - - -### Retrieve config files -if [[ $config_dir == *"vos:"* ]]; then - command_sp "$VCP $config_dir ." "Retrieve shapepipe config files" -else - if [[ ! -L cfis ]]; then - command_sp "ln -s $config_dir cfis" "Retrieve shapepipe config files" - fi -fi - - -## Retrieve config files and images (online if retrieve=vos) -## Retrieve and save star catalogues for masking (if star_cat_for_mask=save) -(( do_job = $job & 1 )) -if [[ $do_job != 0 ]]; then - - ### Retrieve files - command_cfg_shapepipe \ - "config_GitFeGie_$retrieve.ini" \ - "Retrieve images" \ - -1 \ - $exclusive - - ### Retrieve and save star catalogues for masking - if [ "$star_cat_for_mask" == "save" ]; then - #### For tiles - mkdir $SP_RUN/star_cat_tiles - command_sp \ - "create_star_cat $SP_RUN/output/run_sp_GitFeGie_*/get_images_runner_run_1/output $SP_RUN/star_cat_tiles" \ - "Save star cats for masking (tile)" - - #### For single-exposures - mkdir $SP_RUN/star_cat_exp - command_sp \ - "create_star_cat $SP_RUN/output/run_sp_GitFeGie_*/get_images_runner_run_2/output $SP_RUN/star_cat_exp exp" \ - "Save star cats for masking (exp)" - fi - -fi - -## Prepare images (offline) -(( do_job = $job & 2 )) -if [[ $do_job != 0 ]]; then - - ### Uncompress tile weights - command_cfg_shapepipe "config_tile_Uz.ini" "Run shapepipe (uncompress tile weights)" $n_smp $exclusive - - ### Split images into single-HDU files, merge headers for WCS info - command_cfg_shapepipe \ - "config_exp_SpMh.ini" \ - "Run shapepipe (split images, merge headers)" \ - $n_smp \ - $exclusive - -fi - -## Mask tiles: add star, halo, and Messier object masks (online if "star_cat_for_mask" is "onthefly") -(( do_job = $job & 4 )) -if [[ $do_job != 0 ]]; then - - ### Mask tiles - command_cfg_shapepipe \ - "config_tile_Ma_$star_cat_for_mask.ini" \ - "Run shapepipe (mask tiles)" \ - $n_smp \ - $exclusive - -fi - -## Mask exposures: add star, halo, and Messier object masks (online if "star_cat_for_mask" is "onthefly") -(( do_job = $job & 8 )) -if [[ $do_job != 0 ]]; then - - ### Mask exposures - command_cfg_shapepipe \ - "config_exp_Ma_$star_cat_for_mask.ini" \ - "Run shapepipe (mask exposures)" \ - $n_smp \ - $exclusive - -fi - - -## Remaining exposure processing (offline) -(( do_job = $job & 16 )) -if [[ $do_job != 0 ]]; then - - ### Object detection on tiles - command_cfg_shapepipe \ - "config_tile_Sx.ini" \ - "Run shapepipe (tile detection)" \ - $n_smp \ - $exclusive - -fi - -## Exposure processing (offline) -(( do_job = $job & 32 )) -if [[ $do_job != 0 ]]; then - - ### Star detection, selection, PSF model. setools can exit with an error for CCD with insufficient stars, - ### the script should continue - STOP=0 - command_cfg_shapepipe \ - "config_exp_${psf}.ini" \ - "Run shapepipe (exp $psf)" \ - $n_smp \ - $exclusive - STOP=1 - -fi - -## Process tiles up to shape measurement -(( do_job = $job & 64 )) -if [[ $do_job != 0 ]]; then - - ### PSF model letter: 'P' (psfex) or 'M' (mccd) - letter=${psf:0:1} - Letter=${letter^} - command_cfg_shapepipe \ - "config_tile_${Letter}iViSmVi_canfar.ini" \ - "Run shapepipe (tile PsfInterp=$Letter}: up to ngmix+galsim)" \ - $n_smp \ - $exclusive - -fi - -## Shape measurement (offline) -(( do_job = $job & 128 )) -if [[ $do_job != 0 ]]; then - - ### Prepare config files - n_min=0 - n_obj=`get_number_objects` - if [ "$n_obj" == "-1" ]; then - echo "No tile SExtractor run found, exiting after et_number_objects call" - exit 10 - fi - nsh_step=`echo "$(($n_obj/$nsh_jobs))"` - - n_max=$((nsh_step - 1)) - for k in $(seq 1 $nsh_jobs); do - cat $SP_CONFIG/config_tile_Ng_template_batch.ini | \ - perl -ane \ - 's/(ID_OBJ_MIN =) X/$1 '$n_min'/; s/(ID_OBJ_MAX =) X/$1 '$n_max'/; s/NgXu/Ng'$k'u/; s/X_interp/'$psf'_interp/g; print' \ - > $SP_CONFIG_MOD/config_tile_Ng${k}u.ini - n_min=$((n_min + nsh_step)) - if [ "$k" == $((nsh_jobs - 1)) ]; then - n_max=-1 - else - n_max=$((n_min + nsh_step - 1)) - fi - done - - ### Shapes, run $nsh_jobs parallel processes - VERBOSE=0 - for k in $(seq 1 $nsh_jobs); do - - # if output dir for subrun exists but no output: re-run - ngmix_run=$OUTPUT/"run_sp_tile_ngmix_Ng${k}u/ngmix_runner" - if [ -e "$ngmix_run" ]; then - ngmix_out="$ngmix_run/output" - n_out=`ls -rlt $ngmix_out | wc -l` - if [ "$n_out" -lt 2 ]; then - command \ - "rm -rf $OUTPUT/run_sp_tile_ngmix_Ng${k}u" \ - "Re-running existing empty ngmix subrun $k" - command_sp \ - "shapepipe_run -c $SP_CONFIG_MOD/config_tile_Ng${k}u.ini" \ - "Run shapepipe (tile: ngmix $k)" & - else - echo "Skipping existing non-empty ngmix subrun $k" - fi - else - command_sp \ - "shapepipe_run -c $SP_CONFIG_MOD/config_tile_Ng${k}u.ini" \ - "Run shapepipe (tile: ngmix $k)" & - fi - done - wait - VERBOSE=1 - -fi - -## Create final catalogues (offline) -(( do_job = $job & 256 )) -if [[ $do_job != 0 ]]; then - - cat $SP_CONFIG/config_merge_sep_cats_template.ini | \ - perl -ane \ - 's/(N_SPLIT_MAX =) X/$1 '$nsh_jobs'/; print' \ - > $SP_CONFIG_MOD/config_tile_Ms.ini - - ### Merge separated shapes catalogues - command_sp \ - "shapepipe_run -c $SP_CONFIG_MOD/config_tile_Ms.ini" \ - "Run shapepipe (tile: merge sep cats)" \ - "$VERBOSE" \ - "$ID" -fi - -(( do_job = $job & 512 )) -if [[ $do_job != 0 ]]; then - - # spread_model suffix for config file with or without SM input - if [ "$sm" == "0" ]; then - suff_sm="_nosm" - else - suff_sm="" - fi - - ### Merge all relevant information into final catalogue - command_cfg_shapepipe \ - "config_make_cat_$psf${suff_sm}.ini" \ - "Run shapepipe (tile: create final cat $psf)" \ - $n_smp \ - $exclusive - -fi - -# MKDEBUG: Putting Mh at the end for now, could be integrated before 16. -(( do_job = $job & 1024 )) -if [[ $do_job != 0 ]]; then - - command_cfg_shapepipe \ - "config_exp_Mh.ini" \ - "Run shapepipe (merge exp headers)" \ - $n_smp \ - $exclusive - -fi - -if [ "$debug_out" != "-1" ]; then - echo "${pat}End $(basename "$0") ID=$exclusive success" >> $debug_out -fi diff --git a/scripts/sh/job_sp_canfar_v2.0.bash b/scripts/sh/job_sp_canfar_v2.0.bash index 592cae70d..e0fb28a01 100755 --- a/scripts/sh/job_sp_canfar_v2.0.bash +++ b/scripts/sh/job_sp_canfar_v2.0.bash @@ -1,6 +1,6 @@ #!/bin/bash -# Name: job_sp_canfar.bash +# Name: job_sp_canfar_v2.0.bash # Description: General script to process one or more tiles # with all contributing exposures. # This works as job submission script for diff --git a/scripts/sh/post_proc_sp.bash b/scripts/sh/post_proc_sp.bash deleted file mode 100755 index 930d5b79d..000000000 --- a/scripts/sh/post_proc_sp.bash +++ /dev/null @@ -1,86 +0,0 @@ -#!/usr/bin/env bash - -# Name: post_proc_sp.bash -# Description: Post-process downloaded canfar results and -# creates final merge catalog. -# Author: Martin Kilbinger -# Date: 06/2020 -# Package: shapepipe - -# The shapepipe python virtual environment needs to -# be active to run this script. - - -# Command line arguments - -## Default values -psf='mccd' - -## Help string -usage="Usage: $(basename "$0") [OPTIONS] -\n\nOptions:\n - -h\tthis message\n - -p, --psf MODEL\n - \tPSF model, one in ['psfex'|'mccd'], default='$psf'\n -" - -## Parse command line -while [ $# -gt 0 ]; do - case "$1" in - -h) - echo -ne $usage - exit 0 - ;; - -p|--psf) - psf="$2" - shift - ;; - *) - echo -ne $usage - exit 1 - ;; - esac - shift -done - -## Check options -if [ "$psf" != "psfex" ] && [ "$psf" != "mccd" ]; then - echo "PSF (option -p) needs to be 'psfex' or 'mccd'" - exit 2 -fi - -# Paths -export SP_RUN=. -SP_BASE=$HOME/astro/repositories/github/shapepipe -SP_CONFIG=$SP_BASE/example/cfis - - -# To download results from canfar, use -# -# canfar_download_results.sh -# -# On candide this needs to be done on -# the login node. - -# To Un-tar all .tgz results files, use -# -# $SP_BASE/scripts/sh/untar_results.sh - - -# PSF - -## Collect all psfinterp results -combine_runs -p $psf -t psf - -## Merge all psfinterp results and compute PSF residuals -shapepipe_run -c $SP_CONFIG/config_MsPl_$psf.ini - - -# Galaxies - -## Prepare output directory with links to all 'final_cat' result files -combine_runs - -## Merge final output files to single mother catalog -input_final=output/run_sp_combined/make_catalog_runner/output -merge_final_cat -i $input_final -p $SP_CONFIG/final_cat.param -v diff --git a/scripts/sh/remove_duplicates_tiles.sh b/scripts/sh/remove_duplicates_tiles.sh deleted file mode 100755 index 6f789fe6e..000000000 --- a/scripts/sh/remove_duplicates_tiles.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/usr/bin/env bash - -# Script to remove duplicate tile runs, keeping only the newest -# Run from directory containing P?/tile_runs subdirectories - -count=0 -total_removed=0 - -echo "Starting duplicate tile run removal..." -echo "Looking for patches in P?/tile_runs structure..." -echo "" - -# Process each patch directory -for patch_dir in P*/tile_runs; do - if [ ! -d "$patch_dir" ]; then - continue - fi - - patch_name=$(dirname "$patch_dir") - echo "Processing patch: $patch_name" - - # Process each tile directory within the patch - for dir in "$patch_dir"/*; do - if [ ! -d "$dir" ]; then - continue - fi - - # Check each of the three run types - for run_type in run_sp_tile_PsViSmVi run_sp_tile_Mc run_sp_tile_Ms; do - if compgen -G "$dir/output/${run_type}*" > /dev/null; then - n=$(ls -dt "$dir/output/${run_type}"* 2>/dev/null | wc -l) - if [ "$n" -gt 1 ]; then - ((n_remove=n-1)) - echo " Found $n copies of $run_type in $(basename "$dir"), removing $n_remove oldest" - - # Remove all but the newest - ls -dt "$dir/output/${run_type}"* | tail -n "$n_remove" | while read old_run; do - echo " Removing: $(basename "$old_run")" - rm -rf "$old_run" - ((total_removed++)) - done - - ((count++)) - fi - fi - done - done - echo "" -done - -echo "========================================" -echo "Summary:" -echo " Processed tile directories: $count" -echo " Total run directories removed: $total_removed" -echo "========================================" diff --git a/src/shapepipe/canfar/canfar_submit.py b/src/shapepipe/canfar/canfar_submit.py deleted file mode 100644 index 685584b99..000000000 --- a/src/shapepipe/canfar/canfar_submit.py +++ /dev/null @@ -1,576 +0,0 @@ -# canfar_submit_job.py - -# Submit job with the canfar client - -import os -import sys -import asyncio -import math - -from canfar.sessions import Session -from canfar.sessions import AsyncSession -from datetime import datetime - -from cs_util import args as cs_args -from cs_util import logging - - -class Job(object): - """Class Job. - - Handles job submission with the canfar system. - - """ - - def __init__(self): - - self.params_default() - - self._patch = os.environ["patch"] if "patch" in os.environ else "P0" - - # Maximum replicas per batch (CANFAR limit is 512) - self._max_replicas_per_batch = 512 - if self._max_replicas_per_batch != 512: - print(f"Setting the maximum number of replicas to the non-standard {self._max_replicas_per_batch}") - - # Default number of cores, can be overwritten by -c or -P (parallel jobs) options - self._cores_default = 2 - - - def set_params_from_command_line(self, args): - """Set Params From Command line. - - Only use when calling using python from command line. - Does not work from ipython or jupyter. - - """ - # Read command line options - options = cs_args.parse_options( - self._params, - self._short_options, - self._types, - self._help_strings, - ) - self._params = options - - # Save calling command - logging.log_command(args) - - def params_default(self): - """Params Default. - - Set default parameters. - - """ - self._params = { - "job": -1, - "exclusive": None, - "file_IDs": None, - "psf": "psfex", - "n": "0", - "debug_out": None, - "dry_run": 0, - "log": "log_jobs.txt", - "sync": "async", - "cores": 2, - "ram": 4, - "parallel_jobs": 1, - "jobs_per_session": 0, - "stats": False, - "version": "1.1", - } - - self._short_options = { - "job": "-j", - "exclusive": "-e", - "file_IDs": "-f", - "psf": "-p", - "dry_run": "-n", - "log": "-l", - "sync": "-S", - "cores": "-c", - "ram": "-r", - "parallel_jobs": "-P", - "jobs_per_session": "-J", - "stats": "-s", - "version": "-V", - } - - self._types = { - "job": "int", - "cores": "int", - "ram": "int", - "parallel_jobs": "int", - "jobs_per_session": "int", - "stats": "bool", - } - - self._help_strings = { - "job": "Running JOB, bit-coded, default is {}", - "exclusive": "Run exclusively given image", - "file_IDs": "file containing IDs", - "psf": "PSF model, allowed are 'psfex' and 'mccd'; default is {}", - "dry_run": "if dry run > 0 no actual processing; allowed are 2, 1, 0; default is {}", - "debug_out": "debug output file path, default:set automatically", - "log": "output log file with job and tile IDs", - "sync": "job submission mode, allowed are async, sync; default is {}", - "cores": "number of CPU cores per replica; default is {} or overwritten by -P", - "ram": "RAM in GB per replica; default is {}", - "parallel_jobs": "number of tiles to process in parallel per replica; default is {}", - "jobs_per_session": ( - "number of jobs (tiles) per session; if 0, create one session per job (default is {});" - + " Number of replicas is #FILE_IDs / JOBS_PER_SESSION" - ), - "stats": "print stats of sessions/batches/replicas and exit", - "version": "shapepipe image version; default is {}", - } - - def update_params(self): - """Update Params. - - Update parameters. - """ - - # Set version and image - version = self._params["version"] - self._image = f"images.canfar.net/unions/shapepipe:{version}" - - # If not given, set number of cores to number of parallel jobs; - # else set to default. - if self._params["cores"] == -1: - if self._params["parallel_jobs"] != 1: - self._params["cores"] = self._params["parallel_jobs"] - else: - self._params["cores"] = self._cores_default - - def check_params(self): - """Check Params. - - Check command line arguments and parameters. - - Returns - ------- - bool - status of checks; True (False) if passed (failed) - - """ - if self._params["job"] == -1: - raise ValueError(f"need to specify job (option -j)") - - if self._params["sync"] == "async" and not self._params["file_IDs"]: - raise ValueError("asynchronous mode only possible when tile_IDs file given (-f)") - - if not self._params["exclusive"] and not self._params["file_IDs"]: - raise ValueError("No image ID(s) indicated, use option -e ID or -f file_IDs") - - def set_options_base(self): - """Set Options Base. - - Set basic options from command line options and parameters. - - """ - opt = {} - - cwd = os.getcwd() - - # asynchronous mode - if self._params["sync"] == "async": - opt["-f"] = f"-f {cwd}/{self._params['file_IDs']}" - - # Options - opt["j"] = f"-j {self._params['job']}" - opt["p"] = f"-p {self._params['psf']}" - - # Set identifier for debug file name - if self._params["exclusive"]: - deb_ID = self._params["exclusive"] - else: - deb_ID = os.path.basename(self._params["file_IDs"]) - - if self._params['debug_out']: - opt["debug_out"] = f"--debug_out {cwd}/debug/{self._params['debug_out']}" - else: - opt["debug_out"] = f"--debug_out {cwd}/debug/debug_{deb_ID}.txt" - - opt["d"] = f"-d {cwd}" - opt["m_s"] = "-m 1 -s 1" if self._patch in ("P8", "P9") else "" - if self._params["dry_run"] != 0: - opt["n"] = f"-n {self._params['dry_run']}" - if self._params["parallel_jobs"] > 1: - opt["P"] = f"--parallel_jobs {self._params['parallel_jobs']}" - - options = "" - for key in opt: - options = f"{options} {opt[key]}" - - self._options_base = options - - def set_job_name(self, suf=None): - """Set Job Name. - - Set job name as it will appear on the canfar submission system. - - Parameters - ---------- - suf : str, optional - suffix, default is ``None`` - - """ - if suf is None: - suf = self._params["exclusive"] - - suf1 = suf.replace(".", "-") - - job_name = f"sp-{self._patch}-j{self._params['job']}-{suf1}" - - return job_name - - def set_command(self, mode): - """Set Command. - - Set shell command to run. - - Parameters - ---------- - mode: str - job submission mode, allowed are "async_single" (for asynchronous - submission of single job), "async_bulk" (asynchronous submission of many jobs), - "sync" (synchronous submission of one or more jobs) - - """ - if mode == "async_single": - self._cmd = f"{os.environ['HOME']}/shapepipe/scripts/sh/canfar_async_job.sh" - elif mode == "async_bulk": - self._cmd = f"{os.environ['HOME']}/shapepipe/scripts/python/distribute_tiles.py" - elif mode == "sync": - self._cmd = f"{os.environ['HOME']}/shapepipe/scripts/sh/init_run_exclusive_canfar.sh" - - def get_tile_IDs(self): - """Get Tile IDs. - - Return tile ID information. - - Returns - ------- - str - tile_ID(s) - int - length of tile ID list - str - suffix - - """ - if self._params["exclusive"]: - tile_IDs = [self._params["exclusive"]] - suf = None - else: - with open(self._params["file_IDs"], "r") as f: - tile_IDs = [line.strip() for line in f] - suf = "" - - return tile_IDs, len(tile_IDs), suf - - async def run_async(self): - """Run Async. - - Run asynchronous job submission. - - """ - async with AsyncSession() as session: - - _, total_n, suf = self.get_tile_IDs() - - # Calculate number of replicas (sessions) based on jobs_per_session - if self._params["jobs_per_session"] > 0: - # Each replica processes multiple jobs - num_replicas = math.ceil(total_n / self._params["jobs_per_session"]) - print(f"Distributing {total_n} jobs across {num_replicas} sessions") - print(f"Each session will process ~{self._params['jobs_per_session']} jobs") - else: - # Old behavior: one replica per job - num_replicas = total_n - print(f"Creating {num_replicas} sessions (one per job)") - - # Calculate number of batches needed - num_batches = math.ceil(num_replicas / self._max_replicas_per_batch) - - if num_batches == 1: - # Submit single batch - self.set_command(mode="async_bulk") - print(f"Submitting {num_replicas} replicas in a single batch") - return await self._submit_single_batch(num_replicas, total_n) - else: - # Submit multiple batches as chunks - self.set_command(mode="async_bulk") - print(f"Splitting into {num_batches} batches (max {self._max_replicas_per_batch} replicas per batch)") - return await self._submit_multiple_batches(num_replicas, total_n, num_batches) - - async def _submit_single_batch(self, num_replicas, total_n): - """Submit Single Batch. - - Submit a single batch of jobs. - - Parameters - ---------- - num_replicas: int - number of replicas (sessions) to create - total_n: int - total number of jobs (tiles) to process - - Returns - ------- - list - IDs of submitted sessions - """ - print(f"Submitting {num_replicas} replicas to process {total_n} jobs") - - job_name = self.set_job_name(suf="") - options = self._options_base.lstrip() - print(f"Running '{self._cmd} {options}'") - - async with AsyncSession() as session: - try: - sessions = await session.create( - name=job_name, - image=self._image, - cmd=self._cmd, - args=options, - replicas=num_replicas, - cores=self._params["cores"], - ram=self._params["ram"], - ) - print(f"✓ Batch submitted successfully: {len(sessions)} sessions created") - print(f"Each session will process ~{math.ceil(total_n / num_replicas)} jobs using chunk()") - print("Sessions = ", sessions) - except Exception as e: - print(f"❌ CANFAR session.create() failed: {type(e).__name__}: {e}") - raise - - return sessions - - async def _submit_multiple_batches(self, num_replicas, total_n, num_batches): - """Submit Multiple Batches. - - Submit multiple batches of jobs. - - Parameters - ---------- - num_replicas: int - total number of replicas (sessions) to create - total_n: int - total number of jobs (tiles) to process - num_batches: int - number of batches to split replicas into - - Each batch creates a subset of replicas. chunk() automatically - distributes all tiles across all replicas. - - Returns - ------- - list - IDs of submitted sessions - - """ - all_sessions = [] - - async with AsyncSession() as session: - for batch_num in range(1, num_batches + 1): - # Calculate batch size (number of replicas in this batch) - if batch_num < num_batches: - batch_size = self._max_replicas_per_batch - else: - # Last batch gets the remainder - batch_size = num_replicas - (num_batches - 1) * self._max_replicas_per_batch - - print(f"\n--- Batch {batch_num}/{num_batches} ---") - print(f"Submitting {batch_size} replicas") - print(f"Total {num_replicas} replicas will process {total_n} jobs") - print(f"chunk() will distribute jobs across all replicas") - - job_name = self.set_job_name(suf=f"b{batch_num}") - options = ( - f"{self._options_base} --batch_num {batch_num}" - + f" --batch_tot {num_batches}" - + f" --batch_size {batch_size}" - ) - options = options.lstrip() - print(f"Running '{self._cmd} {options}'") - - try: - sessions = await session.create( - name=job_name, - image=self._image, - cmd=self._cmd, - args=options, - replicas=batch_size, - cores=self._params["cores"], - ram=self._params["ram"], - - ) - print(f"✓ Batch {batch_num} submitted: {len(sessions)} sessions created") - print(sessions) - all_sessions.extend(sessions) - - # Small delay between batches to avoid overwhelming the system - if batch_num < num_batches: - await asyncio.sleep(1) - - except Exception as e: - print(f"❌ Batch {batch_num} failed: {type(e).__name__}: {e}") - # Continue with other batches even if one fails - continue - - print(f"\n=== Submission complete ===") - print(f"Total sessions created: {len(all_sessions)} replicas (for {total_n} jobs)") - print(f"Each replica will process ~{math.ceil(total_n / num_replicas)} jobs") - - return all_sessions - - def run_no_async(self): - """Run No Async. - - Run synchronous job submission. - - Returns - ------- - list - IDs of submitted jobs - - """ - session = Session() - - self.set_command(mode="sync") - - tile_IDs, _, _ = self.get_tile_IDs() - - job_ids = [] - for tile_ID in tile_IDs: - - job_name = self.set_job_name(tile_ID) - - options = f"{self._options_base} -e {tile_ID}" - - # Remove leading whitespace, problem in passing args below - options = options.lstrip() - - print(f"Running '{self._cmd} {options}'") - - # Submit flexible job (default - auto-scaling) - job_id = session.create( - name=job_name, - image=self._image, - cmd=self._cmd, - args=options, - ) - - print(f"Submitted job: {job_id}") - job_ids.append(job_id) - - return job_ids - - def write_job_tile_IDs(self, job_ids): - """Write Job Tile IDs - - Write tile IDs of jobs to log file. - - Parameters - ---------- - jobs_ids: list - IDs of submitted jobs. - - """ - tile_IDs, _, _ = self.get_tile_IDs() - - cwd = os.getcwd() - log_path = f"{cwd}/{self._params['log']}" - - print(f"Writing job and tile ID log file {log_path}") - with open(log_path, "w") as log: - for job_id, tile_ID in zip(job_ids, tile_IDs): - print(job_id, tile_ID, file=log) - - def print_stats(self): - """Print Stats. - - Print statistics of (to be submitted) jobs, helpfule to fine-tune parameters - before the actual submission. - - """ - # Total number of jobs (= length of input ID file) - _, total_n, _ = self.get_tile_IDs() - - if self._params["jobs_per_session"] == 0: - # Compute max jobs per session to fit in single batch - self._params["jobs_per_session"] = math.ceil(total_n / self._max_replicas_per_batch) - print(f"Max jobs per session (-J) = {self._params['jobs_per_session']}") - - # Number of sessions - n_session = math.ceil(total_n / self._params["jobs_per_session"]) - - # Number of batches - n_batch = math.ceil(n_session / self._max_replicas_per_batch) - - # Number of jobs per session - n_jobs = math.ceil(self._params["jobs_per_session"] / n_batch) - - n_jobs_serial = max( - math.ceil(n_jobs / self._params["parallel_jobs"]), - 1, - ) - - print( - f"Number of jobs = {total_n}\t\t(#{self._params['file_IDs']})") - print( - f"Number of sessions = {n_session:5d}\t\t({total_n} /" - + f" {self._params['jobs_per_session']})" - ) - print( - f"Number of batches = {n_batch:5d}\t\t({n_session} /" - + f" {self._max_replicas_per_batch})" - ) - print( - f"Number of jobs per session = {n_jobs:5d}" - + f"\t\t({self._params['jobs_per_session']} / {n_batch}" - ) - print( - f"Number of serial jobs = {n_jobs_serial:5d}\t\t({n_jobs}" - + f" / {self._params['parallel_jobs']})" - ) - - def run(self, args=None): - """Run. - - Run instance. - - Parameters - ---------- - args: list, optional - command line arguments, default is ``None`` - - """ - obj = self - - if args is None: - args = sys.argv - obj.set_params_from_command_line(args) - obj.update_params() - obj.check_params() - - obj.set_options_base() - - if self._params["stats"]: - obj.print_stats() - sys.exit(0) - - # Initialize session manager - self._session = Session() - - if obj._params["sync"] == "async": - print("Async mode") - job_ids = asyncio.run(obj.run_async()) - else: - print("Sync mode") - job_ids = obj.run_no_async() - print(f"Submitting jobs: done") - - obj.write_job_tile_IDs(job_ids) diff --git a/src/shapepipe/canfar_run.py b/src/shapepipe/canfar_run.py index 8330b8013..8f54c08ef 100644 --- a/src/shapepipe/canfar_run.py +++ b/src/shapepipe/canfar_run.py @@ -4,25 +4,7 @@ # executables are created by pyproject.toml. import sys -from shapepipe.canfar import canfar_submit, canfar_monitor, canfar_log_monitor - -def run_job(args=None): - """Run Job. - - Handles job submission with the canfar library - - Parameters - ---------- - args : list, optional - command line arguments, default is ``None`` - - """ - # Create instance - obj = canfar_submit.Job() - - # Run instance - obj.run(args=args) - +from shapepipe.canfar import canfar_monitor, canfar_log_monitor def run_log(args=None): """Run Log. diff --git a/src/shapepipe/coverage_run.py b/src/shapepipe/coverage_run.py index 15aaa1a37..9a167d8d6 100644 --- a/src/shapepipe/coverage_run.py +++ b/src/shapepipe/coverage_run.py @@ -1,79 +1,20 @@ """COVERAGE_RUN -Call coverage processing classes. +Console entry point for plotting a coverage map. + +Building the map is the Snakemake ``coverage_map`` rule's job; plotting a +finished ``.hsp`` is a human act on a durable product, so it stays a +hand-run command, on the same argument that keeps ``run_report.py`` out of the +DAG. The plot windows for the UNIONS SGC and NGC fields are in +``workflow/config.yaml``'s ``coverage:`` block. Author: Martin Kilbinger """ -import sys - -from shapepipe.utilities.header_downloader import HeaderDownloader -from shapepipe.utilities.field_corners_extractor import FieldCornersExtractor -from shapepipe.utilities.coverage_map_builder import CoverageMapBuilder from shapepipe.utilities.coverage_plotter import CoveragePlotter -def run_download_headers(args=None): - """Run Download Headers. - - Download FITS headers from VOSpace for exposures in a CCD list. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - obj = HeaderDownloader() - return obj.run(args=args) - - -def run_extract_corners(args=None): - """Run Extract Corners. - - Extract per-CCD sky-footprint corner coordinates from FITS headers. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - obj = FieldCornersExtractor() - return obj.run(args=args) - - -def run_build_coverage(args=None): - """Run Build Coverage. - - Build HealSparse coverage maps from per-CCD corner coordinates. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - obj = CoverageMapBuilder() - return obj.run(args=args) - - def run_plot_coverage(args=None): """Run Plot Coverage. @@ -92,13 +33,3 @@ def run_plot_coverage(args=None): """ obj = CoveragePlotter() return obj.run(args=args) - - -def main(argv=None): - """Main. - - Main program. - - """ - # Scripts to call coverage classes are created by pyproject.toml - return 0 diff --git a/src/shapepipe/get_ccds_run.py b/src/shapepipe/get_ccds_run.py deleted file mode 100644 index fb18b0a74..000000000 --- a/src/shapepipe/get_ccds_run.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python3 - -"""GET_CCDS_WITH_PSF - -Obtain list of CCDs (single-exposure single-HDU files) for which valid PSF information -is available. This can serve to create a footprint coverage mask. - -Author: Martin Kilbinger - -""" - -import sys - -from shapepipe.utilities.ccd_psf_handler import CcdPsfHandler - - -def run_ccd_psf_handler(args=None): - """Run CCD PSF Handler. - - Create instance and run the CCD PSF handler. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - # Create instance - - obj = CcdPsfHandler() - - return obj.run(args=args) - - -def main(argv=None): - """Main. - """ - # A scripts to call the ccd psf class is created by pyproject.toml - return 0 diff --git a/src/shapepipe/modules/merge_headers_package/__init__.py b/src/shapepipe/modules/merge_headers_package/__init__.py index ed8416790..cd47d6a3d 100644 --- a/src/shapepipe/modules/merge_headers_package/__init__.py +++ b/src/shapepipe/modules/merge_headers_package/__init__.py @@ -4,9 +4,10 @@ :Author: Axel Guinot -:Parent module: ``split_exp_runner`` +:Parent module: ``find_exposures_runner`` -:Input: Numpy binary files (``.npy``) with single-exposure header information +:Input: An ``exp_numbers`` text file listing the tile's exposures; the + per-exposure header files (``.npy``) are found under ``EXP_BASE_DIR`` :Output: Single SQL file with combined header information @@ -20,8 +21,10 @@ Module-specific config file entries =================================== -OUTPUT_PATH : str, optional - Overrides the default module output directory under ``[FILE]:OUTPUT_DIR`` +EXP_BASE_DIR : str + Root of the per-exposure work directories; the header ``.npy`` files are + collected from ``///`` for every exposure + listed in the input ``exp_numbers`` file """ diff --git a/src/shapepipe/modules/merge_headers_runner.py b/src/shapepipe/modules/merge_headers_runner.py index d7b0bd035..21a2ec08b 100644 --- a/src/shapepipe/modules/merge_headers_runner.py +++ b/src/shapepipe/modules/merge_headers_runner.py @@ -34,38 +34,32 @@ def merge_headers_runner( output_dir = run_dirs["output"] w_log.info(f"output_dir = {output_dir}") - if config.has_option(module_config_sec, "EXP_BASE_DIR"): - # Tile-level mode: input is an exp_numbers txt file; collect header - # files from each per-exposure work directory via get_exp_output_files. - exp_base_dir = config.getexpanded(module_config_sec, "EXP_BASE_DIR") + # The input is an exp_numbers txt file (find_exposures_runner); the header + # files themselves are collected from each per-exposure work directory + # under EXP_BASE_DIR via get_exp_output_files. + exp_base_dir = config.getexpanded(module_config_sec, "EXP_BASE_DIR") - # In serial mode several tiles' exp_numbers files can arrive in a - # single call; merge each tile into its own per-tile sqlite file. - for exp_numbers_file in (item[0] for item in input_file_list): - w_log.info( - f"Tile-level merge: collecting headers from {exp_base_dir} " - f"using {exp_numbers_file}" - ) - headers_file_list = get_exp_output_files( - exp_base_dir, - exp_numbers_file, - "split_exp_runner", - "headers", - ".npy", - w_log=w_log, - ) - # Extract tile number from the exp_numbers filename, e.g. - # "exp_numbers-284.272-1.000.txt" -> "-284.272-1.000" - base = os.path.splitext(os.path.basename(exp_numbers_file))[0] - tile_number = re.sub(r"^exp_numbers", "", base) - merge_headers(headers_file_list, output_dir, tile_number) - w_log.info( - f"Merged {len(headers_file_list)} exposure header files" - ) - else: - # Per-exposure mode: input_file_list already contains the header files. - merge_headers(input_file_list, output_dir) - w_log.info(f"Merged {len(input_file_list)} input file headers") + # In serial mode several tiles' exp_numbers files can arrive in a + # single call; merge each tile into its own per-tile sqlite file. + for exp_numbers_file in (item[0] for item in input_file_list): + w_log.info( + f"Collecting headers from {exp_base_dir} " + f"using {exp_numbers_file}" + ) + headers_file_list = get_exp_output_files( + exp_base_dir, + exp_numbers_file, + "split_exp_runner", + "headers", + ".npy", + w_log=w_log, + ) + # Extract tile number from the exp_numbers filename, e.g. + # "exp_numbers-284.272-1.000.txt" -> "-284.272-1.000" + base = os.path.splitext(os.path.basename(exp_numbers_file))[0] + tile_number = re.sub(r"^exp_numbers", "", base) + merge_headers(headers_file_list, output_dir, tile_number) + w_log.info(f"Merged {len(headers_file_list)} exposure header files") # No return objects return None, None diff --git a/src/shapepipe/modules/psfex_interp_package/__init__.py b/src/shapepipe/modules/psfex_interp_package/__init__.py index 1c3b51ca4..2b6a161b1 100644 --- a/src/shapepipe/modules/psfex_interp_package/__init__.py +++ b/src/shapepipe/modules/psfex_interp_package/__init__.py @@ -32,14 +32,18 @@ Threshold of stars under which the PSF is not interpolated CHI2_THRESH : int Threshold for chi squared (:math:`\chi^2`) -ME_DOT_PSF_DIR : str - Module name of last run producing PSFEx PSF model files, for multi-epoch - processing. The specifier "last:" is not required +ME_DOT_PSF_EXP_DIR : str + Root of the per-exposure work directories, for multi-epoch processing; + the ``psfex_runner`` output directory of every exposure listed in the + input ``exp_numbers`` file is resolved beneath it ME_DOT_PSF_PATTERN : str Input file name pattern for PSFEx PSF model files, for multi-epoch processing -ME_LOG_WCS : str - Path to world coordinate system log file (``*sqlite``) + +In ``MULTI-EPOCH`` mode the input file list is positional and carries three +entries -- the galaxy catalogue, the world-coordinate-system log +(``*sqlite``) and the ``exp_numbers`` file -- set through ``FILE_PATTERN`` +and ``FILE_EXT``, not through module-specific keys. """ diff --git a/src/shapepipe/modules/psfex_interp_runner.py b/src/shapepipe/modules/psfex_interp_runner.py index 67c21adfc..5627d5d20 100644 --- a/src/shapepipe/modules/psfex_interp_runner.py +++ b/src/shapepipe/modules/psfex_interp_runner.py @@ -10,7 +10,6 @@ from shapepipe.modules.psfex_interp_package import psfex_interp from shapepipe.pipeline.exp_utils import get_exp_output_dirs -from shapepipe.pipeline.run_log import get_last_dir, get_all_dirs @module_runner( @@ -62,38 +61,22 @@ def psfex_interp_runner( # Run in MULTI-EPOCH mode elif mode == "MULTI-EPOCH": - # Fetch multi-epoch parameters - if config.has_option(module_config_sec, "ME_DOT_PSF_EXP_DIR"): - # v2.0: locate psfex_runner output dirs via the $SP_EXP tree - exp_base_dir = config.getexpanded( - module_config_sec, "ME_DOT_PSF_EXP_DIR" - ) - if len(input_file_list) < 3: - raise ValueError( - "ME_DOT_PSF_EXP_DIR requires the exposure-numbers file" - + " as a third input; add 'exp_numbers' to FILE_PATTERN" - + f" and FILE_EXT in the [{module_config_sec}] config" - + " section." - ) - exp_numbers_file = input_file_list[2] - dot_psf_dirs = get_exp_output_dirs( - exp_base_dir, exp_numbers_file, "psfex_runner", w_log - ) - else: - module = config.getexpanded( - module_config_sec, - "ME_DOT_PSF_DIR", + # Fetch multi-epoch parameters. The psfex_runner output dirs are + # located through the per-exposure work tree. + exp_base_dir = config.getexpanded( + module_config_sec, "ME_DOT_PSF_EXP_DIR" + ) + if len(input_file_list) < 3: + raise ValueError( + "ME_DOT_PSF_EXP_DIR requires the exposure-numbers file" + + " as a third input; add 'exp_numbers' to FILE_PATTERN" + + f" and FILE_EXT in the [{module_config_sec}] config" + + " section." ) - module_name = module.split(":")[-1] - if "last" in module: - dot_psf_dirs = [get_last_dir(run_dirs["run_log"], module_name)] - elif "all" in module: - dot_psf_dirs = get_all_dirs(run_dirs["run_log"], module_name) - else: - raise ValueError( - "Expected qualifier 'last:' or 'all' before module" - + f" '{module}' in config entry 'ME_DOT_PSF_DIR'" - ) + exp_numbers_file = input_file_list[2] + dot_psf_dirs = get_exp_output_dirs( + exp_base_dir, exp_numbers_file, "psfex_runner", w_log + ) dot_psf_pattern = config.get( module_config_sec, diff --git a/src/shapepipe/modules/vignetmaker_package/__init__.py b/src/shapepipe/modules/vignetmaker_package/__init__.py index b24f8397e..27c679c23 100644 --- a/src/shapepipe/modules/vignetmaker_package/__init__.py +++ b/src/shapepipe/modules/vignetmaker_package/__init__.py @@ -35,15 +35,22 @@ Run mode for module, options are ``CLASSIC`` or ``MULTI-EPOCH`` PREFIX : str or list Output file name prefix(es) -ME_IMAGE_DIR : list - Module names of last run producing single-exposure flags, images, weights, - and SExtractor background images, for multi-epoch processing. The specifier - "last:" is not required +ME_IMAGE_EXP_DIR : str + Root of the per-exposure work directories, for multi-epoch processing; + each runner output directory is resolved beneath it for every exposure + listed in the input ``exp_numbers`` file +ME_IMAGE_EXP_RUNNERS : list + Names of the runners producing the single-exposure flags, images, weights + and SExtractor background images, in the same order as + ``ME_IMAGE_PATTERN`` ME_IMAGE_PATTERN : list Input file name patterns for flag, image, weight, and SExtractor background files, for multi-epoch processing -ME_LOG_WCS : str - Path to world coordinate system log file (``*sqlite``) + +In ``MULTI-EPOCH`` mode the input file list is positional and carries three +entries -- the galaxy catalogue, the world-coordinate-system log +(``*sqlite``) and the ``exp_numbers`` file -- set through ``FILE_PATTERN`` +and ``FILE_EXT``, not through module-specific keys. """ diff --git a/src/shapepipe/modules/vignetmaker_runner.py b/src/shapepipe/modules/vignetmaker_runner.py index 2d9661faa..bad5a22df 100644 --- a/src/shapepipe/modules/vignetmaker_runner.py +++ b/src/shapepipe/modules/vignetmaker_runner.py @@ -10,7 +10,6 @@ from shapepipe.modules.vignetmaker_package import vignetmaker as vm from shapepipe.pipeline.exp_utils import get_exp_output_dirs -from shapepipe.pipeline.run_log import get_last_dir, get_all_dirs @module_runner( @@ -105,44 +104,27 @@ def vignetmaker_runner( elif mode == "MULTI-EPOCH": # Multi-epoch exposures - if config.has_option(module_config_sec, "ME_IMAGE_EXP_DIR"): - # v2.0: locate runner output dirs via the $SP_EXP tree - exp_base_dir = config.getexpanded( - module_config_sec, "ME_IMAGE_EXP_DIR" + # Locate the runner output dirs through the per-exposure work tree. + exp_base_dir = config.getexpanded( + module_config_sec, "ME_IMAGE_EXP_DIR" + ) + if len(input_file_list) < 3: + raise ValueError( + "ME_IMAGE_EXP_DIR requires the exposure-numbers" + + " file as a third input; add 'exp_numbers' to" + + " FILE_PATTERN and FILE_EXT in the" + + f" [{module_config_sec}] config section." ) - if len(input_file_list) < 3: - raise ValueError( - "ME_IMAGE_EXP_DIR requires the exposure-numbers" - + " file as a third input; add 'exp_numbers' to" - + " FILE_PATTERN and FILE_EXT in the" - + f" [{module_config_sec}] config section." - ) - exp_numbers_file = input_file_list[2] - exp_runner_names = config.getlist( - module_config_sec, "ME_IMAGE_EXP_RUNNERS" + exp_numbers_file = input_file_list[2] + exp_runner_names = config.getlist( + module_config_sec, "ME_IMAGE_EXP_RUNNERS" + ) + image_dirs = [] + for runner_name in exp_runner_names: + dirs = get_exp_output_dirs( + exp_base_dir, exp_numbers_file, runner_name, w_log ) - image_dirs = [] - for runner_name in exp_runner_names: - dirs = get_exp_output_dirs( - exp_base_dir, exp_numbers_file, runner_name, w_log - ) - image_dirs.append(dirs) - else: - # v1: run-log based lookup via symlinked exposure run dirs - modules = config.getlist(module_config_sec, "ME_IMAGE_DIR") - image_dirs = [] - for module in modules: - module_name = module.split(":")[-1] - if "last" in module: - dirs = [get_last_dir(run_dirs["run_log"], module_name)] - elif "all" in module: - dirs = get_all_dirs(run_dirs["run_log"], module_name) - else: - raise ValueError( - "Expected qualifier 'last:' or 'all' before module" - + f" '{module}' in config entry 'ME_IMAGE_DIR'" - ) - image_dirs.append(dirs) + image_dirs.append(dirs) image_pattern = config.getlist( module_config_sec, diff --git a/src/shapepipe/pipeline/exp_utils.py b/src/shapepipe/pipeline/exp_utils.py index b934f0bb3..889ff31c1 100644 --- a/src/shapepipe/pipeline/exp_utils.py +++ b/src/shapepipe/pipeline/exp_utils.py @@ -1,11 +1,10 @@ """EXPOSURE UTILITIES. Utility functions for accessing per-exposure runner outputs from the -tile level. In the v2.0 pipeline, each single exposure is processed in -its own work directory ``///``. Tile-level -modules that need files produced by a per-exposure runner use the -functions here to discover those files by scanning the contributing -exposure directories. +tile level. Each single exposure is processed in its own work directory +``///``. Tile-level modules that need files +produced by a per-exposure runner use the functions here to discover those +files by scanning the contributing exposure directories. :Author: Martin Kilbinger @@ -40,7 +39,7 @@ def get_exp_output_files( ---------- exp_base_dir : str Root directory that contains all per-exposure work directories, - e.g. ``/arc/home/kilbinger/v2.0/exp`` + e.g. ``/exp`` exp_numbers_file : str Path to a text file listing one exposure ID per line, e.g. ``exp_numbers-301-279.txt`` @@ -91,7 +90,8 @@ def get_exp_output_files( missing = [] for exp_id in exp_ids: - # Directory structure mirrors run_job_canfar_v2.0.sh: + # Directory structure mirrors the workflow's per-exposure store, built + # by rule tile_exp_forest / workflow/scripts/build_forest.py: # exp_prefix = first 2 chars of exp_id (e.g. "21") # exp_base = exp_id without trailing letter if present (e.g. "2113864"), # or full exp_id for numeric-only ids (image sims) diff --git a/src/shapepipe/summary_run.py b/src/shapepipe/summary_run.py deleted file mode 100755 index 39d60ce08..000000000 --- a/src/shapepipe/summary_run.py +++ /dev/null @@ -1,102 +0,0 @@ -#!/usr/bin/env python - -import argparse -import sys -import os - -from shapepipe.utilities import summary - -from shapepipe.utilities import summary_params_pre_v2 as summary_params - - -def run(patch, job_exclusive=None, verbose=False): - - jobs, list_tile_IDs_dot = summary_params.set_jobs_v2_pre_v2(patch, verbose) - - list_tile_IDs = summary.job_data.replace_dot_dash(list_tile_IDs_dot) - - # Numbers updated at runtime - par_runtime = summary.init_par_runtime(list_tile_IDs) - - summary.job_data.print_stats_header() - - exp_IDs_path = "exp_numbers.txt" - if os.path.exists(exp_IDs_path): - # Read exposure ID list if file exists - all_exposures = summary.get_IDs_from_file(exp_IDs_path) - par_runtime = summary.update_par_runtime_after_find_exp( - par_runtime, all_exposures - ) - - if ( - not os.path.exists(exp_IDs_path) - or not job_exclusive - or int(job_exclusive) & 1 - ): - # Run job 1 if exposure ID list file does not exist or - # job_exclusive is 1 or not set - key = "1" - jobs[key].print_intro() - jobs[key].check_numbers(par_runtime=par_runtime, indices=[0, 1]) - - all_exposures = summary.get_all_exposures( - jobs[key]._paths_in_dir[1], verbose=True - ) - par_runtime = summary.update_par_runtime_after_find_exp( - par_runtime, all_exposures - ) - - jobs[key].write_IDs_to_file("exp_numbers.txt", all_exposures) - - jobs[key].check_numbers(par_runtime, indices=[2]) - - summary.print_par_runtime(par_runtime, verbose=verbose) - - # Get all keys after "1" - keys = sorted(jobs.keys(), key=int) - _ = keys.pop(0) - - for key in keys: - if job_exclusive and not int(key) & int(job_exclusive): - continue - jobs[key].print_intro() - jobs[key].check_numbers(par_runtime=par_runtime) - - return 0 - - -def parse_args(argv=None): - parser = argparse.ArgumentParser( - description=( - 'Print summary of ShapePipe job status for a given UNIONS patch.' - ), - ) - parser.add_argument( - 'patch', - help='Patch identifier (e.g. P3, P9).', - ) - parser.add_argument( - 'job_exclusive', - nargs='?', - default=None, - help=( - 'Bitmask selecting which jobs to run; ' - + 'if omitted, all jobs run.' - ), - ) - parser.add_argument( - '-v', - '--verbose', - action='store_true', - help='Verbose output.', - ) - return parser.parse_args(argv) - - -def main(argv=None): - args = parse_args(argv) - run(args.patch, args.job_exclusive, args.verbose) - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/src/shapepipe/utilities/__init__.py b/src/shapepipe/utilities/__init__.py index 42e234003..8173e49b5 100644 --- a/src/shapepipe/utilities/__init__.py +++ b/src/shapepipe/utilities/__init__.py @@ -7,4 +7,4 @@ """ -__all__ = ["file_system", "cfis", "galaxy", "mask_query", "summary"] +__all__ = ["file_system", "cfis", "galaxy", "mask_query"] diff --git a/src/shapepipe/utilities/ccd_footprint.py b/src/shapepipe/utilities/ccd_footprint.py new file mode 100644 index 000000000..41cff7612 --- /dev/null +++ b/src/shapepipe/utilities/ccd_footprint.py @@ -0,0 +1,92 @@ +"""CCD_FOOTPRINT + +Project a single CCD onto the sky: image shape from a header, sky corners +from a WCS. + +The two functions here are the geometry the coverage mask rests on, and they +are all that survives of the pre-Snakemake header scrape. The ``exp_footprint`` +rule (``workflow/scripts/exp_footprint.py``) calls them once per CCD of an +exposure, reading the headers out of ``headers-.npy`` rather than out of +text files downloaded from VOSpace; the ``coverage_map`` rule then stamps the +corners into the campaign's HealSparse map. + +Author: Mike Hudson, Martin Kilbinger + +""" + + +def _image_shape(header): + """Return the CCD image ``(nx, ny)`` pixel shape from a header. + + For fpack tile-compressed HDUs (``ZIMAGE = T``), ``NAXIS1``/``NAXIS2`` + describe the compressed *binary table* (row width in bytes, row count), + not the image, and astropy's WCS never maps the true dimensions into + ``pixel_shape``. The true image size is carried by ``ZNAXIS1``/``ZNAXIS2``, + which are preferred here; a plain image HDU falls back to + ``NAXIS1``/``NAXIS2``. + + Both branches are live. ``headers-.npy`` carries the *decompressed* + header astropy hands back for a tile-compressed HDU, so it takes the plain + ``NAXIS`` path; a header read straight off a fpacked file takes the ``Z*`` + one. + + Parameters + ---------- + header : astropy.io.fits.Header + single-HDU header + + Returns + ------- + tuple + ``(nx, ny)`` image pixel dimensions + + Raises + ------ + ValueError + if neither the ``Z*`` nor the plain ``NAXIS`` dimensions are present + """ + if header.get("ZIMAGE", False): + keys = ("ZNAXIS1", "ZNAXIS2") + else: + keys = ("NAXIS1", "NAXIS2") + + if keys[0] not in header or keys[1] not in header: + raise ValueError( + f"Header is missing image dimensions ({keys[0]}/{keys[1]})" + ) + + return int(header[keys[0]]), int(header[keys[1]]) + + +def _ccd_corners(w, shape): + """Return (ra, dec) of the 4 corners of a single CCD. + + The polygon is built from the CCD's pixel bounds ``shape = (nx, ny)`` using + pixel edges ``(-0.5, n - 0.5)`` so the quadrilateral covers the full CCD + area rather than pixel centres. Corners are returned in a consistent + counterclockwise pixel order (bottom-left, bottom-right, top-right, + top-left); since ``pixel_to_world`` maps this order to a simple, + non-self-intersecting sky quadrilateral for a CCD-sized field, and + HealSparse ``Polygon`` is orientation-agnostic, the resulting polygon is a + valid convex footprint. + + Parameters + ---------- + w : astropy.wcs.WCS + WCS of a single CCD + shape : tuple + ``(nx, ny)`` CCD image pixel dimensions + + Returns + ------- + tuple + ``(ra_list, dec_list)`` each of length 4, in degrees + """ + nx, ny = shape + + # Pixel-edge corners, counterclockwise: BL, BR, TR, TL + px = [-0.5, nx - 0.5, nx - 0.5, -0.5] + py = [-0.5, -0.5, ny - 0.5, ny - 0.5] + + sky = w.pixel_to_world(px, py) + return list(sky.ra.deg), list(sky.dec.deg) diff --git a/src/shapepipe/utilities/ccd_psf_handler.py b/src/shapepipe/utilities/ccd_psf_handler.py deleted file mode 100644 index 2d0dc90eb..000000000 --- a/src/shapepipe/utilities/ccd_psf_handler.py +++ /dev/null @@ -1,387 +0,0 @@ -"""CCD_PSF_HANDLER - -Obtain list of CCDs (single-exposure single-HDU files) for which valid PSF -information is available. This can serve to create a footprint coverage mask. - -Author: Mike Hudson, Martin Kilbinger - -""" - -import glob -import os -import re -import sys - -import numpy as np - -from cs_util import args as cs_args -from cs_util import logging - -from shapepipe.utilities import summary - - -class CcdPsfHandler(object): - """CCD PSF Handler Class. - - Handles extraction of CCDs with valid PSF information from shapepipe - patches. - """ - - def __init__(self): - """Initialize the handler.""" - self.params_default() - - def params_default(self): - """Set default parameters and command line options.""" - - self._params = { - "version_cat": "v1.6", - "n_CCD": 40, - "output": None, - } - - self._short_options = { - "version_cat": "-V", - "n_CCD": "-n", - "output": "-o", - } - - self._types = { - "n_CCD": "int", - } - - self._help_strings = { - "version_cat": "catalogue major version, allowed are v1.3, v1.4, v1.5, v1.6; default is {}", - - "n_CCD": "number of CCDs per exposure; default is {}", - "output": "output file path; default is ccds_with_psf_.txt", - } - - def set_params_from_command_line(self, args): - """Set Params From Command line. - - Only use when calling using python from command line. - Does not work from ipython or jupyter. - - Parameters - ---------- - args : list - command line arguments - - """ - # Read command line options - options = cs_args.parse_options( - self._params, - self._short_options, - self._types, - self._help_strings, - args=args, - ) - self._params = options - - # Save calling command - logging.log_command(args) - - def update_params(self): - """Update parameters. - - Set derived parameters based on input parameters. - """ - # Determine number of patches based on version - version = self._params["version_cat"] - if version in ("v1.3", "v1.4"): - n_patch = 7 - elif version == "v1.5": - n_patch = 8 - elif version == "v1.6": - n_patch = 9 - else: - raise ValueError(f"Invalid version {version}") - - self._params["n_patch"] = n_patch - self._params["patches"] = [f"P{x}" for x in np.arange(n_patch) + 1] - - # Set output file if not specified - if self._params["output"] is None: - self._params["output"] = f"ccds_with_psf_{version}.txt" - - def check_params(self): - """Check parameters for validity.""" - # Add any parameter validation logic here - pass - - def get_lines(self, fname): - """Get Lines. - - Return list of lines read from a text file. - - Parameters - ---------- - fname : str - input file name - - Returns - ------- - list - IDs - - """ - IDs = [] - with open(fname) as f: - lines = f.readlines() - for line in lines: - IDs.append(line.rstrip()) - - return IDs - - def get_exp_shdu_missing(self, patches): - """Get Exp Shdu Missing. - - Returns set of missing CCDs (single-exposure single-HDU IDs) from a list of patches. - - Parameters - ---------- - patches : list - input patches - - Returns - ------- - set - missing CCD IDs - - """ - exp_shdu_missing_all = set() - - for patch in patches: - - path_exp_shdu_missing = f"{patch}/summary/missing_job_32_all.txt" - exp_shdu_missing = self.get_lines(path_exp_shdu_missing) - - print( - f"Patch {patch}: Found {len(exp_shdu_missing)} missing ccds", - end="; ", - ) - - exp_shdu_missing_all.update(exp_shdu_missing) - - print(f"cumulative {len(exp_shdu_missing_all)} missing ccds") - - print() - - return exp_shdu_missing_all - - def get_exp(self, patches): - """Get Exp. - - Return set of exposures from a list of patches. - - Parameters - ---------- - patches : list - input patches - - Returns - ------- - set - exposure IDs - - """ - exp_all = set() - - for patch in patches: - - path_exp = f"{patch}/exp_numbers.txt" - exp = self.get_lines(path_exp) - - print(f"Patch {patch}: Found {len(exp)} exposures", end="; ") - - exp_all.update(exp) - - print(f"cumulative {len(exp_all)} exposures") - - print() - - return exp_all - - def get_ccds_with_psf(self, patches, n_CCD=40): - """Get CCDs With PSF. - - Return set of CCDs with valid PSF from a list of patches. A CCD has a - valid PSF if it appears among the exposures' single-HDU IDs but not in - the per-patch missing-CCD lists. - - Parameters - ---------- - patches : list - input patches - n_CCD : int - number of CCDs per exposure - - Returns - ------- - set - CCD IDs with valid PSF - - """ - # Get missing CCDs - print("=== get missing CCDs ===") - exp_shdu_missing_all = self.get_exp_shdu_missing(patches) - - # Get all exposures used in tiles - print("=== get exposures ===") - exp_all = self.get_exp(patches) - - # Turn exposures into exposure-single-HDU names (CCDs) - exp_shdu_all = set(summary.get_all_shdus(exp_all, n_CCD)) - - # Subtract the CCDs whose PSF model is missing - exp_shdu_with_psf = exp_shdu_all - exp_shdu_missing_all - - print( - f"Found {len(exp_shdu_all)} CCDs, " - f"{len(exp_shdu_missing_all)} missing, " - f"{len(exp_shdu_with_psf)} with valid PSF" - ) - - return exp_shdu_with_psf - - def get_ccds_with_psf_method_v1_3(self, patches, n_CCD=40): - """Get CCDs With PSF Method v1.3. - - Return set of CCDs with valid PSF by scanning run directories. - Finds star_selection files and checks corresponding stat files - for valid (non-nan) FWHM values. - - Parameters - ---------- - patches : list - input patches - n_CCD : int - number of CCDs per exposure (unused, kept for API consistency) - - Returns - ------- - set - CCD IDs with valid PSF - - """ - exp_shdu_all = set() - - for patch in patches: - # Find all star_selection files - mask_pattern = ( - f"{patch}/output/run_sp_exp_SxSePs*/" - f"setools_runner/output/mask/star_selection-*.fits" - ) - mask_files = glob.glob(mask_pattern) - - print( - f"Patch {patch}: Found {len(mask_files)} star_selection FITS" - + " files" - ) - - for mask_file in mask_files: - # Extract exp_shdu from filename - basename = os.path.basename(mask_file) - match = re.match(r"star_selection-(.+)\.fits", basename) - if not match: - print(f"Warning: Non-matching file name {basename}") - continue - exp_shdu = match.group(1) - - # Get corresponding stat file path - # Replace mask dir with stat/stat dir in the path - stat_file = mask_file.replace( - "/mask/star_selection-", - "/stat/star_stat-", - ).replace(".fits", ".txt") - - #print("MKDEBUG ", exp_shdu, stat_file) - - # Check if stat file exists and has valid FWHM - if not os.path.exists(stat_file): - print("MKDEBUG stat file ", stat_file, " does not exist") - continue - - # Read stat file and check for nan in "Mean star fwhm selected" - has_nan = False - with open(stat_file) as f: - for line in f: - if "Mean star fwhm selected" in line: - if "nan" in line.lower(): - has_nan = True - break - - if not has_nan: - #print("MKDEBUG append", exp_shdu) - exp_shdu_all.add(exp_shdu) - - print(f"After patch {patch}: {len(exp_shdu_all)} CCDs with valid PSF") - - print(f"Found {len(exp_shdu_all)} CCDs with valid PSF") - - return exp_shdu_all - - def save(self, IDs, path): - """Save. - - Save list of IDs to text file. - - Parameters - ---------- - IDs : set - input IDs - path : str - output file name - - """ - with open(path, "w") as f_out: - for ID in IDs: - print(ID, file=f_out) - - def run(self, args=None): - """Run. - - Main execution method. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code (0 for success) - - """ - if args is None: - args = sys.argv[1:] - - # Set parameters from command line - self.set_params_from_command_line(args) - self.update_params() - self.check_params() - - # Get parameters - patches = self._params["patches"] - version = self._params["version_cat"] - n_CCD = self._params["n_CCD"] - output = self._params["output"] - - print( - f"=== get_ccds_with_psf for version {version}, patches {patches} ===" - ) - - if self._params["version_cat"] == "v1.3": - print("=== method for 1.3: exp with PSF star sample === ") - exp_shdu_all = self.get_ccds_with_psf_method_v1_3(patches, n_CCD) - else: - print("=== method for >= 1.4: exp_list - missing === ") - exp_shdu_all = self.get_ccds_with_psf(patches, n_CCD) - - self.save(exp_shdu_all, output) - - print(f"Results saved to {output}") - - return 0 diff --git a/src/shapepipe/utilities/coverage_map_builder.py b/src/shapepipe/utilities/coverage_map_builder.py index c7144ef5c..659a0d71f 100644 --- a/src/shapepipe/utilities/coverage_map_builder.py +++ b/src/shapepipe/utilities/coverage_map_builder.py @@ -1,25 +1,27 @@ """COVERAGE_MAP_BUILDER -Build HealSparse coverage maps from per-CCD corner coordinates. +Stamp per-CCD sky footprints into a HealSparse coverage (nexp) map. -Each input row is one CCD footprint. Since the CCDs of a single exposure do -not overlap, stamping value 1 per CCD polygon and accumulating makes the map -count, per sky pixel, the number of exposures with a valid PSF model covering -that pixel. +Each polygon is one CCD footprint. Since the CCDs of a single exposure do not +overlap, stamping value 1 per CCD polygon and accumulating makes the map count, +per sky pixel, the number of exposures with a valid PSF model covering that +pixel. + +Arrays in, map out. The only caller is the ``coverage_map`` rule +(``workflow/scripts/coverage_map.py``), which reads the corners out of the +per-exposure ``exp_footprint.json`` records; the nine-column ``exp_ra_dec.txt`` +the pre-Snakemake chain appended to is gone, and so is the CLI that parsed it. +What is left here is survey geometry — the RA-seam guard, the pole guard, the +nside validation and the median smoothing — none of which depends on how the +corners arrived. Author: Mike Hudson, Martin Kilbinger """ -import sys -from os.path import exists - -import numpy as np import healsparse as hsp import hpgeom as hpg - -from cs_util import args as cs_args -from cs_util import logging +import numpy as np # Declination beyond which a polygon is considered too close to a pole for # HealSparse's planar polygon fill to be reliable. CCD footprints are ~10 @@ -52,162 +54,134 @@ def unwrap_ra(ra): return ra +def check_nside(nside_coverage, nside): + """Raise unless both HealSparse resolutions are positive powers of two. + + Checked before any polygon is stamped: healsparse itself complains only + once a map is being made, by which time the caller has already paid for + reading every footprint record on the products root. -class CoverageMapBuilder(object): - """Coverage Map Builder Class. + Parameters + ---------- + nside_coverage : int + HealSparse coverage nside + nside : int + HealSparse map nside + + Raises + ------ + ValueError + if either value is not a positive power of two + + """ + for name, n in (("nside_coverage", nside_coverage), ("nside", nside)): + if n <= 0 or n & (n - 1): + raise ValueError(f"{name} must be a power of 2, got {n}") + + +def build_map( + ccd_ids, ra, dec, nside_coverage, nside, verbose=False +): + """Stamp one polygon per CCD footprint into a HealSparse nexp map. + + Since the CCDs of a single exposure do not overlap, accumulating value 1 + per CCD polygon makes the map count, per sky pixel, the number of + exposures with a valid PSF model covering it. + + Parameters + ---------- + ccd_ids : array_like + length-N CCD IDs, used only in the skipped-polygon warnings + ra : array_like + ``(N, 4)`` polygon RA corners, in degrees + dec : array_like + ``(N, 4)`` polygon Dec corners, in degrees + nside_coverage : int + HealSparse coverage nside + nside : int + HealSparse map nside + verbose : bool, optional + print progress + + Returns + ------- + healsparse.HealSparseMap + the nexp map - Builds HealSparse coverage maps from field corner coordinates. """ + check_nside(nside_coverage, nside) - def __init__(self): - """Initialize the builder.""" - self.params_default() - - def params_default(self): - """Set default parameters and command line options.""" - - self._params = { - "input_file": "exp_ra_dec.txt", - "output_file": "coverage.hsp", - "nside_coverage": 32, - "nside": 2048, - "apply_median_filter": False, - "n_median_iterations": 2, - "create_boolean": False, - "boolean_threshold": 3, - "boolean_output": None, - "create_plot": False, - "plot_output": None, - "plot_region": None, - "verbose": False, - } - - self._short_options = { - "input_file": "-i", - "output_file": "-o", - "nside_coverage": "-c", - "nside": "-n", - "apply_median_filter": "-m", - "n_median_iterations": "-N", - "create_boolean": "-b", - "boolean_threshold": "-t", - "boolean_output": "-B", - "create_plot": "-p", - "plot_output": "-P", - "plot_region": "-g", - } - - self._types = { - "nside_coverage": "int", - "nside": "int", - "apply_median_filter": "bool", - "n_median_iterations": "int", - "create_boolean": "bool", - "boolean_threshold": "int", - "create_plot": "bool", - "verbose": "bool", - } - - self._help_strings = { - "input_file": "input file with field corners; default is {}", - "output_file": "output HealSparse map file; default is {}", - "nside_coverage": "HealSparse coverage nside; default is {}", - "nside": "HealSparse map nside; default is {}", - "apply_median_filter": "apply median filter to smooth map; default is {}", - "n_median_iterations": "number of median filter iterations; default is {}", - "create_boolean": "create boolean coverage map; default is {}", - "boolean_threshold": "threshold value for boolean map; default is {}", - "boolean_output": "output file for boolean map; default is _bool.hsp", - "create_plot": "create plot of coverage map; default is {}", - "plot_output": "output file for plot; default is .png", - "plot_region": "predefined region for plot (NGC, SGC, fullsky); default is {}", - } - - def set_params_from_command_line(self, args): - """Set Params From Command line. - - Only use when calling using python from command line. - Does not work from ipython or jupyter. - - Parameters - ---------- - args : list - command line arguments - - """ - # Read command line options - options = cs_args.parse_options( - self._params, - self._short_options, - self._types, - self._help_strings, - args=args, + ra = np.atleast_2d(np.asarray(ra, dtype=float)) + dec = np.atleast_2d(np.asarray(dec, dtype=float)) + + if verbose: + print( + f"Creating HealSparse map (nside_coverage={nside_coverage}," + f" nside={nside})" ) - self._params = options - - # Save calling command - logging.log_command(args) - - def update_params(self): - """Update parameters. - - Set derived parameters based on input parameters. - """ - # Set boolean output filename if not specified - if self._params["boolean_output"] is None: - output_file = self._params["output_file"] - if output_file.endswith(".hsp"): - base = output_file[:-4] - else: - base = output_file - self._params["boolean_output"] = f"{base}_bool.hsp" - - # Set plot output filename if not specified - if self._params["plot_output"] is None: - output_file = self._params["output_file"] - if output_file.endswith(".hsp"): - base = output_file[:-4] - else: - base = output_file - self._params["plot_output"] = f"{base}.png" - - def check_params(self): - """Check parameters for validity.""" - if not exists(self._params["input_file"]): - raise FileNotFoundError( - f"Input file not found: {self._params['input_file']}" - ) + m = hsp.HealSparseMap.make_empty(nside_coverage, nside, np.uint16) + + if verbose: + print("Adding polygons to map") - # Check nside values are powers of 2 - nside_coverage = self._params["nside_coverage"] - nside = self._params["nside"] + n_added = 0 + n_skipped = 0 + for i in range(len(ccd_ids)): + dec_i = dec[i] - if not (nside_coverage & (nside_coverage - 1) == 0): - raise ValueError( - f"nside_coverage must be a power of 2, got {nside_coverage}" + # Pole guard: HealSparse's planar polygon fill degrades near the + # poles. CCD footprints never reach here, so warn and skip. + if np.any(np.abs(dec_i) >= _DEC_POLE_LIMIT): + print( + f"Warning: skipping CCD {ccd_ids[i]} with |dec| >= " + f"{_DEC_POLE_LIMIT} (too close to a pole)" ) + n_skipped += 1 + continue + + # RA-wrap guard: put corners on a common branch across the seam. + ra_i = unwrap_ra(ra[i]) - if not (nside & (nside - 1) == 0): - raise ValueError(f"nside must be a power of 2, got {nside}") + m += hsp.Polygon(ra=list(ra_i), dec=list(dec_i), value=1) + n_added += 1 - def median_filter(self, hsp_map): - """Median Filter. + if verbose and i % 1000 == 0: + print(f"{i:6d} / {len(ccd_ids):6d}") - Apply median filter to HealSparse map using neighbors. + print(f"Added {n_added} polygons to map") + if n_skipped > 0: + print(f"Skipped {n_skipped} polygons near a pole") - Parameters - ---------- - hsp_map : healsparse.HealSparseMap - input map + return m + + +def median_filter(hsp_map, n_iterations=1): + """Smooth an nexp map: each pixel becomes its neighbourhood median. + + The neighbourhood is a pixel and its eight HEALPix neighbours; out-of-map + neighbours read as the map's sentinel and so pull an edge pixel down, + which is the intended behaviour for a coverage map (a lone pixel is noise, + not depth). The ``coverage_map`` rule does NOT smooth — the map it writes + is the raw exposure count, which is what sp_validation's mask application + wants — so this is the offline step, for looking at a map rather than + applying one. + + Parameters + ---------- + hsp_map : healsparse.HealSparseMap + input nexp map + n_iterations : int, optional + number of smoothing passes - Returns - ------- - healsparse.HealSparseMap - filtered map + Returns + ------- + healsparse.HealSparseMap + filtered map - """ - nside = hsp_map.nside_sparse + """ + nside = hsp_map.nside_sparse + for _ in range(n_iterations): new_hsp = hsp_map.copy() pixs = hsp_map.valid_pixels n = hpg.neighbors(nside, pixs) @@ -217,156 +191,6 @@ def median_filter(self, hsp_map): mn = hsp_map[n] new = np.median(mn, axis=1).astype(np.uint16) new_hsp[pixs] = new + hsp_map = new_hsp - return new_hsp - - def run(self, args=None): - """Run. - - Main execution method. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code (0 for success) - - """ - if args is None: - args = sys.argv[1:] - - # Set parameters from command line - self.set_params_from_command_line(args) - self.update_params() - self.check_params() - - # Get parameters - input_file = self._params["input_file"] - output_file = self._params["output_file"] - nside_coverage = self._params["nside_coverage"] - nside = self._params["nside"] - apply_median_filter = self._params["apply_median_filter"] - n_median_iterations = self._params["n_median_iterations"] - create_boolean = self._params["create_boolean"] - boolean_threshold = self._params["boolean_threshold"] - boolean_output = self._params["boolean_output"] - create_plot = self._params["create_plot"] - plot_output = self._params["plot_output"] - plot_region = self._params["plot_region"] - verbose = self._params["verbose"] - - if verbose: - print(f"Reading per-CCD corners from: {input_file}") - - # Load per-CCD corner data. Column 0 is a "-" string - # ID; the remaining 8 columns are the 4 RA then 4 Dec corners. - ccd_ids = np.atleast_1d(np.loadtxt(input_file, usecols=(0), dtype=str)) - corners = np.atleast_2d( - np.loadtxt(input_file, usecols=range(1, 9), dtype=float) - ) - ra_all = corners[:, :4] - dec_all = corners[:, 4:] - - print(f"Loaded {len(ccd_ids)} CCD footprints") - - if verbose: - print( - f"Creating HealSparse map (nside_coverage={nside_coverage}, nside={nside})" - ) - - # Create empty map - m = hsp.HealSparseMap.make_empty(nside_coverage, nside, np.uint16) - - # Add one polygon per CCD footprint - if verbose: - print("Adding polygons to map") - - n_added = 0 - n_skipped = 0 - for i in range(len(ccd_ids)): - dec = dec_all[i] - - # Pole guard: HealSparse's planar polygon fill degrades near the - # poles. CCD footprints never reach here, so warn and skip. - if np.any(np.abs(dec) >= _DEC_POLE_LIMIT): - print( - f"Warning: skipping CCD {ccd_ids[i]} with |dec| >= " - f"{_DEC_POLE_LIMIT} (too close to a pole)" - ) - n_skipped += 1 - continue - - # RA-wrap guard: put corners on a common branch across the seam. - ra = unwrap_ra(ra_all[i]) - - m += hsp.Polygon(ra=list(ra), dec=list(dec), value=1) - n_added += 1 - - if verbose and i % 1000 == 0: - print(f"{i:6d} / {len(ccd_ids):6d}") - - print(f"Added {n_added} polygons to map") - if n_skipped > 0: - print(f"Skipped {n_skipped} polygons near a pole") - - # Apply median filter if requested - if apply_median_filter: - if verbose: - print( - f"Applying median filter ({n_median_iterations} iterations)" - ) - - for i in range(n_median_iterations): - m = self.median_filter(m) - if verbose: - print(f" Iteration {i+1}/{n_median_iterations} complete") - - # Write main coverage map - if verbose: - print(f"Writing coverage map to: {output_file}") - - m.write(output_file, clobber=True) - print(f"Coverage map saved to {output_file}") - - # Create and write boolean map if requested - if create_boolean: - if verbose: - print( - f"Creating boolean map (threshold={boolean_threshold})" - ) - - c = hsp.HealSparseMap.make_empty(nside_coverage, nside, bool) - c[m.valid_pixels] = m[m.valid_pixels] >= boolean_threshold - - if verbose: - print(f"Writing boolean map to: {boolean_output}") - - c.write(boolean_output, clobber=True) - print(f"Boolean map saved to {boolean_output}") - - # Create plot if requested - if create_plot: - if verbose: - print(f"Creating plot of coverage map") - - try: - from shapepipe.utilities.coverage_plotter import CoveragePlotter - plotter = CoveragePlotter() - plotter.plot_coverage_map( - m, - plot_output, - region=plot_region, - vmax=boolean_threshold if create_boolean else 3, - colorbar=True, - colorbar_label="Coverage depth", - ) - print(f"Plot saved to {plot_output}") - except ImportError as e: - print(f"Warning: Could not create plot: {e}") - print("Install the cs_util package for plotting support.") - - return 0 + return hsp_map diff --git a/src/shapepipe/utilities/field_corners_extractor.py b/src/shapepipe/utilities/field_corners_extractor.py deleted file mode 100644 index 7e3273456..000000000 --- a/src/shapepipe/utilities/field_corners_extractor.py +++ /dev/null @@ -1,493 +0,0 @@ -"""FIELD_CORNERS_EXTRACTOR - -Extract per-CCD sky-footprint corner coordinates from FITS headers. - -For each CCD (image HDU) in an exposure's multi-HDU header, the four corners -of the CCD are projected from pixel bounds to the sky, producing one row per -CCD keyed on its ``-`` ID. - -Author: Mike Hudson, Martin Kilbinger - -""" - -import sys -import os -import glob -import re -from os.path import exists -from multiprocessing import Pool, cpu_count - -import numpy as np -from astropy import wcs -from astropy.io.fits import Header - -from cs_util import args as cs_args -from cs_util import logging - -# Filename suffix convention for header files: <6+ digit exposure number>.txt -_EXPNUM_RE = re.compile(r'(\d+)\.txt$') - - -def _expnum_from_path(path): - """Extract exposure number from a header filename like ``1234567.txt``.""" - match = _EXPNUM_RE.search(path) - if match is None: - raise ValueError(f"Could not extract exposure number from {path!r}") - return int(match.group(1)) - - -def _image_shape(header): - """Return the CCD image ``(nx, ny)`` pixel shape from a header. - - For fpack tile-compressed HDUs (``ZIMAGE = T``), ``NAXIS1``/``NAXIS2`` - describe the compressed *binary table* (row width in bytes, row count), - not the image, and astropy's WCS never maps the true dimensions into - ``pixel_shape``. The true image size is carried by ``ZNAXIS1``/``ZNAXIS2``, - which are preferred here; a plain image HDU falls back to - ``NAXIS1``/``NAXIS2``. - - Parameters - ---------- - header : astropy.io.fits.Header - single-HDU header - - Returns - ------- - tuple - ``(nx, ny)`` image pixel dimensions - - Raises - ------ - ValueError - if neither the ``Z*`` nor the plain ``NAXIS`` dimensions are present - """ - if header.get("ZIMAGE", False): - keys = ("ZNAXIS1", "ZNAXIS2") - else: - keys = ("NAXIS1", "NAXIS2") - - if keys[0] not in header or keys[1] not in header: - raise ValueError( - f"Header is missing image dimensions ({keys[0]}/{keys[1]})" - ) - - return int(header[keys[0]]), int(header[keys[1]]) - - -def _parse_header_to_wcs(path): - """Parse a multi-HDU header text file into ``(wcs, (nx, ny))`` per HDU. - - The primary HDU is skipped. Each remaining HDU yields its WCS together with - the CCD image pixel shape (``ZNAXIS1/2`` for compressed HDUs, else - ``NAXIS1/2``); the shape is read from the header because the WCS drops the - ``Z*`` keywords. - """ - with open(path, "r") as f: - string = f.read() - tokens = re.split(r"^(END\s+)", string, flags=re.MULTILINE) - result = [] - for i in range(2, len(tokens) - 1, 2): - header = Header.fromstring(tokens[i] + tokens[i + 1], sep="\n") - result.append((wcs.WCS(header), _image_shape(header))) - return result - - -def _ccd_corners(w, shape): - """Return (ra, dec) of the 4 corners of a single CCD. - - The polygon is built from the CCD's pixel bounds ``shape = (nx, ny)`` using - pixel edges ``(-0.5, n - 0.5)`` so the quadrilateral covers the full CCD - area rather than pixel centres. Corners are returned in a consistent - counterclockwise pixel order (bottom-left, bottom-right, top-right, - top-left); since ``pixel_to_world`` maps this order to a simple, - non-self-intersecting sky quadrilateral for a CCD-sized field, and - HealSparse ``Polygon`` is orientation-agnostic, the resulting polygon is a - valid convex footprint. - - Parameters - ---------- - w : astropy.wcs.WCS - WCS of a single CCD - shape : tuple - ``(nx, ny)`` CCD image pixel dimensions - - Returns - ------- - tuple - ``(ra_list, dec_list)`` each of length 4, in degrees - """ - nx, ny = shape - - # Pixel-edge corners, counterclockwise: BL, BR, TR, TL - px = [-0.5, nx - 0.5, nx - 0.5, -0.5] - py = [-0.5, -0.5, ny - 0.5, ny - 0.5] - - sky = w.pixel_to_world(px, py) - return list(sky.ra.deg), list(sky.dec.deg) - - -class FieldCornersExtractor(object): - """Field Corners Extractor Class. - - Extracts RA/Dec coordinates of field corners from FITS headers. - """ - - def __init__(self): - """Initialize the extractor.""" - self.params_default() - - def params_default(self): - """Set default parameters and command line options.""" - - self._params = { - "input_dir": "header", - "output_file": "exp_ra_dec.txt", - "ccd_list": None, - "resume": False, - "n_processes": 1, - "verbose": False, - } - - self._short_options = { - "input_dir": "-i", - "output_file": "-o", - "ccd_list": "-l", - "resume": "-r", - "n_processes": "-n", - } - - self._types = { - "resume": "bool", - "n_processes": "int", - "verbose": "bool", - } - - self._help_strings = { - "input_dir": "input directory containing header files; default is {}", - "output_file": "output file for per-CCD corners; default is {}", - "ccd_list": "file of valid CCD IDs (output of get_ccds_with_psf); when given, only listed CCDs are written; on --resume, CCDs new to an expanded list are added without duplicating existing rows; default is all CCDs", - "resume": "resume from existing output file; default is {}", - "n_processes": f"number of parallel processes (1=serial, 0=auto={cpu_count()}); default is {{}}", - } - - def set_params_from_command_line(self, args): - """Set Params From Command line. - - Only use when calling using python from command line. - Does not work from ipython or jupyter. - - Parameters - ---------- - args : list - command line arguments - - """ - # Read command line options - options = cs_args.parse_options( - self._params, - self._short_options, - self._types, - self._help_strings, - args=args, - ) - self._params = options - - # Save calling command - logging.log_command(args) - - def update_params(self): - """Update parameters. - - Set derived parameters based on input parameters. - """ - # Ensure input directory ends without trailing slash - if self._params["input_dir"].endswith("/"): - self._params["input_dir"] = self._params["input_dir"][:-1] - - def check_params(self): - """Check parameters for validity.""" - if not exists(self._params["input_dir"]): - raise FileNotFoundError( - f"Input directory not found: {self._params['input_dir']}" - ) - - if self._params["ccd_list"] is not None and not exists( - self._params["ccd_list"] - ): - raise FileNotFoundError( - f"CCD list file not found: {self._params['ccd_list']}" - ) - - # Set n_processes to cpu_count if 0 - if self._params["n_processes"] == 0: - self._params["n_processes"] = cpu_count() - - if self._params["n_processes"] < 0: - raise ValueError( - f"n_processes must be >= 0, got {self._params['n_processes']}" - ) - - @staticmethod - def load_ccd_list(path): - """Load CCD List. - - Read valid CCD IDs (one ``-`` per line) into a set. - - Parameters - ---------- - path : str - path to the CCD list file - - Returns - ------- - set - valid CCD IDs - - """ - with open(path) as f: - return {line.strip() for line in f if line.strip()} - - @staticmethod - def process_single_header(args): - """Process Single Header. - - Worker function to process a single header file into per-CCD corners. - Static method so it can be pickled for multiprocessing. - - Parameters - ---------- - args : tuple - ``(path, verbose, valid_ccds)`` where ``path`` is the header file - path, ``verbose`` is a bool, and ``valid_ccds`` is a set of CCD IDs - to keep (or ``None`` to keep all) - - Returns - ------- - list or None - list of ``(ccd_id, ra_list, dec_list)`` for the exposure's CCDs on - success, ``None`` on failure - - """ - path, verbose, valid_ccds = args - expnum = _expnum_from_path(path) - - try: - wcs_shapes = _parse_header_to_wcs(path) - except Exception as e: - if verbose: - print(f"Failed to process {expnum}: {e}") - return None - - rows = [] - for ccd_idx, (w, shape) in enumerate(wcs_shapes): - ccd_id = f"{expnum}-{ccd_idx}" - if valid_ccds is not None and ccd_id not in valid_ccds: - continue - try: - ra, dec = _ccd_corners(w, shape) - except Exception as e: - if verbose: - print(f"Failed to process CCD {ccd_id}: {e}") - continue - rows.append((ccd_id, ra, dec)) - - return rows - - def get_done_ccds(self): - """Get Done CCDs. - - Read the set of CCD IDs already present in the output file. Resume is - keyed on individual CCD IDs, not exposure numbers: a CCD counts as done - only if its own row is present. This keeps resume correct when a write - was interrupted mid-exposure (the missing CCDs are filled in) and when - a rerun uses an expanded ``--ccd_list`` (the newly requested CCDs are - added), and it never duplicates a row. - - Returns - ------- - set - CCD IDs already written - - """ - output_file = self._params["output_file"] - - if not exists(output_file): - return set() - - try: - ids = np.atleast_1d( - np.loadtxt(output_file, usecols=(0), dtype=str) - ) - return set(ids.tolist()) - except Exception as e: - if self._params["verbose"]: - print(f"Could not read existing output file: {e}") - return set() - - def run(self, args=None): - """Run. - - Main execution method. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code (0 for success) - - """ - if args is None: - args = sys.argv[1:] - - # Set parameters from command line - self.set_params_from_command_line(args) - self.update_params() - self.check_params() - - # Get parameters - input_dir = self._params["input_dir"] - output_file = self._params["output_file"] - resume = self._params["resume"] - verbose = self._params["verbose"] - - # Load the set of valid CCD IDs to keep, if given - valid_ccds = None - if self._params["ccd_list"] is not None: - valid_ccds = self.load_ccd_list(self._params["ccd_list"]) - print(f"{len(valid_ccds)} valid CCDs in {self._params['ccd_list']}") - - # Find all header files - paths = glob.glob(f"{input_dir}/*.txt") - n = len(paths) - - if n == 0: - print(f"No header files found in {input_dir}/") - return 1 - - print(f"{n} header files found") - - # On resume, read the CCD IDs already written so we can skip them - # per-CCD (not per-exposure): a partially written exposure is completed - # rather than skipped, and no row is ever duplicated. - done_ccds = set() - if resume: - done_ccds = self.get_done_ccds() - print(f"{len(done_ccds)} CCDs already done") - - # Every header still needs parsing (a header may hold both done and - # not-yet-done CCDs); the per-CCD filter below decides what is written. - todo = paths - - # Get n_processes - n_processes = self._params["n_processes"] - - # Process headers - if n_processes == 1: - # Serial processing - results = self._process_serial(todo, verbose, valid_ccds) - else: - # Parallel processing - print(f"Using {n_processes} parallel processes") - results = self._process_parallel( - todo, n_processes, verbose, valid_ccds - ) - - # Flatten per-exposure CCD lists (dropping failed exposures), then drop - # CCDs already present in the output. - rows = [ - row - for res in results if res is not None - for row in res - if row[0] not in done_ccds - ] - - # Sort by exposure number, then CCD index - rows.sort(key=lambda r: (int(r[0].split("-")[0]), - int(r[0].split("-")[1]))) - - # Write results to file: " ra1 ra2 ra3 ra4 dec1 dec2 dec3 dec4" - mode = "a" if resume else "w" - with open(output_file, mode, buffering=1) as f: - for ccd_id, ra, dec in rows: - f.write(f"{ccd_id} ") - np.savetxt(f, ra, fmt="%9.5f", newline=" ") - np.savetxt(f, dec, fmt="%9.5f", newline=" ") - f.write("\n") - - n_exp_success = sum(1 for res in results if res is not None) - n_failed = len(todo) - n_exp_success - - print(f"Processed {n_exp_success} exposures, {len(rows)} new CCDs") - if n_failed > 0: - print(f"Failed to process {n_failed} exposures") - - print(f"Results written to {output_file}") - - return 0 - - def _process_serial(self, todo, verbose, valid_ccds): - """Process Serial. - - Process headers serially. - - Parameters - ---------- - todo : list - list of header file paths to process - verbose : bool - verbose output - valid_ccds : set or None - CCD IDs to keep, or ``None`` to keep all - - Returns - ------- - list - list of per-exposure CCD-row lists (or ``None`` for failures) - - """ - results = [] - n_todo = len(todo) - - for i, p in enumerate(todo): - result = self.process_single_header((p, verbose, valid_ccds)) - results.append(result) - - if verbose and i % 100 == 0: - print(f"{i:6d} / {n_todo:6d}") - - return results - - def _process_parallel(self, todo, n_processes, verbose, valid_ccds): - """Process Parallel. - - Process headers in parallel using multiprocessing. - - Parameters - ---------- - todo : list - list of header file paths to process - n_processes : int - number of parallel processes - verbose : bool - verbose output - valid_ccds : set or None - CCD IDs to keep, or ``None`` to keep all - - Returns - ------- - list - list of per-exposure CCD-row lists (or ``None`` for failures) - - """ - # Prepare arguments for worker function - args = [(p, verbose, valid_ccds) for p in todo] - - # Create pool and process - with Pool(processes=n_processes) as pool: - results = pool.map(self.process_single_header, args) - - return results diff --git a/src/shapepipe/utilities/header_downloader.py b/src/shapepipe/utilities/header_downloader.py deleted file mode 100644 index 122327a58..000000000 --- a/src/shapepipe/utilities/header_downloader.py +++ /dev/null @@ -1,291 +0,0 @@ -"""HEADER_DOWNLOADER - -Download FITS headers from VOSpace for exposures listed in a CCD file. - -Author: Mike Hudson, Martin Kilbinger - -""" - -import sys -import os -from os.path import exists - -import numpy as np -import vos - -from cs_util import args as cs_args -from cs_util import logging - - -class HeaderDownloader(object): - """Header Downloader Class. - - Downloads FITS headers from VOSpace for exposures in a CCD list. - """ - - def __init__(self): - """Initialize the downloader.""" - self.params_default() - - def params_default(self): - """Set default parameters and command line options.""" - - self._params = { - "input_file": None, - "output_dir": "header", - "vospace_path": "vos:cfis/pitcairn", # UNIONS/CFIS default; override for other surveys - "overwrite": False, - "dir_for_links": None, - "verbose": False, - } - - self._short_options = { - "input_file": "-i", - "output_dir": "-o", - "vospace_path": "-p", - "overwrite": "-O", - "dir_for_links": "-d", - } - - self._types = { - "overwrite": "bool", - "verbose": "bool", - } - - self._help_strings = { - "input_file": "input CCD list file (txt or csv); required", - "output_dir": "output directory for headers; default is {}", - "vospace_path": "VOSpace base path; default is {}", - "overwrite": "overwrite existing header files; default is {}", - "dir_for_links": "directory to check for existing headers to link instead of download; default is {}", - } - - def set_params_from_command_line(self, args): - """Set Params From Command line. - - Only use when calling using python from command line. - Does not work from ipython or jupyter. - - Parameters - ---------- - args : list - command line arguments - - """ - # Read command line options - options = cs_args.parse_options( - self._params, - self._short_options, - self._types, - self._help_strings, - args=args, - ) - self._params = options - - # Save calling command - logging.log_command(args) - - def update_params(self): - """Update parameters. - - Set derived parameters based on input parameters. - """ - # Ensure output directory ends without trailing slash - if self._params["output_dir"].endswith("/"): - self._params["output_dir"] = self._params["output_dir"][:-1] - - def check_params(self): - """Check parameters for validity.""" - if self._params["input_file"] is None: - raise ValueError("Input file is required (use -i or --input_file)") - - if not exists(self._params["input_file"]): - raise FileNotFoundError( - f"Input file not found: {self._params['input_file']}" - ) - - # Check if dir_for_links exists if specified - if self._params["dir_for_links"] is not None: - if not exists(self._params["dir_for_links"]): - raise FileNotFoundError( - f"Directory for links not found: {self._params['dir_for_links']}" - ) - if not os.path.isdir(self._params["dir_for_links"]): - raise ValueError( - f"Path is not a directory: {self._params['dir_for_links']}" - ) - - # Create output directory if it doesn't exist - if not exists(self._params["output_dir"]): - os.makedirs(self._params["output_dir"]) - if self._params["verbose"]: - print(f"Created output directory: {self._params['output_dir']}") - - def get_exposures(self, ccd_list_file): - """Get Exposures. - - Extract unique exposure numbers from CCD list file. - - Parameters - ---------- - ccd_list_file : str - path to CCD list file (txt or csv) - - Returns - ------- - np.array - unique exposure numbers - - """ - exps = [] - - # Check if CSV format - if ccd_list_file.endswith(".csv"): - import astropy.table - - t = astropy.table.Table.read(ccd_list_file) - expccd = t["CCD"].data - r = np.char.split(expccd, sep="-") - - for i, r1 in enumerate(r): - exp = int(r1[0]) - exps.append(exp) - else: - # Text format; atleast_1d so a single-line file (0-d array) is - # still iterable. - f = np.atleast_1d( - np.loadtxt(ccd_list_file, dtype="str", encoding="ascii") - ) - r = np.char.split(f, sep="-") - - for i, r1 in enumerate(r): - exp = int(r1[0]) - exps.append(exp) - - exps = np.array(exps) - uniq = np.unique(exps) - - return uniq - - def get_fits_header(self, expnum, client): - """Get FITS Header. - - Download FITS header from VOSpace, or create a symbolic link - if the header exists in dir_for_links. - - Parameters - ---------- - expnum : int - exposure number - client : vos.Client - VOSpace client - - Returns - ------- - bool - True if successful, False otherwise - - """ - vospace_path = self._params["vospace_path"] - output_dir = self._params["output_dir"] - overwrite = self._params["overwrite"] - dir_for_links = self._params["dir_for_links"] - - source = f"{vospace_path}/{expnum:d}p.fits.fz" - dest = f"{output_dir}/{expnum:d}.txt" - - if exists(dest) and not overwrite: - return True - - # Check if header exists in dir_for_links - if dir_for_links is not None: - link_source = os.path.abspath(f"{dir_for_links}/{expnum:d}.txt") - if exists(link_source): - try: - # Remove existing file/link if overwrite is True - if exists(dest): - os.remove(dest) - # Create symbolic link - os.symlink(link_source, dest) - return True - except Exception as e: - print(f"Could not create symlink from {link_source}: {e}") - # Fall through to download if symlink fails - - # Download from VOSpace atomically: copy to a temp file in the same - # directory, then rename on success. An interrupted transfer leaves - # only the temp file behind, so resume never treats a partial download - # as complete. - tmp_dest = f"{dest}.part" - try: - client.copy(source, tmp_dest, head=True) - os.rename(tmp_dest, dest) - return True - except Exception as e: - print(f"Could not copy {source}: {e}") - if exists(tmp_dest): - os.remove(tmp_dest) - return False - - def run(self, args=None): - """Run. - - Main execution method. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code (0 for success) - - """ - if args is None: - args = sys.argv[1:] - - # Set parameters from command line - self.set_params_from_command_line(args) - self.update_params() - self.check_params() - - # Get parameters - input_file = self._params["input_file"] - output_dir = self._params["output_dir"] - verbose = self._params["verbose"] - - if verbose: - print(f"Reading CCD list from: {input_file}") - - # Extract unique exposures - exps = self.get_exposures(input_file) - - print(f"Found {len(exps)} unique exposures") - - if verbose: - print(f"Downloading headers to: {output_dir}/") - - # Initialize VOSpace client once - client = vos.Client() - - # Download headers - n_success = 0 - n_failed = 0 - - for i, exp in enumerate(exps): - success = self.get_fits_header(exp, client) - if success: - n_success += 1 - else: - n_failed += 1 - - if verbose and i % 100 == 0: - print(f"{i:6d} / {len(exps):6d}") - - print(f"Downloaded {n_success} headers") - if n_failed > 0: - print(f"Failed to download {n_failed} headers") - - return 0 diff --git a/src/shapepipe/utilities/summary.py b/src/shapepipe/utilities/summary.py deleted file mode 100755 index dd5009f3f..000000000 --- a/src/shapepipe/utilities/summary.py +++ /dev/null @@ -1,832 +0,0 @@ -"""SUMMARY - -Author: Martin Kilbinger - -""" - -import sys -import os -import re -import fnmatch - -import logging - -from collections import Counter - -from tqdm import tqdm - - -def init_par_runtime(list_tile_IDs): - - # Numbers updated at runtime - par_runtime = {} - - par_runtime["n_tile_IDs"] = len(list_tile_IDs) - par_runtime["list_tile_IDs"] = list_tile_IDs - - return par_runtime - - -def update_par_runtime_after_find_exp(par_runtime, all_exposures): - - # Single-exposure images - par_runtime["n_exposures"] = len(all_exposures) - par_runtime["list_exposures"] = all_exposures - - # Single-HDU single exposure images - n_CCD = 40 - par_runtime["n_shdus"] = get_par_runtime(par_runtime, "exposures") * n_CCD - par_runtime["list_shdus"] = get_all_shdus(all_exposures, n_CCD) - - return par_runtime - - -def get_IDs_from_file(path): - """Get IDs From File. - - Return IDs from text file. Removes letters and replaces - dots "." with dashes "-". - - Parameters - ---------- - path: str - input file path - - Returns - -------- - list - IDs - - """ - numbers = [] - with open(path) as f_in: - for line in f_in: - entry = line.rstrip() - number = re.sub("[a-zA-Z]", "", entry) - numbers.append(number) - - return numbers - - -def get_all_exposures(exp_number_file_list, verbose=False): - """Get All Exposures. - - Return all exposure names from a list of text files. - - Parameters - ---------- - exp_number_list: list - input file names - - """ - exposures = set() - for idx, path in enumerate(exp_number_file_list): - exps = get_IDs_from_file(path) - exposures.update(exps) - - return list(exposures) - - -def get_all_shdus(exposures, n_CCD): - """Get All SHDUs. - - Return all single-exposure single-HDU (CCD) IDs. - - Parameters - ---------- - exposures: list - exposure names - n_CCD: int - number of CCDs per exposure - - Returns - -------- - list - single-exposure single-HDU IDs - - """ - shdus = [] - for exposure in exposures: - for idx_CCD in range(n_CCD): - shdus.append(f"{exposure}-{idx_CCD}") - - return shdus - - -def set_as_list(item=None, n=None, default=1): - """Set As List. - - Return input as list. - - Parameters - ----------- - item: str, int, or list, optional - input item(s); default is None, in which - case the return is [1] * n - n: int, optional - number of list items to return, default is None, - in which case the number will be set to 1. If item and - n are not None, n has to be equal to len(item) - default: int, optional - value to return if item is not given; - default is 1 - - Raises - ------- - IndexError - if n != len(item) - - Returns - ------- - list - input item(s) as list - """ - my_n = n or 1 - - if not item: - result = [default] * my_n - elif not isinstance(item, list): - result = [item] * my_n - else: - result = item - if len(item) != my_n: - raise IndexError(f"item has length {len(item)} != {n}") - - return result - - -def check_special_one(module, path): - - ngmix_finished = False - - with open(path) as f_in: - lines = f_in.readlines() - for line in lines: - entry = line.rstrip() - - if module == "setools_runner": - m = re.search("Nb stars = (\S*)", line) - if m: - value = int(m[1]) - if value < 2: - code = 0 - msg = ( - f"Not enough stars for random split:" - + f" #stars = {value}" - ) - return msg, code - break - m = re.search("Mode computation failed", line) - if m: - code = 1 - msg = "Mode computation of stellar locus failed" - return msg, code - - if module == "psfex_interp_runner": - m = re.search("Key N_EPOCH not found", line) - if m: - code = 2 - msg = "N_EPOCH not in SEx cat, rerun job 16" - return msg, code - - m = re.search( - "ValueError: cannot reshape array of size 0 into shape", - line, - ) - if m: - code = 3 - msg = "found array of size 0" - return msg, code - - if module == "mask_runner": - m = re.search("Empty or corrupt FITS file", line) - if m: - code = 4 - msg = "empty or corrult FITS file" - return msg, code - - if module == "sextractor_runner": - m = re.search("sextracted 0", line) - if m: - code = 5 - msg = "No object detected (weight might be 0 everywhere)" - return msg, code - m = re.search("astropy\.wcs\.wcs\.NoConvergence", line) - if m: - code = 8 - msg = "WCS world2pix did not converge" - return msg, code - - if module == "ngmix_runner": - m = re.search("finished", line) - if m: - ngmix_finished = True - break - - if module == "merge_sep_cats_runner": - m = re.search("Input catalogue", line) - if m: - code = 7 - msg = "One or more ngmix catalogues not found for merge" - return msg, code - - if module == "ngmix_runner" and not ngmix_finished: - code = 6 - msg = "ngmix incomplete" - return msg, code - - - return None, None - - -class job_data(object): - """Job Data. - - Class to handle a job. - - Parameters - ---------- - bit: int - bit-coded job number - run_dir: str or list - run directory(ies) - modules: list - module names - key_expected: int or str - number of expected output files; if str: will be updated - with runtime value - n_mult: int or list, optional - multiplicity of output files, default `None`, in which - case it is set to 1 - pattern: list, optional - if not None, file pattern to match; defafult is `None` - path_main: str, optional - main (left-most) part of output directory, default is "." - path_left: str, optional - left (first) part of output directory, default is "./output" - output_subdirs: str, optional - output subdirectories if not `None`; default is `None` - path_right: str, optional - right (last) part of output subdir suffix if not `None`; - default is `None` - path_output: str, optional - module output path, default is "output" - output_path_missing_IDs: list, optional - output path of missing ID, if `None` (default) will be - given by job bit and module. - special: bool, optional - if True check output file content for special messages; - default is False - verbose: bool, optional - verbose output if True; default is False - - """ - - def __init__( - self, - bit, - run_dir, - modules, - key_expected, - n_mult=None, - pattern=None, - path_main=".", - path_left="output", - output_subdirs=None, - path_right=None, - path_output="output", - output_path_missing_IDs=None, - special=False, - verbose=False, - ): - self._bit = bit - self._run_dir = set_as_list(item=run_dir, n=len(modules)) - self._modules = modules - self._key_expected = set_as_list(item=key_expected, n=len(modules)) - self._n_mult = set_as_list(item=n_mult, n=len(modules)) - self._pattern = set_as_list(item=pattern, n=len(modules), default="") - self._path_main = path_main - self._path_left = path_left - self._output_subdirs = output_subdirs or [""] - self._path_right = set_as_list( - path_right, - len(modules), - default=".", - ) - self._path_output = set_as_list( - path_output, - len(modules), - default="output", - ) - self._output_path_missing_IDs = output_path_missing_IDs - self._special = set_as_list( - special, - len(modules), - default=False, - ) - self._path_right = set_as_list(path_right, len(modules), default=".") - self._output_path_missing_IDs = output_path_missing_IDs - self._verbose = verbose - - def print_intro(self): - """Print Intro. - - Print header line for job statistics. - - """ - logging.info(f" # Job {self._bit}:") - - @classmethod - def print_stats_header(self): - """Print Stats Header. - - Print overall header information for stats output. - - """ - logging.info( - "module expected found" - + " missing uniq_miss fr_found" - ) - logging.info("=" * 100) - - def print_stats( - self, - module, - n_expected, - n_found, - n_special, - n_missing, - idx, - ): - """Print Stats. - - Print output file statistics. - - Parameters - ---------- - module: str - module name - n_expected: int - number of expected files - n_found: int - number of found files - n_special: int - number of special cases - n_missing: int - number of missing files - idx: int - module index - - """ - module_str = module - - if not self._special[idx]: - if n_expected > 0: - fraction_found = n_found / n_expected - else: - fraction_found = 1 - - n_missing_per_mult = n_missing / self._n_mult[idx] - - else: - module_str = f"{module_str} (special)" - n_found = n_special - n_missing = -1 - n_missing_per_mult = -1 - fraction_found = n_found / n_expected - n_expected = -1 - - logging.info( - f"{module_str:30s} {n_expected:9d} {n_found:9d}" - + f" {n_missing:9d}" - + f" {n_missing_per_mult:9.1f} {fraction_found:9.1%}" - ) - - @classmethod - def is_ID_in_str(self, ID, path): - if ID in path: - return True - - @classmethod - def is_not_in_any(self, ID, list_str): - return not any(ID in string for string in list_str) - - @classmethod - def replace_dot_dash(self, numbers): - - results = [re.sub("\.", "-", number) for number in numbers] - - return results - - @classmethod - def replace_dash_dot_if_tile(self, numbers): - - pattern = re.compile(r"(\d{3})-(\d{3})") - results = [pattern.sub(r"\1.\2", number) for number in numbers] - - return results - - @classmethod - def get_unique(self, names): - n_all = len(names) - names_unique = list(set(names)) - n_unique = len(names_unique) - - if n_all != n_unique: - if True: # self._verbose: - logging.warning( - f"{n_all - n_unique} duplicates removed from {n_all} IDs" - ) - - return names_unique - - @classmethod - def write_IDs_to_file(self, output_path, IDs): - """Write IDs to file. - - Write list if image IDs to text file. - - Parameters - ---------- - output_path: str - output file path - IDs: list - image IDs - - """ - IDs_dot = self.replace_dash_dot_if_tile(IDs) - if len(IDs_dot) > 0: - # Write IDs to file - with open(output_path, "w") as f_out: - for ID in IDs_dot: - print(ID, file=f_out) - elif os.path.exists(output_path): - # Remove preivous obsolete ID file - os.unlink(output_path) - - def check_special(self, module, idx): - - messages = {} - - if self._special[idx]: - - # Loop over input file names and paths - for name, path in zip(self._names_in_dir[idx], self._paths_in_dir[idx]): - - # Check if special case is found - msg, code = check_special_one(module, path) - if msg: - # First time occurance: create empty list for this code - if code not in messages: - messages[code] = [] - - # Append file name, message, and code - messages[code].append(f"{name} {code} {msg}") - - if len(messages) > 0: - # Loop over codes = key in messages dict - for code in messages: - # Create output file for this code - output_path = ( - f"{self._path_main}/summary/special_job_{self._bit}" - + f"_{module}_{code}.txt" - ) - # Write all messages - with open(output_path, "w") as f_out: - for msg in messages[code]: - print(msg, file=f_out) - - # Count all special cases = sum of cases over all codes - n_all = sum([len(messages[code]) for code in messages]) - return n_all - - def output_missing( - self, - module, - idx, - par_runtime=None, - ): - """Output Missing. - - Writes IDs of missing images to disk. - - """ - key_expected = self._key_expected[idx] - names_in_dir = self._names_in_dir[idx] - paths_in_dir = self._paths_in_dir[idx] - n_mult = self._n_mult[idx] - - list_expected = get_par_runtime(par_runtime, key_expected, kind="list") - - # Count image IDs in names that were found earlier - - # Get file name pattern - if module != "split_exp_runner" or (self._bit != 2 and self._bit != 4096): - pattern = re.compile(r"(?:\d{3}-\d{3}|\d{7}-\d+|\d{7})") - else: - # split_exp_runner with sp_local=0: input is exp, output is shdu - # (images) and exp (header); ignore hdu number. - # If sp_local=1 set bit to != 2 - # Update 11/2025: No longer working for P9 4096. Solution: set n_mult=3. - pattern = re.compile( - r"(?:\d{3}-\d{3}|\d{7})" - ) - - ## Extract image IDs from names - IDs = [] - for name, path in zip(names_in_dir, paths_in_dir): - - match = pattern.search(name) - if match: - ID = match.group() - IDs.append(ID) - else: - msg = f"No ID found in {name}" - #raise ValueError(msg) - print(f"Warning: {msg}, continuing") - - # For split_exp_runner P8, IDs now contain exps and sdus, - # not matching mult. - - ## Count occurences - ID_counts = Counter(IDs) - - ## Add to missing if ocurence less than n_mult - missing_IDs = [] - for ID in list_expected: - if ID_counts[ID] < n_mult: - missing_IDs.append(ID) - - n_all = len(missing_IDs) - missing_IDs_unique = self.get_unique(missing_IDs) - - if not self._output_path_missing_IDs: - # Default name using bit and module - output_path = ( - f"{self._path_main}/summary/missing_job_{self._bit}" - + f"_{module}.txt" - ) - else: - # User-defined name (e.g. ngmix_runner_X) - output_path = self._output_path_missing_IDs[idx] - self.write_IDs_to_file(output_path, missing_IDs_unique) - - return missing_IDs_unique - - def output_missing_job(self): - output_path = ( - f"{self._path_main}/summary/missing_job_{self._bit}_all.txt" - ) - - missing_IDs_all = set(self._missing_IDs_job) - - self.write_IDs_to_file(output_path, missing_IDs_all) - - @classmethod - def get_last_full_path(self, base_and_subdir, matches): - """Get Last Full Path - - Return full path of last file in list. - - """ - # Sort according to creation time - matches_sorted = sorted( - matches, - key=lambda entry: entry.name, - ) - - # Get most recent one - last = matches_sorted[-1] - - # Get full path - full_path = os.path.join(base_and_subdir, last.name) - - return full_path - - @classmethod - def get_module_output_dir(self, full_path, module, path_output): - """Get Module Output Dir. - - Return output directory name for given module. - - """ - directory = f"{full_path}/{module}/{path_output}" - - return directory - - def get_matches_final(self, directory, idx): - - # Loop over files - # os.path.whether exists is twice faster than try/except - - if os.path.exists(directory): - pattern = f"{self._pattern[idx]}*" - for entry2 in os.scandir(directory): - if ( - entry2.is_file() - and (fnmatch.fnmatch(entry2.name, pattern)) - and entry2.stat().st_size > 0 - ): - # Append matching files - self._names_in_dir[idx].append(entry2.name) - self._paths_in_dir[idx].append( - os.path.join(directory, entry2.name) - ) - - def get_names_in_dir(self, iterable, module, idx): - - # Initialise output file names and paths - self._names_in_dir[idx] = [] - self._paths_in_dir[idx] = [] - - # Loop over subdirs - for jdx, subdir in enumerate(iterable): - base_and_subdir = ( - f"{self._path_main}/" - + f"{self._path_left}/{subdir}/" - + f"{self._path_right[idx]}" - ) - if self._verbose: - print(f"**** base_and_subdir {base_and_subdir}") - - if os.path.isdir(base_and_subdir): - - matches = [] - - # Loop over entries (files and dirs) - with os.scandir(base_and_subdir) as entries: - for entry in entries: - - # Append directory name if matches module - if ( - entry.name.startswith(self._run_dir[idx]) - and not entry.name.endswith("prev") - ): - matches.append(entry) - - # This entry does not match module -> next - if not matches: - continue - - if self._verbose: - print("**** Matching entries: ", end="") - for match in matches: - print(match.name) - - full_path = self.get_last_full_path( - base_and_subdir, - matches, - ) - - # Get module output directory - directory = self.get_module_output_dir( - full_path, - module, - self._path_output[idx], - ) - if self._verbose: - print(f"**** Output dir = {directory}") - - # Find matching file names and paths - self.get_matches_final(directory, idx) - else: - if self._verbose: - print(f"Directory {base_and_subdir} not found") - - def update_subdirs(self, par_runtime): - """Update Subdirs. - - Update subdir names with runtime information if required. - - """ - if not isinstance(self._output_subdirs, list): - self._output_subdirs = get_par_runtime( - par_runtime, self._output_subdirs, kind="list" - ) - - def check_numbers(self, par_runtime=None, indices=None): - """Check Numbers. - - Check output file numbers and IDs. - - Parameters - ---------- - par_runtime : dict, optional - runtime parameter. default is None - indices: list, optional - if not None (default), only check modules corresponding - to indices - - """ - # Update subdirs if not already set as list - self.update_subdirs(par_runtime) - - # Initialise variables - self._names_in_dir = {} - self._paths_in_dir = {} - self._missing_IDs_job = [] - n_missing_job = 0 - - # Loop over modules - for idx, module in enumerate(self._modules): - if indices is not None and idx not in indices: - continue - - if self._verbose: - print(f"** module {module}") - - # Look over subdirs - iterable = self._output_subdirs - if len(iterable) > 1 and self._verbose: - iterable = tqdm(iterable, desc="subdirs", leave=False) - - if self._verbose: - print(f"*** subdirs {self._output_subdirs}") - - # Get output file names and paths - self.get_names_in_dir( - iterable, - module, - idx, - ) - - # If expected is string: Update parameter with runtime value - # and set as integer - if isinstance(self._key_expected[idx], str): - n_expected_base = get_par_runtime( - par_runtime, self._key_expected[idx], kind="n" - ) - else: - n_expected_base = self._key_expected[idx] - - # Get some numbers - n_found = len(self._names_in_dir[idx]) - n_expected = n_expected_base * self._n_mult[idx] - n_missing = n_expected - n_found - - n_special = self.check_special( - module, - idx, - ) - - # Print statistics - self.print_stats( - module, - n_expected, - n_found, - n_special, - n_missing, - idx, - ) - - # Write missing IDs for module to file - if n_missing > 0: - missing_IDs = self.output_missing( - module, - idx, - par_runtime=par_runtime, - ) - n_missing_job += n_missing - self._missing_IDs_job.extend(missing_IDs) - - # Empty line after job - logging.info("") - - # Write missing IDs for entire job to file - # if n_missing_job > 0: - self.output_missing_job() - - -def get_par_runtime(par_runtime, key, kind="n"): - """Get Par RunTime. - - Return runtime parameter value. - - Parameters - ---------- - par_runtime: dict - runtime parameter - key: str - key - - """ - combined_key = f"{kind}_{key}" - - return par_runtime[combined_key] - - -def print_par_runtime(par_runtime, verbose=True): - # Print runtime parameter values - if True: - logging.info("") - logging.info("===========") - logging.info("par_runtime") - logging.info("-----------") - for key, value in par_runtime.items(): - if not key.startswith("list"): - logging.info(f"{key:30s} {value:6d}") - else: - # logging.info(f"{key:30s} {len(value):6d} entries") - pass - logging.info("===========") - logging.info("") diff --git a/src/shapepipe/utilities/summary_params_pre_v2.py b/src/shapepipe/utilities/summary_params_pre_v2.py deleted file mode 100644 index ea67442e3..000000000 --- a/src/shapepipe/utilities/summary_params_pre_v2.py +++ /dev/null @@ -1,276 +0,0 @@ -# Parameters for summary run - -import logging -import os - -from shapepipe.utilities import summary - - -def set_jobs_v2_pre_v2(patch, verbose): - """Return information about shapepipe jobs""" - print(f"Set job info for patch {patch}") - - # Main input and output directory - path_main = f"{os.environ['HOME']}/cosmostat/v2/pre_v2/psfex/{patch}" - - # Logging - path = f"{path_main}/summary" - if not os.path.isdir(path): - os.mkdir(path) - log_file_name = f"{path}/summary_log.txt" - handlers = [ - logging.FileHandler(log_file_name, mode="w"), - logging.StreamHandler(), - ] - logging.basicConfig( - level=logging.INFO, format="%(message)s", handlers=handlers - ) - - logging.info(f"Checking main directory = {path_main}") - - # Tile IDs - tile_ID_path = f"{path_main}/tile_numbers.txt" - - ## Tile IDs with dots - list_tile_IDs_dot = summary.get_IDs_from_file(tile_ID_path) - - jobs = {} - - # Set the first job (retrieve images) - - # With "CFIS_" only the linked images are counted. The original - # ones do not match the IDdash pattern. - # If images were downloaded in several runs: - # - Only copy original images, then (re-)set links in SP numbering format - # - get_images_runner_run_[12] consistent - # - remove previous output dirs since only last is searched - jobs["1"] = summary.job_data( - 1, - "run_sp_GitFeGie", - [ - "get_images_runner_run_1", - "find_exposures_runner", - "get_images_runner_run_2", - ], - ["tile_IDs", "tile_IDs", "exposures"], - pattern=["CFIS_", "", ""], - n_mult=[2, 1, 3], - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - if patch in ("P8", "P9"): - jobs["2"] = summary.job_data( - 2, - ["run_sp_Uz", "run_sp_exp_Sp_shdu"], - ["uncompress_fits_runner", "split_exp_runner"], - ["tile_IDs", "shdus"], - n_mult=[1, 3], - path_main=path_main, - path_left=["output", "exp_runs"], - output_subdirs=[None, "shdus"], - path_right=[None, "output"], - verbose=verbose, - ) - jobs["4096"] = summary.job_data( - 4096, - ["run_sp_exp_Sp_shdu"], - ["split_exp_runner"], - ["shdus"], - n_mult=4, - path_main=path_main, - path_left="exp_runs", - output_subdirs="shdus", - path_right="output", - verbose=verbose, - ) - else: - jobs["2"] = summary.job_data( - 2, - ["run_sp_Uz", "run_sp_exp_SpMh"], - ["uncompress_fits_runner", "split_exp_runner"], - ["tile_IDs", "shdus"], - n_mult=[1, 121], - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - jobs["4"] = summary.job_data( - 4, - ["run_sp_Ma_tile"], - ["mask_runner"], - ["tile_IDs"], - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - if patch not in ("P8", "P9"): - jobs["8"] = summary.job_data( - 8, - ["run_sp_Ma_exp"], - ["mask_runner"], - ["shdus"], - path_main=path_main, - path_left="output", - verbose=verbose, - ) - else: - jobs["8"] = summary.job_data( - 8, - ["run_sp_exp_Ma"], - ["mask_runner"], - ["shdus"], - n_mult=[1], - path_main=path_main, - path_left="exp_runs", - output_subdirs= "shdus", - path_right="output", - verbose=verbose, - ) - - - jobs["16"] = summary.job_data( - 16, - "run_sp_tile_Sx", - ["sextractor_runner", "sextractor_runner"], - "tile_IDs", - n_mult=[2, 1], - path_main=path_main, - path_left="tile_runs", - path_output=["output", "logs"], - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - special=[False, True], - verbose=verbose, - ) - - jobs["32"] = summary.job_data( - 32, - [ - "run_sp_exp_SxSePsf", - "run_sp_exp_SxSePsf", - "run_sp_exp_SxSePsf", - "run_sp_exp_SxSePsf", - "run_sp_exp_SxSePsf", - ], - [ - "sextractor_runner", - "setools_runner", - "setools_runner", - "setools_runner", - "psfex_runner", - ], - "shdus", - n_mult=[2, 2, 1, 1, 2], - path_main=path_main, - path_left="exp_runs", - output_subdirs="shdus", - path_right="output", - path_output=[ - "output", - "output/rand_split", - "output/stat", - "logs", - "output", - ], - special=[False, False, True, True, False], - verbose=verbose, - ) - - jobs["64"] = summary.job_data( - "64", - "run_sp_tile_PsViSmVi", - [ - "psfex_interp_runner", - "psfex_interp_runner", - "vignetmaker_runner_run_1", - "spread_model_runner", - "vignetmaker_runner_run_2", - ], - "tile_IDs", - n_mult=[1, 1, 1, 1, 4], - path_main=path_main, - path_left="tile_runs", - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - path_output=["output", "logs", "output", "output", "output"], - special=[False, True, False, False, False], - verbose=verbose, - ) - - if patch in ("P2", "P5", "P8", "P9"): - n_sh = 1 - else: - n_sh = 8 - run_dirs = [f"run_sp_tile_ngmix_Ng{idx+1}u" for idx in range(n_sh)] - - # Add special (unfinished run) - run_dirs.append("run_sp_tile_ngmix_Ng1u") - - output_path_missing_IDs = [ - f"{path_main}/summary/missing_job_128_ngmix_runner_{idx+1}.txt" - for idx in range(n_sh + 1) - ] - jobs["128"] = summary.job_data( - "128", - run_dirs, - ["ngmix_runner"] * (n_sh + 1), - "tile_IDs", - path_main=path_main, - path_left="tile_runs", - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - path_output=["output"] * n_sh + ["logs"], - special=[False] * n_sh + [True], - output_path_missing_IDs=output_path_missing_IDs, - verbose=verbose, - ) - - jobs["256"] = summary.job_data( - "256", - "run_sp_tile_Ms", - ["merge_sep_cats_runner"] * 2, - "tile_IDs", - path_main=path_main, - path_left="tile_runs", - path_output=["output", "logs"], - special=[False, True], - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - verbose=verbose, - ) - - jobs["512"] = summary.job_data( - "512", - ["run_sp_tile_Mc"], - ["make_cat_runner"], - "tile_IDs", - path_main=path_main, - path_left="tile_runs", - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - verbose=verbose, - ) - - # Post-processing - jobs["1024"] = summary.job_data( - "1024", - ["run_sp_combined_final"], - ["make_catalog_runner"], - "tile_IDs", - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - jobs["2048"] = summary.job_data( - "2048", - "run_sp_combined_psf", - ["psfex_interp_runner"], - "shdus", - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - return jobs, list_tile_IDs_dot - - diff --git a/tests/module/test_collate_star_cat.py b/tests/module/test_collate_star_cat.py deleted file mode 100644 index 20d52ce23..000000000 --- a/tests/module/test_collate_star_cat.py +++ /dev/null @@ -1,70 +0,0 @@ -"""UNIT TESTS FOR STAR-CATALOGUE COLLATION PATHS. - -Pin the patch vs patch-less (v2.0) path and filename convention of -``scripts/python/collate_star_cat.py``. Runs up to v1.6 carry a ``P`` -token in both the input run directory and the output filename; v2.0 is -patch-less (``patch is None``) and drops that token, reading from a single -``/output`` root and writing ``validation_psf_conv-.fits`` — the -name still matched by the downstream ``validation_psf_conv-*`` glob. -""" - -import importlib.util -from pathlib import Path - -import pytest - -# The collation script lives under scripts/python (not an importable package), -# so load it by path. -_SCRIPT = ( - Path(__file__).resolve().parents[2] - / "scripts" - / "python" - / "collate_star_cat.py" -) -_spec = importlib.util.spec_from_file_location("collate_star_cat", _SCRIPT) -collate_star_cat = importlib.util.module_from_spec(_spec) -_spec.loader.exec_module(collate_star_cat) - - -@pytest.mark.parametrize( - "patch, exp_input, exp_output", - [ - ("3", "in/P3/output/", "out/P3"), - (None, "in/output/", "out"), - ], -) -def test_collate_paths(patch, exp_input, exp_output): - """v1.x carries the P token; v2.0 (patch None) drops it.""" - assert collate_star_cat.collate_paths("in", "out", patch) == ( - exp_input, - exp_output, - ) - - -@pytest.mark.parametrize( - "patch, expected", - [ - ("3", "validation_psf_conv-3-0.fits"), - (None, "validation_psf_conv-0.fits"), - ], -) -def test_output_filename(patch, expected): - """The patch token is present for v1.x and absent for v2.0.""" - assert collate_star_cat.output_filename("validation_psf", patch, 0) == expected - - -def test_output_filename_matches_downstream_glob(): - """Both layouts stay under the downstream ``validation_psf_conv-*`` glob.""" - for patch in ("1", None): - assert collate_star_cat.output_filename( - "validation_psf", patch, 5 - ).startswith("validation_psf_conv-") - - -@pytest.mark.parametrize("bad", ["v2", "2.0", "v1.7", ""]) -def test_invalid_version_raises(bad): - """A mistyped -V is rejected rather than falling through to v1.x.""" - obj = collate_star_cat.Convert() - obj._params["version_cat"] = bad - with pytest.raises(ValueError): - obj.run() diff --git a/tests/module/test_coverage.py b/tests/module/test_coverage.py index ea48905c3..26d98a809 100644 --- a/tests/module/test_coverage.py +++ b/tests/module/test_coverage.py @@ -1,49 +1,35 @@ -"""UNIT / PROPERTY TESTS FOR THE COVERAGE-MASK FEATURE. - -Covers the pure and lightly-fixtured logic behind the per-CCD coverage nexp -masks: exposure-number parsing, the CCD-list -> unique-exposure reduction, the -handler's missing-CCD subtraction (pinned against the real ID format), per-CCD -corner extraction from plain and fpack-compressed multi-HDU headers, -``--ccd_list`` filtering, the per-CCD resume path, the builder's per-CCD row -parsing, the accumulated exposure-count (nexp) contract, the RA-wrap and pole -guards, the power-of-two ``nside`` validation, the atomic-download rename, and -the ``-h``/argv handling of the console runners. The heavy paths (real VOSpace -download, production-resolution map building, plotting, multiprocessing) are -exercised end to end by the pipeline, not here. +"""UNIT / PROPERTY TESTS FOR THE COVERAGE-MASK GEOMETRY. + +Covers the pure logic the coverage nexp mask rests on, on both sides of the +``exp_footprint`` -> ``coverage_map`` handover: per-CCD image-shape resolution +from plain and fpack-compressed headers, the sky-corner projection, and then +the accumulated exposure-count (nexp) contract, the RA-wrap and pole guards, +the power-of-two ``nside`` validation and the median smoothing of a finished +map. Everything here takes arrays or headers in memory; the way a record +reaches these functions is ``tests/unit/test_exp_footprint.py``'s subject, and +the heavy paths (production-resolution map building, plotting) are exercised +end to end by the pipeline. """ -import sys -from types import SimpleNamespace - +import healsparse as hsp +import hpgeom as hpg import numpy as np import numpy.testing as npt import pytest from astropy import wcs -from hypothesis import given -from hypothesis import strategies as st +from astropy.io.fits import Header -from shapepipe.utilities import summary -from shapepipe.utilities.ccd_psf_handler import CcdPsfHandler +from shapepipe.utilities.ccd_footprint import _ccd_corners, _image_shape from shapepipe.utilities.coverage_map_builder import ( - CoverageMapBuilder, + build_map, + check_nside, + median_filter, unwrap_ra, ) -from shapepipe.utilities.field_corners_extractor import ( - FieldCornersExtractor, - _ccd_corners, - _expnum_from_path, - _image_shape, - _parse_header_to_wcs, -) -from shapepipe.utilities.header_downloader import HeaderDownloader def _tan_wcs(crval_ra, crval_dec=0.0, nx=2080, ny=4612): - """Build a minimal 2-D TAN WCS with a populated pixel shape. - - ``nx``/``ny`` set ``pixel_shape`` so ``_header_text`` can emit matching - ``NAXIS1``/``NAXIS2`` keywords. - """ + """Build a minimal 2-D TAN WCS with a populated pixel shape.""" w = wcs.WCS(naxis=2) w.wcs.ctype = ["RA---TAN", "DEC--TAN"] w.wcs.crval = [crval_ra, crval_dec] @@ -53,244 +39,61 @@ def _tan_wcs(crval_ra, crval_dec=0.0, nx=2080, ny=4612): return w -def _header_text(wcs_list): - """Render a multi-HDU header text file: a primary HDU then one plain image - HDU per WCS, each carrying ``NAXIS1``/``NAXIS2``. - """ - blocks = ["SIMPLE = T"] - for w in wcs_list: - nx, ny = w.pixel_shape - header = w.to_header() - header["NAXIS"] = 2 - header["NAXIS1"] = nx - header["NAXIS2"] = ny - blocks.append("\n".join(str(card) for card in header.cards)) - return "".join(f"{block}\nEND \n" for block in blocks) +def _plain_header(w): + """A plain image-HDU header: ``NAXIS1``/``NAXIS2`` are the image.""" + nx, ny = w.pixel_shape + header = w.to_header() + header["NAXIS"] = 2 + header["NAXIS1"] = nx + header["NAXIS2"] = ny + return header -def _compressed_header_text(w): - """Render a one-CCD header mimicking an fpack tile-compressed HDU. +def _compressed_header(w): + """A header shaped like an fpack tile-compressed HDU. ``NAXIS1``/``NAXIS2`` describe the compressed binary table (byte width, row - count); the true image dimensions live in ``ZNAXIS1``/``ZNAXIS2``. This is - the shape of headers fetched from 'p.fits.fz' with ``head=True``. + count); the true image dimensions live in ``ZNAXIS1``/``ZNAXIS2``. """ nx, ny = w.pixel_shape - wcs_cards = "\n".join(str(card) for card in w.to_header().cards) - ccd = ( - "XTENSION= 'BINTABLE'\n" - "BITPIX = 8\n" - "NAXIS = 2\n" - "NAXIS1 = 8\n" - f"NAXIS2 = {ny}\n" - "PCOUNT = 1000000\n" - "GCOUNT = 1\n" - "TFIELDS = 1\n" - "ZIMAGE = T\n" - "ZBITPIX = -32\n" - "ZNAXIS = 2\n" - f"ZNAXIS1 = {nx}\n" - f"ZNAXIS2 = {ny}\n" - f"{wcs_cards}" - ) - return f"SIMPLE = T\nEND \n{ccd}\nEND \n" - - -# --------------------------------------------------------------------------- -# _expnum_from_path -# --------------------------------------------------------------------------- - -@pytest.mark.parametrize( - "path, expected", - [ - ("1234567.txt", 1234567), - ("/a/b/2143523.txt", 2143523), - ("headers/0000042.txt", 42), - ], -) -def test_expnum_from_path_extracts_trailing_number(path, expected): - """The trailing ``.txt`` is parsed as the exposure number.""" - assert _expnum_from_path(path) == expected - - -@pytest.mark.parametrize("path", ["no_number.txt", "1234567.fits", "abc.txt"]) -def test_expnum_from_path_raises_without_number(path): - """A filename without a trailing numeric stem raises ``ValueError``.""" - with pytest.raises(ValueError): - _expnum_from_path(path) - - -@given(st.integers(min_value=0, max_value=99999999)) -def test_expnum_from_path_roundtrips(expnum): - """Any exposure number round-trips through the filename convention.""" - assert _expnum_from_path(f"vos_headers/{expnum}.txt") == expnum - - -# --------------------------------------------------------------------------- -# HeaderDownloader.get_exposures -# --------------------------------------------------------------------------- - -def test_get_exposures_reduces_to_unique_exposures(tmp_path): - """A ``-`` CCD list collapses to its unique exposure numbers.""" - ccd_list = tmp_path / "ccds.txt" - ccd_list.write_text("2143523-0\n2143523-5\n2143524-3\n2143524-8\n") - - exps = HeaderDownloader().get_exposures(str(ccd_list)) - - npt.assert_array_equal(exps, np.array([2143523, 2143524])) - - -def test_get_exposures_single_line(tmp_path): - """A one-line CCD list (0-d loadtxt array) still yields one exposure.""" - ccd_list = tmp_path / "ccds.txt" - ccd_list.write_text("2143523-0\n") - - exps = HeaderDownloader().get_exposures(str(ccd_list)) - - npt.assert_array_equal(exps, np.array([2143523])) - - -def test_get_exposures_csv_matches_txt(tmp_path): - """The CSV and text code paths yield the same unique exposures.""" - txt = tmp_path / "ccds.txt" - txt.write_text("2143523-0\n2143523-5\n2143524-3\n") - csv = tmp_path / "ccds.csv" - csv.write_text("CCD\n2143523-0\n2143523-5\n2143524-3\n") - - dl = HeaderDownloader() - - npt.assert_array_equal( - dl.get_exposures(str(txt)), dl.get_exposures(str(csv)) - ) - - -# --------------------------------------------------------------------------- -# HeaderDownloader.get_fits_header — atomic rename -# --------------------------------------------------------------------------- - -def test_get_fits_header_writes_atomically(tmp_path): - """A successful download copies to ``.part`` then renames to the dest.""" - dl = HeaderDownloader() - dl._params["output_dir"] = str(tmp_path) - dl._params["overwrite"] = False - dl._params["dir_for_links"] = None - - dest = tmp_path / "42.txt" - tmp_dest = tmp_path / "42.txt.part" - - def fake_copy(source, target, head=True): - # The copy must land on the temp path, not the final destination. - assert target == str(tmp_dest) - with open(target, "w") as f: - f.write("HEADER") - - client = SimpleNamespace(copy=fake_copy) - - assert dl.get_fits_header(42, client) is True - assert dest.exists() - assert not tmp_dest.exists() - assert dest.read_text() == "HEADER" - - -def test_get_fits_header_failed_copy_leaves_no_dest(tmp_path): - """A failed download leaves no destination file (only, if any, ``.part``).""" - dl = HeaderDownloader() - dl._params["output_dir"] = str(tmp_path) - dl._params["overwrite"] = False - dl._params["dir_for_links"] = None - - def failing_copy(source, target, head=True): - raise RuntimeError("transfer interrupted") - - client = SimpleNamespace(copy=failing_copy) - - assert dl.get_fits_header(42, client) is False - assert not (tmp_path / "42.txt").exists() - assert not (tmp_path / "42.txt.part").exists() - - -# --------------------------------------------------------------------------- -# CcdPsfHandler.get_ccds_with_psf — missing-CCD subtraction -# --------------------------------------------------------------------------- - -def test_get_ccds_with_psf_subtracts_missing(monkeypatch): - """Valid CCDs are all exposure single-HDUs minus the missing set. - - The real ``summary.get_all_shdus`` is used so the cross-component - ``-`` ID format is pinned end to end. - """ - handler = CcdPsfHandler() - - # Two exposures, 3 CCDs each -> 6 candidate CCDs; two are missing. - monkeypatch.setattr(handler, "get_exp", lambda patches: {"100", "200"}) - monkeypatch.setattr( - handler, - "get_exp_shdu_missing", - lambda patches: {"100-1", "200-2"}, - ) - - result = handler.get_ccds_with_psf(["P1"], n_CCD=3) - - # get_all_shdus yields "-" for ccd in range(n_CCD). - assert result == {"100-0", "100-2", "200-0", "200-1"} - # Guard the assumption that the missing IDs share the produced format. - assert set(summary.get_all_shdus({"100"}, 3)) == {"100-0", "100-1", "100-2"} - - -@pytest.mark.parametrize( - ("version", "n_patch"), - [("v1.3", 7), ("v1.4", 7), ("v1.5", 8), ("v1.6", 9)], -) -def test_version_to_patch_count(version, n_patch): - """Each v1.x catalogue version maps to its patch count.""" - handler = CcdPsfHandler() - handler._params["version_cat"] = version - handler.update_params() - assert handler._params["n_patch"] == n_patch - assert len(handler._params["patches"]) == n_patch - - -def test_invalid_version_raises(): - """An unknown catalogue version fails loudly.""" - handler = CcdPsfHandler() - handler._params["version_cat"] = "v9.9" - with pytest.raises(ValueError, match="v9.9"): - handler.update_params() + header = w.to_header() + header["NAXIS"] = 2 + header["NAXIS1"] = 8 + header["NAXIS2"] = ny + header["ZIMAGE"] = True + header["ZNAXIS"] = 2 + header["ZNAXIS1"] = nx + header["ZNAXIS2"] = ny + return header # --------------------------------------------------------------------------- # image-shape resolution (fpack ZNAXIS vs plain NAXIS) # --------------------------------------------------------------------------- -def test_image_shape_prefers_znaxis_for_compressed_header(tmp_path): +def test_image_shape_prefers_znaxis_for_compressed_header(): """A compressed HDU reports ZNAXIS dims, not the binary-table NAXIS.""" - w = _tan_wcs(100.0, 20.0, nx=2080, ny=4612) - path = tmp_path / "1234567.txt" - path.write_text(_compressed_header_text(w)) - - (parsed_w, shape), = _parse_header_to_wcs(str(path)) + header = _compressed_header(_tan_wcs(100.0, 20.0, nx=2080, ny=4612)) # WCS pixel_shape would wrongly report the compressed byte width (8). - assert parsed_w.pixel_shape == (8, 4612) + assert wcs.WCS(header).pixel_shape == (8, 4612) # _image_shape recovers the true image dimensions. - assert shape == (2080, 4612) + assert _image_shape(header) == (2080, 4612) -def test_image_shape_falls_back_to_naxis(tmp_path): - """A plain image HDU (no ZIMAGE) uses NAXIS1/NAXIS2.""" - w = _tan_wcs(100.0, 20.0, nx=2080, ny=4612) - path = tmp_path / "1234567.txt" - path.write_text(_header_text([w])) +def test_image_shape_falls_back_to_naxis(): + """A plain image HDU (no ZIMAGE) uses NAXIS1/NAXIS2. - (_, shape), = _parse_header_to_wcs(str(path)) + This is the live path for the workflow: ``headers-.npy`` carries the + decompressed header astropy hands back for a tile-compressed HDU. + """ + header = _plain_header(_tan_wcs(100.0, 20.0, nx=2080, ny=4612)) - assert shape == (2080, 4612) + assert _image_shape(header) == (2080, 4612) def test_image_shape_raises_without_dimensions(): """A header with no image dimensions raises a clear ``ValueError``.""" - from astropy.io.fits import Header - header = Header() header["CTYPE1"] = "RA---TAN" @@ -298,18 +101,14 @@ def test_image_shape_raises_without_dimensions(): _image_shape(header) -def test_compressed_header_corners_are_full_width(tmp_path): +def test_compressed_header_corners_are_full_width(): """Corners from a compressed header span the true CCD width, not 8 px.""" - w = _tan_wcs(100.0, 20.0, nx=2080, ny=4612) - path = tmp_path / "1234567.txt" - path.write_text(_compressed_header_text(w)) + header = _compressed_header(_tan_wcs(100.0, 20.0, nx=2080, ny=4612)) - (parsed_w, shape), = _parse_header_to_wcs(str(path)) - ra, dec = _ccd_corners(parsed_w, shape) + ra, dec = _ccd_corners(wcs.WCS(header), _image_shape(header)) # RA extent must reflect ~2080 px * 1e-5 deg/px * cos(dec), not 8 px. - ra_extent = max(ra) - min(ra) - assert ra_extent > 0.01 + assert max(ra) - min(ra) > 0.01 # --------------------------------------------------------------------------- @@ -332,257 +131,56 @@ def test_ccd_corners_returns_four_corners_around_centre(): npt.assert_allclose(max(dec) - min(dec), ny * 1e-5, rtol=1e-3) -def test_parse_header_to_wcs_returns_one_wcs_per_extension(tmp_path): - """One (wcs, shape) pair is returned per CCD HDU; the primary is skipped.""" - ccd_wcs = [_tan_wcs(10.0), _tan_wcs(20.0), _tan_wcs(30.0)] - - path = tmp_path / "1234567.txt" - path.write_text(_header_text(ccd_wcs)) - - result = _parse_header_to_wcs(str(path)) - - assert len(result) == len(ccd_wcs) - assert all(shape == (2080, 4612) for _, shape in result) - - -# --------------------------------------------------------------------------- -# FieldCornersExtractor.process_single_header — per-CCD rows and filtering -# --------------------------------------------------------------------------- - -def test_process_single_header_emits_one_row_per_ccd(tmp_path): - """Without a CCD list, every HDU yields a ``-`` row.""" - ccd_wcs = [_tan_wcs(10.0), _tan_wcs(20.0), _tan_wcs(30.0)] - path = tmp_path / "1234567.txt" - path.write_text(_header_text(ccd_wcs)) - - rows = FieldCornersExtractor.process_single_header( - (str(path), False, None) - ) - - assert [r[0] for r in rows] == [ - "1234567-0", - "1234567-1", - "1234567-2", - ] - assert all(len(r[1]) == 4 and len(r[2]) == 4 for r in rows) - - -def test_process_single_header_filters_to_ccd_list(tmp_path): - """A ``valid_ccds`` set keeps only the listed CCDs of the exposure.""" - ccd_wcs = [_tan_wcs(10.0), _tan_wcs(20.0), _tan_wcs(30.0)] - path = tmp_path / "1234567.txt" - path.write_text(_header_text(ccd_wcs)) - - rows = FieldCornersExtractor.process_single_header( - (str(path), False, {"1234567-0", "1234567-2"}) - ) - - assert [r[0] for r in rows] == ["1234567-0", "1234567-2"] - - -def test_load_ccd_list_reads_ids(tmp_path): - """The CCD list loader returns the stripped, non-blank IDs as a set.""" - path = tmp_path / "ccds.txt" - path.write_text("100-0\n100-1\n\n200-5\n") - - assert FieldCornersExtractor.load_ccd_list(str(path)) == { - "100-0", - "100-1", - "200-5", - } - - -def test_run_extract_writes_per_ccd_rows(tmp_path): - """End to end: run() writes one per-CCD row filtered by the CCD list.""" - header_dir = tmp_path / "headers" - header_dir.mkdir() - ccd_wcs = [_tan_wcs(10.0), _tan_wcs(20.0), _tan_wcs(30.0)] - (header_dir / "1234567.txt").write_text(_header_text(ccd_wcs)) - - ccd_list = tmp_path / "ccds.txt" - ccd_list.write_text("1234567-0\n1234567-2\n") - - out = tmp_path / "corners.txt" - - extractor = FieldCornersExtractor() - extractor.run( - args=[ - "-i", str(header_dir), - "-l", str(ccd_list), - "-o", str(out), - ] - ) - - lines = out.read_text().splitlines() - assert len(lines) == 2 - ids = [line.split()[0] for line in lines] - assert ids == ["1234567-0", "1234567-2"] - # Each row: 1 ID + 4 RA + 4 Dec = 9 columns. - assert all(len(line.split()) == 9 for line in lines) - - -# --------------------------------------------------------------------------- -# FieldCornersExtractor resume path (per-CCD done-set) -# --------------------------------------------------------------------------- - -def _write_headers(header_dir, expnums, n_ccd=3): - """Write one plain multi-HDU header per exposure into ``header_dir``.""" - header_dir.mkdir(exist_ok=True) - for j, expnum in enumerate(expnums): - ccd_wcs = [_tan_wcs(10.0 + j + i) for i in range(n_ccd)] - (header_dir / f"{expnum}.txt").write_text(_header_text(ccd_wcs)) - - -def test_get_done_ccds_reads_present_ids(tmp_path): - """The done-set is the exact set of CCD IDs already in the output.""" - out = tmp_path / "corners.txt" - out.write_text( - "1234567-0 10 10 10 10 20 20 20 20\n" - "1234567-2 10 10 10 10 20 20 20 20\n" - ) - - extractor = FieldCornersExtractor() - extractor._params["output_file"] = str(out) - - assert extractor.get_done_ccds() == {"1234567-0", "1234567-2"} - - -def test_resume_adds_new_exposure_without_duplicating(tmp_path): - """Resume appends a new exposure and leaves existing rows untouched.""" - header_dir = tmp_path / "headers" - _write_headers(header_dir, [1000001, 1000002]) - - out = tmp_path / "corners.txt" - # Pre-populate with exposure 1's three CCD rows. - pre = FieldCornersExtractor().process_single_header( - (str(header_dir / "1000001.txt"), False, None) - ) - with open(out, "w") as f: - for ccd_id, ra, dec in pre: - f.write(f"{ccd_id} " + " ".join(f"{v:.5f}" for v in ra + dec) + "\n") - - FieldCornersExtractor().run( - args=["-i", str(header_dir), "-o", str(out), "-r"] - ) - - ids = [line.split()[0] for line in out.read_text().splitlines()] - # No duplicate exposure-1 rows; exposure 2's three CCDs added. - assert ids.count("1000001-0") == 1 - assert sorted(ids) == [ - "1000001-0", "1000001-1", "1000001-2", - "1000002-0", "1000002-1", "1000002-2", - ] - - -def test_resume_completes_partial_exposure(tmp_path): - """An exposure interrupted mid-write is completed, not skipped.""" - header_dir = tmp_path / "headers" - _write_headers(header_dir, [1000001]) - - out = tmp_path / "corners.txt" - # Simulate an interrupt: only the first of exposure 1's CCDs was written. - pre = FieldCornersExtractor().process_single_header( - (str(header_dir / "1000001.txt"), False, None) - ) - ccd_id, ra, dec = pre[0] - with open(out, "w") as f: - f.write(f"{ccd_id} " + " ".join(f"{v:.5f}" for v in ra + dec) + "\n") - - FieldCornersExtractor().run( - args=["-i", str(header_dir), "-o", str(out), "-r"] - ) - - ids = [line.split()[0] for line in out.read_text().splitlines()] - # The missing CCDs are filled in; the present one is not duplicated. - assert sorted(ids) == ["1000001-0", "1000001-1", "1000001-2"] - assert ids.count("1000001-0") == 1 - - # --------------------------------------------------------------------------- -# CoverageMapBuilder.check_params (nside validation) +# nside validation # --------------------------------------------------------------------------- -def test_check_params_accepts_power_of_two_nside(tmp_path): +def test_check_nside_accepts_powers_of_two(): """Powers of two for both nside values pass validation.""" - infile = tmp_path / "corners.txt" - infile.write_text("100-0 0.0 1.0 1.0 0.0 0.0 0.0 1.0 1.0\n") - - builder = CoverageMapBuilder() - builder._params["input_file"] = str(infile) - builder._params["nside_coverage"] = 32 - builder._params["nside"] = 2048 - - builder.check_params() + check_nside(32, 2048) @pytest.mark.parametrize( - "nside_coverage, nside", [(33, 2048), (32, 100), (32, 3000)] + "nside_coverage, nside", [(33, 2048), (32, 100), (32, 3000), (0, 2048)] ) -def test_check_params_rejects_non_power_of_two_nside( - tmp_path, nside_coverage, nside -): +def test_check_nside_rejects_non_power_of_two(nside_coverage, nside): """A non-power-of-two nside raises ``ValueError``.""" - infile = tmp_path / "corners.txt" - infile.write_text("100-0 0.0 1.0 1.0 0.0 0.0 0.0 1.0 1.0\n") - - builder = CoverageMapBuilder() - builder._params["input_file"] = str(infile) - builder._params["nside_coverage"] = nside_coverage - builder._params["nside"] = nside - with pytest.raises(ValueError): - builder.check_params() - + check_nside(nside_coverage, nside) -def test_check_params_missing_input_file_raises(tmp_path): - """A missing input file raises ``FileNotFoundError``.""" - builder = CoverageMapBuilder() - builder._params["input_file"] = str(tmp_path / "does_not_exist.txt") - with pytest.raises(FileNotFoundError): - builder.check_params() +def test_build_map_validates_nside_before_stamping(): + """``build_map`` refuses a bad nside before it reaches healsparse.""" + with pytest.raises(ValueError, match="nside"): + build_map(["100-0"], [0.0, 0.1, 0.1, 0.0], [0.0, 0.0, 0.1, 0.1], + 32, 3000) # --------------------------------------------------------------------------- -# CoverageMapBuilder — parsing, nexp contract, RA-wrap and pole guards +# build_map — nexp contract, RA-wrap and pole guards # --------------------------------------------------------------------------- -def _read_map(path): - import healsparse as hsp - - return hsp.HealSparseMap.read(str(path)) - - -def test_build_map_single_row(tmp_path): - """A one-row corners file exercises the atleast_1d/2d parsing guards.""" - infile = tmp_path / "corners.txt" - infile.write_text("100-0 9.9 10.1 10.1 9.9 0.0 0.0 0.2 0.2\n") - out = tmp_path / "cov.hsp" - - CoverageMapBuilder().run( - args=["-i", str(infile), "-o", str(out), "-c", "32", "-n", "1024"] +def test_build_map_single_footprint(): + """One footprint given as flat corner lists exercises the shape guards.""" + m = build_map( + ["100-0"], [9.9, 10.1, 10.1, 9.9], [0.0, 0.0, 0.2, 0.2], 32, 1024 ) - m = _read_map(out) assert 0 < len(m.valid_pixels) < 200 assert m[m.valid_pixels].max() == 1 -def test_build_map_nexp_counts_overlapping_exposures(tmp_path): - """Two overlapping CCDs from different exposures give value 2 in overlap.""" +def test_build_map_nexp_counts_overlapping_exposures(): + """Two overlapping CCDs from two exposures give value 2 in the overlap.""" # Two 0.4x0.4 deg CCDs offset by 0.2 deg in RA -> a central overlap strip. - infile = tmp_path / "corners.txt" - infile.write_text( - "100-0 9.8 10.2 10.2 9.8 19.8 19.8 20.2 20.2\n" - "200-0 10.0 10.4 10.4 10.0 19.8 19.8 20.2 20.2\n" + m = build_map( + ["100-0", "200-0"], + [[9.8, 10.2, 10.2, 9.8], [10.0, 10.4, 10.4, 10.0]], + [[19.8, 19.8, 20.2, 20.2], [19.8, 19.8, 20.2, 20.2]], + 32, + 1024, ) - out = tmp_path / "cov.hsp" - CoverageMapBuilder().run( - args=["-i", str(infile), "-o", str(out), "-c", "32", "-n", "1024"] - ) - - m = _read_map(out) values = m[m.valid_pixels] # The overlap is covered by both exposures (value 2); the union edges by # one (value 1). Both must be present; nothing exceeds 2. @@ -606,30 +204,19 @@ def test_unwrap_ra_leaves_normal_polygon_unchanged(): ) -def test_build_map_invariant_under_ra_shift(tmp_path): +def test_build_map_invariant_under_ra_shift(): """A seam CCD's footprint matches an identical CCD shifted +10 deg in RA. - Rotating both the seam CCD (via +360/unwrap) and a reference CCD onto the - same RA and comparing pixel counts fails if the seam polygon were filling + Comparing the seam polygon's pixel count against a reference polygon of the + same size at the same declination fails if the seam polygon were filling the ~360 deg complement. This is the real RA-wrap regression guard. """ - # Seam CCD straddling RA=0, and the same CCD translated to RA~10. - seam = tmp_path / "seam.txt" - seam.write_text("100-0 359.9 0.1 0.1 359.9 20.0 20.0 20.2 20.2\n") - ref = tmp_path / "ref.txt" - ref.write_text("200-0 9.9 10.1 10.1 9.9 20.0 20.0 20.2 20.2\n") - - m_seam = tmp_path / "seam.hsp" - m_ref = tmp_path / "ref.hsp" - CoverageMapBuilder().run( - args=["-i", str(seam), "-o", str(m_seam), "-c", "32", "-n", "1024"] - ) - CoverageMapBuilder().run( - args=["-i", str(ref), "-o", str(m_ref), "-c", "32", "-n", "1024"] - ) + dec = [20.0, 20.0, 20.2, 20.2] + m_seam = build_map(["100-0"], [359.9, 0.1, 0.1, 359.9], dec, 32, 1024) + m_ref = build_map(["200-0"], [9.9, 10.1, 10.1, 9.9], dec, 32, 1024) - n_seam = len(_read_map(m_seam).valid_pixels) - n_ref = len(_read_map(m_ref).valid_pixels) + n_seam = len(m_seam.valid_pixels) + n_ref = len(m_ref.valid_pixels) # Same-size footprints at the same declination: pixel counts agree to # within a few boundary pixels, and are nowhere near a hemisphere. @@ -637,17 +224,14 @@ def test_build_map_invariant_under_ra_shift(tmp_path): assert 0 < n_seam < 200 -def test_build_map_pole_guard_skips_polygon(tmp_path, capsys): +def test_build_map_pole_guard_skips_polygon(capsys): """A polygon with |dec| near 90 deg is skipped with a warning.""" - infile = tmp_path / "corners.txt" - infile.write_text( - "100-0 10.0 10.2 10.2 10.0 89.5 89.5 89.7 89.7\n" - "200-0 10.0 10.2 10.2 10.0 20.0 20.0 20.2 20.2\n" - ) - out = tmp_path / "cov.hsp" - - CoverageMapBuilder().run( - args=["-i", str(infile), "-o", str(out), "-c", "32", "-n", "1024"] + build_map( + ["100-0", "200-0"], + [[10.0, 10.2, 10.2, 10.0], [10.0, 10.2, 10.2, 10.0]], + [[89.5, 89.5, 89.7, 89.7], [20.0, 20.0, 20.2, 20.2]], + 32, + 1024, ) captured = capsys.readouterr() @@ -656,19 +240,17 @@ def test_build_map_pole_guard_skips_polygon(tmp_path, capsys): # --------------------------------------------------------------------------- -# console-runner argv handling (regression guard for the -h entry points) +# median smoothing (offline; the coverage_map rule writes the raw nexp map) # --------------------------------------------------------------------------- -def test_run_help_flag_exits_cleanly(monkeypatch): - """``run()`` with no args reads ``sys.argv[1:]`` so ``-h`` exits 0. - - Guards the regression where the runners parsed the full ``sys.argv`` - (including ``argv[0]``), which made ``-h`` collide with the integer - ``-c`` option and exit non-zero. - """ - monkeypatch.setattr(sys, "argv", ["extract_field_corners", "-h"]) +def test_median_filter_fills_a_lone_hole(): + """A lone low pixel in a covered disc is pulled up to its neighbours.""" + nside = 1024 + m = hsp.HealSparseMap.make_empty(32, nside, np.uint16) + m[hpg.query_circle(nside, 10.0, 20.0, 0.1)] = 2 - with pytest.raises(SystemExit) as excinfo: - FieldCornersExtractor().run() + centre = hpg.angle_to_pixel(nside, 10.0, 20.0) + m[[centre]] = 1 + assert m[centre] == 1 - assert excinfo.value.code == 0 + assert median_filter(m)[centre] == 2 diff --git a/tests/module/test_coverage_map.py b/tests/module/test_coverage_map.py new file mode 100644 index 000000000..7ba895066 --- /dev/null +++ b/tests/module/test_coverage_map.py @@ -0,0 +1,158 @@ +"""The workflow's coverage map: exposure footprint records -> a HealSparse nexp map. + +``tests/module/test_coverage.py`` pins the nexp contract through the +``exp_ra_dec.txt`` CLI path. This module pins the SAME contract through the path +the Snakemake workflow actually takes — one ``exp_footprint.json`` per exposure +on the products root, globbed and fed to ``build_map`` as arrays — because that +path has two properties the text one does not and neither is visible in the map: + + * it is CAMPAIGN-CUMULATIVE by construction. The script globs every record + under ``/exp/*/*/manifests/``, sharded, including exposures + whose scratch stores were reclaimed. A regression that fed it only the + declared rule inputs would build a map of one batch and look fine. + * the records are RAW SKY. Unwrapping across RA=0 happens once, inside + ``build_map``; a record written pre-unwrapped, or unwrapped twice, silently + moves a footprint by 360 degrees. + +Two exposures, one CCD each, offset so they overlap — the same fixture geometry +as the CLI test, so the two routes are comparable by eye. +""" + +import importlib.util +import json +import sys +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] +SCRIPT = REPO_ROOT / "workflow" / "scripts" / "coverage_map.py" + + +def _load(): + """Import the script by path — ``workflow/scripts`` is not a package.""" + assert SCRIPT.exists(), f"{SCRIPT} not found; the rule calls it by path" + spec = importlib.util.spec_from_file_location("_coverage_map", SCRIPT) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +coverage_map = _load() + + +def write_footprint(products, exp, ccds): + """One exposure's record, at the sharded path exp_footprint writes to.""" + path = (products / "exp" / exp[:2] / exp / "manifests" + / "exp_footprint.json") + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps({ + "stage": "exp_footprint", "level": "exp", "unit": exp, + "status": "complete", + "n_ccd_headers": len(ccds), "n_valid_psf": len(ccds), + "ccds": [{"id": f"{exp}-{i}", "ra": ra, "dec": dec} + for i, (ra, dec) in enumerate(ccds)], + "ccds_no_psf": [], + }, indent=2, sort_keys=True)) + return path + + +def box(ra_min, ra_max, dec_min, dec_max): + """A rectangular CCD footprint, corners counterclockwise from bottom-left.""" + return ([ra_min, ra_max, ra_max, ra_min], + [dec_min, dec_min, dec_max, dec_max]) + + +def run(products, tmp_path, monkeypatch, nside="1024"): + out = tmp_path / "coverage" / "coverage.hsp" + manifest = tmp_path / "coverage" / "manifests" / "coverage_map.json" + monkeypatch.setattr(sys, "argv", [ + "coverage_map.py", + "--products-dir", str(products), + "--out", str(out), "--manifest", str(manifest), + "--nside-coverage", "32", "--nside", nside]) + coverage_map.main() + import healsparse as hsp + return hsp.HealSparseMap.read(str(out)), json.loads(manifest.read_text()) + + +def test_two_exposures_accumulate_to_nexp(tmp_path, monkeypatch): + """Overlapping CCDs from two exposures give 2 in the overlap, 1 outside. + + The exposures live in DIFFERENT shard directories, which is the campaign + layout: a glob that assumed one shard would find only one of them. + """ + products = tmp_path / "products" + write_footprint(products, "1000001", [box(9.8, 10.2, 19.8, 20.2)]) + write_footprint(products, "2000001", [box(10.0, 10.4, 19.8, 20.2)]) + + m, manifest = run(products, tmp_path, monkeypatch) + + values = m[m.valid_pixels] + assert values.max() == 2 + assert values.min() == 1 + assert (values == 2).sum() > 0 + assert (values == 1).sum() > 0 + + assert manifest["n_exposures"] == 2 + assert manifest["exposures"] == ["1000001", "2000001"] + assert manifest["n_ccds"] == 2 + assert manifest["nside_coverage"] == 32 + + +def test_every_record_on_the_root_is_used(tmp_path, monkeypatch): + """The map is campaign-cumulative: nothing selects a subset of the records. + + A third exposure appears on the products root with no involvement from any + caller — the state a reclaimed or out-of-scope exposure is in — and it must + still be in the map. + """ + products = tmp_path / "products" + write_footprint(products, "1000001", [box(9.8, 10.2, 19.8, 20.2)]) + write_footprint(products, "2000001", [box(10.0, 10.4, 19.8, 20.2)]) + write_footprint(products, "3000001", [box(40.0, 40.4, 19.8, 20.2)]) + + m, manifest = run(products, tmp_path, monkeypatch) + + assert manifest["exposures"] == ["1000001", "2000001", "3000001"] + # The far-away exposure is real sky in the map, not just a row in the record. + assert m.get_values_pos(40.2, 20.0, lonlat=True) == 1 + + +def test_seam_record_is_unwrapped_by_the_builder(tmp_path, monkeypatch): + """A raw-sky record across RA=0 lands where it belongs, not 360 deg away.""" + products = tmp_path / "products" + write_footprint(products, "1000001", + [([359.8, 0.2, 0.2, 359.8], [19.8, 19.8, 20.2, 20.2])]) + + m, _ = run(products, tmp_path, monkeypatch) + + assert m.get_values_pos(0.0, 20.0, lonlat=True) == 1 + assert m.get_values_pos(359.9, 20.0, lonlat=True) == 1 + # Nothing was stamped on the far side of the sky. + assert m.get_values_pos(180.0, 20.0, lonlat=True) == 0 + + +def test_no_records_is_a_loud_failure(tmp_path, monkeypatch): + """An empty products root must not write an empty map and exit green.""" + products = tmp_path / "products" + products.mkdir() + with pytest.raises(SystemExit) as exc: + run(products, tmp_path, monkeypatch) + assert "nothing to build a map from" in str(exc.value) + + +def test_records_naming_no_ccd_is_a_loud_failure(tmp_path, monkeypatch): + """Records that name no CCD are the quieter empty map, and equally fatal. + + Every exposure on the root having lost every CCD is a broken PSF stage, not + a survey with no coverage — but the .hsp it would write is valid and + plausible, and its consumer would mask everything. + """ + products = tmp_path / "products" + write_footprint(products, "1000001", []) + write_footprint(products, "2000001", []) + + with pytest.raises(SystemExit) as exc: + run(products, tmp_path, monkeypatch) + assert "not one names a CCD with a PSF model" in str(exc.value) diff --git a/tests/unit/test_exp_footprint.py b/tests/unit/test_exp_footprint.py new file mode 100644 index 000000000..5410d4a27 --- /dev/null +++ b/tests/unit/test_exp_footprint.py @@ -0,0 +1,277 @@ +"""The exp_footprint record, and the CCD-index contract it rests on. + +``workflow/scripts/exp_footprint.py`` joins two records that name CCDs in two +different ways and never cross-check each other: + + * ``headers-.npy`` names a CCD by its POSITION in the array — the same + ``idx-1`` split_exp used for ``image--.fits``; + * ``exp_persist.json`` names a CCD inside a FILENAME, + ``validation_psf--.fits``, written by psfex_interp. + +The whole design depends on those being the same integer, and no code asserts +it: split_exp writes the image and the array element in one loop, psfex_interp +inherits the numbering through the file handler, and the coverage map would be +silently WRONG — right pixels, wrong exposure count — if they ever diverged by a +permutation. This module is where that contract is pinned. It inherits the role +of ``tests/module/test_coverage.py::test_get_all_shdus``, which pinned the same +``-`` format against the summary scrape, both now retired. + +The fixtures are real astropy WCSs, one per CCD with a DISTINCT centre, so a +permutation of the array shows up as corners on the wrong ``id`` rather than as +a passing test. One CCD straddles RA=0 deliberately: the record is raw sky, and +the seam is the map builder's business (``unwrap_ra``), not this script's. + +Deliberately not a module test: nothing here runs shapepipe, only its two +geometry helpers, so it belongs with the fast structural suite. +""" + +import importlib.util +import json +import sys +from pathlib import Path + +import numpy as np +import pytest +from astropy.io.fits import Header +from astropy.wcs import WCS + +REPO_ROOT = Path(__file__).resolve().parents[2] +SCRIPTS = REPO_ROOT / "workflow" / "scripts" +SCRIPT = SCRIPTS / "exp_footprint.py" + +EXP = "2605805" +# MegaCam-ish: 2048 x 4612 pixels at 0.187"/pixel, i.e. ~0.106 x 0.240 deg. +NX, NY = 2048, 4612 +PIXSCALE = 0.187 / 3600.0 + + +def _load(): + """Import the script by path — ``workflow/scripts`` is not a package. + + Its ``persist_exp`` import is a sibling it reaches through ``sys.path[0]``, + which is how the rule invokes it, so the directory goes on the path here too. + """ + assert SCRIPT.exists(), f"{SCRIPT} not found; the rule calls it by path" + sys.path.insert(0, str(SCRIPTS)) + try: + spec = importlib.util.spec_from_file_location("_exp_footprint", SCRIPT) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + finally: + sys.path.remove(str(SCRIPTS)) + return module + + +exp_footprint = _load() + + +def ccd_header(ra_centre, dec_centre): + """A single-CCD header + WCS, as split_exp stores them. + + The header carries plain ``NAXIS1/2`` and no ``ZIMAGE``, which is what + astropy hands back for a tile-compressed HDU once it is decompressed — the + case ``_image_shape`` takes its second branch for. + """ + w = WCS(naxis=2) + w.wcs.ctype = ["RA---TAN", "DEC--TAN"] + w.wcs.crpix = [NX / 2.0, NY / 2.0] + w.wcs.crval = [ra_centre, dec_centre] + w.wcs.cdelt = [-PIXSCALE, PIXSCALE] + header = w.to_header() + header["NAXIS"] = 2 + header["NAXIS1"] = NX + header["NAXIS2"] = NY + return {"WCS": w, "header": header.tostring()} + + +def headers_array(centres): + """``headers-.npy``'s content: object array, one entry per CCD.""" + arr = np.zeros(len(centres), dtype="O") + for i, (ra, dec) in enumerate(centres): + arr[i] = ccd_header(ra, dec) + return arr + + +# Six CCDs on a row, each 0.3 deg from the last so no two share a footprint, and +# CCD 3 sitting on the RA=0 seam. +CENTRES = [(10.0, 30.0), (10.3, 30.0), (10.6, 30.0), + (0.0, 30.0), (11.2, 30.0), (11.5, 30.0)] + + +@pytest.fixture +def store(tmp_path): + """A scratch exposure store with a headers npy, and a products root.""" + npy_dir = tmp_path / "exp" / EXP / exp_footprint.HEADERS_DIR + npy_dir.mkdir(parents=True) + np.save(npy_dir / f"headers-{EXP}.npy", headers_array(CENTRES)) + return tmp_path + + +def persist_manifest(path, ccds, patterns=("validation_psf-*.fits",), + label=True): + """An ``exp_persist`` manifest naming exactly ``ccds`` as PSF-bearing. + + Shaped as persist_exp.py writes it, including the decoy member: a keep list + of several patterns packs files this script must ignore, and reading a CCD + index out of one of them would be a real bug. + + ``label=False`` drops the ``product`` field, which is how a tar packed + before that field existed reads back — the case the name glob catches. + """ + files = [] + for c in ccds: + entry = {"name": f"validation_psf-{EXP}-{c}.fits", + "pattern": "validation_psf-*.fits", "bytes": 1} + if label: + entry["product"] = "psf_validation" + files.append(entry) + files.append({"name": f"{EXP}-0.psf", "product": "psf_model", + "pattern": "*.psf", "bytes": 1}) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps( + {"stage": "exp_persist", "unit": EXP, "status": "complete", + "patterns": list(patterns), "n_files": len(files), "files": files}, + indent=2, sort_keys=True)) + return path + + +def run(store, persist, out, monkeypatch): + """Invoke the script's CLI, as the rule's shell does.""" + monkeypatch.setattr(sys, "argv", [ + "exp_footprint.py", + "--exp-dir", str(store / "exp" / EXP), + "--exp", EXP, + "--persist-manifest", str(persist), + "--manifest", str(out)]) + exp_footprint.main() + return json.loads(out.read_text()) + + +def test_ccd_index_alignment(store, monkeypatch): + """npy index ``i`` == ``-`` == ``validation_psf--.fits``. + + The three CCDs with a PSF are non-adjacent on purpose: an off-by-one or a + "renumber the survivors 0..n" bug passes a contiguous fixture and fails + this one. + """ + persist = persist_manifest(store / "prod" / "exp_persist.json", [0, 2, 5]) + record = run(store, persist, store / "prod" / "exp_footprint.json", + monkeypatch) + + assert record["n_ccd_headers"] == len(CENTRES) + assert record["n_valid_psf"] == 3 + assert [c["id"] for c in record["ccds"]] == [ + f"{EXP}-0", f"{EXP}-2", f"{EXP}-5"] + assert record["ccds_no_psf"] == [f"{EXP}-1", f"{EXP}-3", f"{EXP}-4"] + + # The corners on `-i` are the corners of ARRAY ELEMENT i, not of the + # i-th surviving CCD: each fixture CCD has its own centre, so this catches + # any permutation. Checked against the same two helpers the script imports, + # which is the point — the contract under test is the INDEXING, not the + # projection arithmetic (tests/module/test_coverage.py owns that). + arr = headers_array(CENTRES) + for entry in record["ccds"]: + i = int(entry["id"].rsplit("-", 1)[1]) + shape = exp_footprint._image_shape(Header.fromstring(arr[i]["header"])) + ra, dec = exp_footprint._ccd_corners(arr[i]["WCS"], shape) + assert entry["ra"] == pytest.approx(list(ra)) + assert entry["dec"] == pytest.approx(list(dec)) + # ... and that centre really is CCD i's, not its neighbour's. + assert np.mean(entry["dec"]) == pytest.approx(CENTRES[i][1], abs=1e-3) + + +def test_record_is_raw_sky_across_the_ra_seam(store, monkeypatch): + """CCD 3 straddles RA=0 and the record says so, unwrapped by nobody. + + The seam is handled once, in ``coverage_map_builder.unwrap_ra``, at the + moment a polygon is stamped. Unwrapping here as well would put negative RA + into a durable record that other consumers read, and double-unwrapping is + not idempotent. + """ + persist = persist_manifest(store / "prod" / "exp_persist.json", [3]) + record = run(store, persist, store / "prod" / "exp_footprint.json", + monkeypatch) + + ra = record["ccds"][0]["ra"] + assert record["ccds"][0]["id"] == f"{EXP}-3" + assert min(ra) >= 0.0 and max(ra) < 360.0, "raw sky, not a shifted branch" + assert max(ra) - min(ra) > 180.0, "the fixture must actually cross RA=0" + # The four corners land on both sides of the seam, ~0.05 deg out. + assert sorted(round(r) for r in ra) == [0, 0, 360, 360] + + +def test_byte_stable_rerun_keeps_the_mtime(store, monkeypatch): + """A rerun over an unchanged store must not move the manifest's mtime. + + mtime is a rerun trigger, and this manifest is an input of clean_exposure: + an unconditional rewrite would make every downstream reclamation look out + of date once per invocation. Same contract as persist_exp.py's. + """ + persist = persist_manifest(store / "prod" / "exp_persist.json", [0, 2, 5]) + out = store / "prod" / "exp_footprint.json" + + first = run(store, persist, out, monkeypatch) + raw = out.read_bytes() + before = out.stat().st_mtime_ns + + second = run(store, persist, out, monkeypatch) + assert second == first + assert out.read_bytes() == raw + assert out.stat().st_mtime_ns == before + # No stray tmp left behind on the persistent root. + assert not list(out.parent.glob("*.tmp")) + + +def test_unlabelled_members_are_found_by_name(store, monkeypatch): + """A tar packed before persist_exp labelled its members still reads. + + Membership is the product label OR the file name, merge_star_cat's test + exactly. The label is what persist_exp writes today; the name glob is what + an older manifest, or a keep list written as a raw glob, leaves behind. + Neither may make an exposure silently contribute no sky. + """ + persist = persist_manifest(store / "prod" / "exp_persist.json", [0, 2, 5], + label=False) + record = run(store, persist, store / "prod" / "exp_footprint.json", + monkeypatch) + assert [c["id"] for c in record["ccds"]] == [f"{EXP}-{i}" + for i in (0, 2, 5)] + + +def test_the_keep_list_cannot_turn_the_psf_set_off(store, monkeypatch): + """`persist_exp:` is optional retention and no precondition of this rule. + + exp_persist packs every CCD's psf_validation whatever the keep list says + (its ALWAYS), so a manifest whose `patterns` name only other products still + carries the valid-PSF set, and this rule must read it rather than refuse. + """ + persist = persist_manifest(store / "prod" / "exp_persist.json", [0, 2, 5], + patterns=("*.psf",)) + record = run(store, persist, store / "prod" / "exp_footprint.json", + monkeypatch) + assert [c["id"] for c in record["ccds"]] == [f"{EXP}-{i}" + for i in (0, 2, 5)] + + +def test_psf_for_a_ccd_the_split_never_wrote_is_fatal(store, monkeypatch): + """The one disagreement the index alignment cannot absorb, made loud.""" + persist = persist_manifest(store / "prod" / "exp_persist.json", [0, 99]) + with pytest.raises(SystemExit) as exc: + run(store, persist, store / "prod" / "exp_footprint.json", monkeypatch) + assert "99" in str(exc.value) + + +def test_missing_headers_array_is_fatal(store, monkeypatch): + """A purged or unbuilt split store fails here, never against VOS. + + This is the other half of the rule declaring only its DURABLE input + (exposure.smk): a persist manifest outliving its scratch store is a real + state, and it must cost one error line rather than an exposure rebuild. + """ + npy = (store / "exp" / EXP / exp_footprint.HEADERS_DIR + / f"headers-{EXP}.npy") + npy.unlink() + persist = persist_manifest(store / "prod" / "exp_persist.json", [0]) + with pytest.raises(SystemExit) as exc: + run(store, persist, store / "prod" / "exp_footprint.json", monkeypatch) + assert "no WCS array" in str(exc.value) diff --git a/uv.lock b/uv.lock index eecd374c7..727601ad5 100644 --- a/uv.lock +++ b/uv.lock @@ -3305,7 +3305,6 @@ dependencies = [ { name = "python-dateutil" }, { name = "python-pysap" }, { name = "sf-tools" }, - { name = "skaha" }, { name = "skyproj" }, { name = "sqlitedict" }, { name = "termcolor" }, @@ -3400,7 +3399,6 @@ requires-dist = [ { name = "ruff", marker = "extra == 'lint'" }, { name = "sf-tools", specifier = ">=2.0.4" }, { name = "shapepipe", extras = ["doc", "jupyter", "lint", "plot", "release", "test", "fitsio"], marker = "extra == 'dev'" }, - { name = "skaha", specifier = ">=1.7" }, { name = "skyproj" }, { name = "skyproj", marker = "extra == 'plot'" }, { name = "snakemake", marker = "extra == 'jupyter'", specifier = ">=9.22.0" }, @@ -3433,24 +3431,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, ] -[[package]] -name = "skaha" -version = "1.7.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "defusedxml" }, - { name = "httpx" }, - { name = "pydantic" }, - { name = "rich" }, - { name = "toml" }, - { name = "typer" }, - { name = "vos" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/76/62/a733508ff88200c41459f5cdb72a797eb8795c833b4e9d51d31b5efe4073/skaha-1.7.0.tar.gz", hash = "sha256:ef9d69e7a4da8653cdbde3e62f18caef137130064ccb6fabed57c7c6f8bf8ef9", size = 54414, upload-time = "2025-05-28T21:17:03.631Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/d9/d3/a3c1d569ae714ed9985adccefdfe2d6dd9f8ad7a297bd8fa4c7ed30c587b/skaha-1.7.0-py3-none-any.whl", hash = "sha256:68b0d3c925b98bf145c5f695237474fe1cea07cf9b69b4a9bae9910375dfa01a", size = 40795, upload-time = "2025-05-28T21:17:02.296Z" }, -] - [[package]] name = "skyproj" version = "2.6.1" diff --git a/workflow/README.md b/workflow/README.md index c00d66c18..8a439ba6a 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -1,13 +1,25 @@ # ShapePipe Snakemake orchestration -Snakemake workflow that orchestrates real-data ShapePipe runs. It replaces the -`curl_canfar_local.sh → run_job_sp_canfar_v2.0.bash → job_sp_canfar_v2.0.bash` -bash layers and the per-site sbatch reimplementations. **Module code is -untouched**: rules call `shapepipe_run -c ` on the existing config -chains. Design and rationale: +Snakemake workflow that orchestrates real-data ShapePipe runs. It is the one +production orchestration. It replaced the bit-coded bash layers +(`run_job_sp_canfar_v2.0.bash → job_sp_canfar_v2.0.bash`) for real data, and +retired the CANFAR submission front end that drove them +(`curl_canfar_local.sh`, `canfar_submit_job`) and the per-site sbatch +reimplementations — those were last carried at `2ef07e45`. The two Gen-2 bash +scripts are still in the tree and still installed (`pyproject.toml`'s +`[tool.setuptools] script-files`): sp_validation's image-simulation workflow +calls `run_job_sp_canfar_v2.0.bash` by path, so they retire when that chain is +ported or parked, not with this one. **Module code is untouched**: rules call +`shapepipe_run -c ` on the existing config chains. Design and rationale: [CosmoStat/shapepipe#848](https://github.com/CosmoStat/shapepipe/issues/848) (the design document). +**There is no concept of a catalogue version in the code.** No path branches on +`v1.3`..`v1.6` or `v2.0`, and there are no sky patches: a campaign is a tile +list, and the version of a catalogue is the git tag of the code that produced +it. What that retirement removed, and where each piece was last carried, is in +the PR that made this sentence true. + Use `workflow/bin/sp` for everything. Bare `snakemake all` outside `sp` is unsupported: `sp` sets the state directory, the SLURM profile, and `SP_PHASE`, which the Snakefile needs to build the tile/exposure index at @@ -158,19 +170,24 @@ workflow/ bin/sp committed launcher (module load + /project venv + launch code snapshot + run/report/container/cancel) rules/ prepare.smk tile get_images/uncompress/find_exposures - exposure.smk per-exposure: get_images, split, psf, persist (no temp()); campaign star_cat_merge + exposure.smk per-exposure: get_images, split, psf, persist, footprint (no temp()); campaign star_cat_merge tile.smk per-tile: exp forest, merge_headers, detect, vignets, ngmix, merge, make_cat; campaign final_cat_merge + coverage.smk campaign-level: the HealSparse nexp mask from the exposure footprints scripts/ build_index.py prepare-phase run_index.sqlite builder (plain script) build_forest.py per-tile exposure symlink forest (group-compatible shell) - completeness.py the ported count table (shared by sp_rule + run_report) + completeness.py the ported count-floor table + the `check` every rule ends with run_report.py standalone report (NOT a DAG node; run_report hooks call it) container.py image layers + the resolution order behind `sp container` (stdlib-only) + ngmix_range.py the ngmix chunk partition: written once by tile_vignets, read by each chunk persist_exp.py ONE exposure's keepable PSF products -> one tar on products_dir (the exp_persist rule) + exp_footprint.py ONE exposure's per-CCD sky corners, for the CCDs with a PSF (the exp_footprint rule) + coverage_map.py every exposure footprint on products_dir -> coverage.hsp (the coverage_map rule) hdf5_reconcile.py bring an hdf5 catalogue into agreement with a campaign (shared by both merges) merge_star_cat.py ALL exposures' validation_psf, out of the tars -> full_starcat_.hdf5 merge_final_cat.py ALL tiles' final_cat -> final_cat_.hdf5 (the final_cat_merge rule) clean_exposure.py ONE exposure's store + manifests + logs -> tombstone (the clean_exposure rule) + clean_tile.py ONE finished tile's store -> tombstone (the clean_tile rule) profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; keep-going ``` @@ -192,10 +209,13 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee output natively and never touches its log, which is why the profile runs *without* `keep-incomplete`. `sp report` reads both dirs — the manifest for success, the log for failure — and a unit with neither ran nothing. -- **Completeness is an exact-count check, not a taxonomy.** After a run, - `sp_rule.py` counts products per mandatory runner against - `completeness.py`'s expected count and exits nonzero below it. A shortfall - below `expect` fails unless `warn` is set. No 3-class taxonomy, no +- **Completeness is a count floor, not a taxonomy.** There is no per-unit + wrapper script: the Snakefile's `unit_pre()` builds the unit's furniture and + clears the stage's run dir as bash inlined into the rule's `params`, and + `sp_shell()` composes every rule's shell as *prologue, one `shapepipe_run`, + one `completeness.py check`*. That check counts products per mandatory runner + against `completeness.py`'s floor and exits nonzero below it. Per-CCD + attrition between floor and `expect` is tolerated. No 3-class taxonomy, no error-signature whitelist. `--keep-going` isolates a failure to its own DAG cone. - **Stores are sharded.** Every tile/exposure runs its own `shapepipe_run` @@ -338,6 +358,30 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee an order of magnitude over Linux's 128 KiB `MAX_ARG_STRLEN` for a single argv entry, so each job is handed the tile list and the run index and derives the same set from them. +- **Coverage is a workflow product, built from records the DAG already + writes.** `exp_footprint` writes one JSON per exposure to + `/exp///manifests/exp_footprint.json`, giving the + four sky corners of every CCD that got a PSF model. It reads the valid-PSF CCD + set off `exp_persist.json`'s tar members — exact, because `psfex_interp` + returns *without* writing `validation_psf-*.fits` on NOT_ENOUGH_STARS, + BAD_CHI2 or FILE_NOT_FOUND — and the WCS off `headers-.npy`, written by + `exp_split`. It needs nothing of `persist_exp:`: those catalogues are the + `psf_validation` product `exp_persist` packs for every exposure whatever the + keep list says. Like `exp_persist` it runs whatever `clean:` and `coverage:` + say, because its input is on /scratch and the purge takes it; `clean_exposure` + takes its manifest as an input. Set + `coverage: {enabled: true}` and one further job, `coverage_map`, stamps every + footprint into `/coverage/coverage.hsp` — a HealSparse map + counting, per sky pixel, the exposures with a valid PSF there. That job is + **campaign-cumulative**: its declared inputs are the in-scope footprints, but + the script reads *every* record on the products root, reclaimed exposures + included, so appending tiles grows the map instead of replacing it. `nside` is + set in `config.yaml` to the production 128/131072 pair, ~0.1"/pixel, chosen to + align pixel-wise with the UNIONS bit masks; nothing defaults to it, and a + coarser map would look plausible and not align. Plotting stays out of the DAG: run `plot_coverage_map -i + /coverage/coverage.hsp ...` by hand, with the sky windows under + `coverage.plot` in `config.yaml`. sp_validation consumes the map in + `notebooks/demo_apply_hsp_masks.py`. - **A dead tile can be told to stop pinning exposures.** An exposure is cleanable only once every consuming tile has its vignets, so one permanently-failed tile holds its ~80 exposures for the life of the diff --git a/workflow/Snakefile b/workflow/Snakefile index 4317e871f..8e419deea 100644 --- a/workflow/Snakefile +++ b/workflow/Snakefile @@ -386,6 +386,7 @@ MERGE_FINAL_HASH = ":".join(( path_hash(Path(workflow.basedir).parent / "scripts" / "python" / "create_final_cat.py"), path_hash(CONFIG_DIR / "final_cat.param"))) +FOOTPRINT_HASH = script_hash("exp_footprint.py") # ngmix_range.py earns a hash for a stronger reason than the others. What it # emits is not a stale RESULT but a stale BOUNDARY, and a tile's eight chunks are # a PARTITION of its object IDs: resume a tile across an edit to the split and @@ -923,6 +924,38 @@ def final_cat_targets(): return [] return [final_cat_hdf5()] +# --- per-CCD sky footprints (the coverage map's raw material) --------------- +# `exp_footprint` reads the valid-PSF CCD set off exp_persist.json's members. It +# needs no precondition on `persist_exp:`: exp_persist packs every CCD's +# psf_validation catalogue whatever that list says (persist_exp.py's ALWAYS set), +# so the manifest always carries the valid-PSF set this rule selects on. The keep +# list is optional retention of OTHER products and cannot turn this off. + + +def footprint_targets(): + """Which exposures this invocation must record a per-CCD sky footprint for. + + REQUESTED BY `rule all` DIRECTLY, AND NOT GATED ON `coverage:`. The record is + derived from `headers-.npy`, which lives on /scratch and which the + 60-day purge takes whether or not this campaign ever builds a map — so + building it only when a map is asked for means that turning coverage on a + month later finds the WCS gone and no way back short of re-downloading the + exposures. This is persist_targets' argument above, applied to the other + scratch-only input coverage needs: milliseconds and a few KB now, against an + irrecoverable loss later. + + Scope, and the tombstone exclusion, are persist_targets' exactly: a cleaned + exposure's footprint was written before its store went (clean_exposure takes + this manifest as an input), and asking again would rebuild the chain from VOS. + + HEAD PROCESS ONLY, for the same reason as clean_targets() above. + """ + if not workflow.is_main_process: + return [] + exps = {e for t in TILES_READY for e in tile_exposures(t)} + return sorted(prod_exp_manifest(e, "exp_footprint") for e in exps + if not Path(tombstone(e)).exists()) + # --- tile reclamation (D5) -------------------------------------------------- # A separate flag from `clean:` (config.yaml carries the full # argument): exposure reclamation costs nothing but a rebuild if a tile is @@ -1082,6 +1115,7 @@ def sp_shell(stage, config_name, *, check_args="", post=""): include: "rules/prepare.smk" include: "rules/exposure.smk" include: "rules/tile.smk" +include: "rules/coverage.smk" # --- top-level targets ------------------------------------------------------ # The aggregation targets, clean_exposure and clean_tile run in the head @@ -1089,23 +1123,29 @@ include: "rules/tile.smk" # they would be ~20k (clean_exposure) or ~23k (clean_tile) sbatch submissions at # DR6 scale for work shorter than the scheduling latency. # -# Both are DAG LEAVES, so neither constrains a `group:` label. (A mid-chain -# localrule would: a local job cannot be fused into a submitted group. The old -# star-catalogue rules were exactly that, and they are gone with the internal -# mask generation.) +# Both clean rules are DAG LEAVES, so neither constrains a `group:` label. (A +# mid-chain localrule would: a local job cannot be fused into a submitted group. +# The old star-catalogue rules were exactly that, and they are gone with the +# internal mask generation.) # # exp_persist joins them for the same arithmetic — one tar of a few MB per # exposure, ~20k of them at DR6 scale, each far shorter than the scheduling -# latency that would submit it (exposure.smk argues the placement in full). It -# sits mid-chain between exp_psf and clean_exposure, but both of those are -# outside every group already (exp_psf is heavy, clean_exposure is local), so it -# adds no new grouping constraint. -localrules: all, prepare_all_tiles, clean_exposure, clean_tile, exp_persist +# latency that would submit it (exposure.smk argues the placement in full). +# exp_footprint sits beside it and is smaller still: one pickle load and ~160 +# pixel_to_world calls. Both sit mid-chain between exp_psf and clean_exposure, +# but both of those are outside every group already (exp_psf is heavy, +# clean_exposure is local), so they add no new grouping constraint. +# +# coverage_map is deliberately NOT one — it is a single submitted job that stamps +# ~1M polygons at nside=131072 (coverage.smk). +localrules: all, prepare_all_tiles, clean_exposure, clean_tile, exp_persist, exp_footprint rule all: input: [final_cat(t) for t in TILES_READY], persist_targets(), + footprint_targets(), + coverage_targets(), star_cat_targets(), final_cat_targets(), clean_targets(), diff --git a/workflow/config.yaml b/workflow/config.yaml index 38979b2d0..c2e5c3a3f 100644 --- a/workflow/config.yaml +++ b/workflow/config.yaml @@ -252,6 +252,53 @@ max_mem_mb: 750000 # per-tile, in-job, from the tile's own sexcat). ngmix_chunks: 8 +# COVERAGE MASK. `enabled: true` adds ONE campaign-level job, `coverage_map` +# (workflow/rules/coverage.smk), which builds a HealSparse nexp map from every +# exposure footprint on the products root. OFF by default because the map is a +# campaign-END product: a half-finished campaign's map is a picture of how far it +# has got, not of the survey. Turning it on costs one job and no rebuild — the +# per-exposure records it reads are written all along, by `exp_footprint`, +# whether or not this block is on: its input is on /scratch and the purge takes +# it. +# +# The map counts, per sky pixel, the number of exposures with a VALID PSF MODEL +# — only CCDs whose PSF model was actually written are stamped. In the v1.x +# chain that set was approximated by subtracting a summary-scrape file from all +# 40*N candidates; in the workflow it is exact, read off exp_persist.json's +# files[] (psfex_interp returns WITHOUT writing on NOT_ENOUGH_STARS / BAD_CHI2 / +# FILE_NOT_FOUND). Those catalogues are the `psf_validation` product, which +# exp_persist packs for every exposure whatever `persist_exp:` above says, so +# there is nothing to configure here. +# +# nside_coverage / nside: 128 / 131072 is the PRODUCTION pair, and NOTHING +# defaults to it — build_map takes both as required arguments, and this block is +# now the only place the pair is written down (the v1.x shell script that carried +# it is retired). nside=131072 is ~0.1"/pixel, CHOSEN TO MATCH THE UNIONS +# BIT-MASK RESOLUTION so that coverage and mask align pixel-wise. A coarser map +# looks entirely reasonable and does not align, and the consumer would not +# notice. +# +# Consumer: sp_validation applies these healsparse structural masks in +# notebooks/demo_apply_hsp_masks.py (it reads the coverage .hsp and writes the +# masked comprehensive catalogue). +# +# Plot windows are the two UNIONS sky regions, with the colourbar clipped to the +# 1-5 exposure range; plotting stays OUT of the DAG (a human act on a durable +# product), so these are defaults for `plot_coverage_map`, not rule params. +# +coverage: + enabled: false + nside_coverage: 128 + nside: 131072 + # Read by `plot_coverage_map`, not by any rule (plotting stays out of the DAG). + plot: + colorbar: true + n_exp_min: 1 + n_exp_max: 5 + regions: + SGC: {ra_min: -20, ra_max: 45, dec_min: 18, dec_max: 40} + NGC: {ra_min: 110, ra_max: 270, dec_min: 28, dec_max: 90} + # The container's installed shapepipe is overridden by the prod worktree via # --env PYTHONPATH in profiles/nibi (settled call 3); no config knob here. # build_index.py's missing-tile fraction threshold is passed by workflow/bin/sp diff --git a/workflow/config/cfis/config_exp_Gie.ini b/workflow/config/cfis/config_exp_Gie.ini index b0cce78e8..192b3a236 100644 --- a/workflow/config/cfis/config_exp_Gie.ini +++ b/workflow/config/cfis/config_exp_Gie.ini @@ -95,5 +95,3 @@ N_TRY = 3 # Retrieve command options, optional RETRIEVE_OPTIONS = --certfile=$HOME/.ssl/cadcproxy.pem - -#CHECK_EXISTING_DIR = $SP_RUN/output/run_sp_Gie_prev diff --git a/workflow/config/cfis/config_exp_Sp.ini b/workflow/config/cfis/config_exp_Sp.ini index dd27d6ccd..3b573cbc4 100644 --- a/workflow/config/cfis/config_exp_Sp.ini +++ b/workflow/config/cfis/config_exp_Sp.ini @@ -34,8 +34,8 @@ LOG_NAME = log_sp # Runner log file name, optional, default: shapepipe_runs RUN_LOG_NAME = log_run_sp -# NUMBER_LIST selects this unit; the workflow sets SP_UNIT_NUM to the -# dashed exposure ID (exp split is a "tile-scheme" stage per sp_rule.py). +# NUMBER_LIST selects this unit; the workflow sets SP_UNIT_NUM to the dashed +# exposure ID (the Snakefile's unit_num(), exported by unit_pre()). NUMBER_LIST = $SP_UNIT_NUM # Input directory, containing input files, single string or list of names with length matching FILE_PATTERN diff --git a/workflow/config/cfis/config_tile_PiViVi_mccd.ini b/workflow/config/cfis/config_tile_PiViVi_mccd.ini index d312ca494..ea8a1c8fb 100644 --- a/workflow/config/cfis/config_tile_PiViVi_mccd.ini +++ b/workflow/config/cfis/config_tile_PiViVi_mccd.ini @@ -171,8 +171,8 @@ STAMP_SIZE = 51 PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure -# run outputs. ME_IMAGE_EXP_DIR/ME_IMAGE_EXP_RUNNERS replace ME_IMAGE_DIR for -# the v2.0 per-exposure pipeline; output dirs are discovered by scanning $SP_EXP. +# run outputs. Output dirs are discovered by scanning $SP_EXP for the +# exposures listed in the exp_numbers input file. ME_IMAGE_EXP_DIR = $SP_EXP ME_IMAGE_EXP_RUNNERS = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner, sextractor_runner ME_IMAGE_PATTERN = flag, image, weight, background, background_rms diff --git a/workflow/config/cfis/config_tile_PiViVi_psfex.ini b/workflow/config/cfis/config_tile_PiViVi_psfex.ini index 27a759528..820843528 100644 --- a/workflow/config/cfis/config_tile_PiViVi_psfex.ini +++ b/workflow/config/cfis/config_tile_PiViVi_psfex.ini @@ -106,9 +106,9 @@ CHI2_THRESH = 2 # Multi-epoch mode parameters -# Root directory of per-exposure work directories; replaces ME_DOT_PSF_DIR -# for v2.0 per-exposure pipeline. psfex_runner/output/ dirs are discovered -# by scanning $SP_EXP for the exposures listed in the exp_numbers input file. +# Root directory of per-exposure work directories. psfex_runner/output/ +# dirs are discovered by scanning $SP_EXP for the exposures listed in the +# exp_numbers input file. ME_DOT_PSF_EXP_DIR = $SP_EXP # Input psf file pattern @@ -181,8 +181,8 @@ STAMP_SIZE = 51 PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure -# run outputs. ME_IMAGE_EXP_DIR/ME_IMAGE_EXP_RUNNERS replace ME_IMAGE_DIR for -# the v2.0 per-exposure pipeline; output dirs are discovered by scanning $SP_EXP. +# run outputs. Output dirs are discovered by scanning $SP_EXP for the +# exposures listed in the exp_numbers input file. ME_IMAGE_EXP_DIR = $SP_EXP ME_IMAGE_EXP_RUNNERS = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner, sextractor_runner ME_IMAGE_PATTERN = flag, image, weight, background, background_rms diff --git a/workflow/rules/coverage.smk b/workflow/rules/coverage.smk new file mode 100644 index 000000000..4e1e19f22 --- /dev/null +++ b/workflow/rules/coverage.smk @@ -0,0 +1,86 @@ +"""Coverage — the campaign's HealSparse nexp mask, from the exposure footprints. + + exp_footprint (per exposure, exposure.smk) -> coverage_map (per campaign) + +One rule, and it is the only campaign-level product besides the report. The +precedent it follows is ``star_catalogue`` (exposure.smk): a rule keyed by the +campaign rather than by a unit, whose inputs are the units' own records. + +CAMPAIGN-CUMULATIVE. The declared inputs are the IN-SCOPE, non-tombstoned +footprint manifests — ordering and rerun semantics for free, without dragging +out-of-scope tiles into the DAG through a new target. The SCRIPT then reads every +footprint record on the persistent root, tombstoned exposures included: their +records outlive their scratch stores and are still valid sky. So appending tiles +grows the map instead of replacing it, which is what a survey coverage mask +should do, and rebuilding is one job rather than a campaign. + +NOT A LOCALRULE, unlike every other rule that writes to the persistent root: at +DR6 scale this stamps ~1M polygons at nside=131072 in a Python loop +(coverage_map_builder.build_map). The resources below are a first sizing from +that count and not a measurement — the polygon loop is unmeasured above a few +thousand CCDs. + +PLOTS STAY OUT OF THE DAG. `plot_coverage_map -i /coverage/ +coverage.hsp ...`, by hand, with the windows in config.yaml's `coverage.plot` +block — the same argument that keeps run_report.py a standalone script: it is a +human act on a durable product. +""" + +# `coverage:` is OFF by default: the map is a campaign-end product, and a +# half-finished campaign's map is a picture of how far it has got rather than of +# the survey. flag() because `--config` delivers booleans as strings. +COVERAGE = config.get("coverage") or {} +COVERAGE_ENABLED = flag(COVERAGE.get("enabled", False)) +NSIDE_COVERAGE = int(COVERAGE.get("nside_coverage", 128)) +NSIDE = int(COVERAGE.get("nside", 131072)) + +COVERAGE_DIR = f"{PRODUCTS_DIR}/coverage" +COVERAGE_HSP = f"{COVERAGE_DIR}/coverage.hsp" +COVERAGE_MANIFEST = f"{COVERAGE_DIR}/manifests/coverage_map.json" +COVERAGE_HASH = script_hash("coverage_map.py") + +# PARSE-TIME GUARD, and it fails before submission rather than after: healsparse +# requires powers of two, and it says so only once a job has reached a node. +# +# There is no guard on `persist_exp:`. The valid-PSF CCD set exp_footprint reads +# comes from the psf_validation members exp_persist packs for every exposure +# whatever the keep list says, so the chain has no config precondition to check. +for _key, _n in (("nside_coverage", NSIDE_COVERAGE), ("nside", NSIDE)): + if COVERAGE_ENABLED and (_n <= 0 or _n & (_n - 1)): + raise WorkflowError(f"coverage.{_key} must be a power of 2, got {_n}") + + +def coverage_targets(): + """The campaign map, when `coverage:` asks for one. + + HEAD PROCESS ONLY, for the reason clean_targets() gives: this feeds `rule + all` at module level, so it is evaluated on every per-job re-parse too, none + of which can schedule `all`. + """ + if not COVERAGE_ENABLED or not workflow.is_main_process: + return [] + return [COVERAGE_HSP, COVERAGE_MANIFEST] + + +rule coverage_map: + input: + footprint_targets() + output: + hsp = COVERAGE_HSP, + manifest = COVERAGE_MANIFEST + # No `log:`: this is one job with one verdict, and its stderr is the job's. + params: + products = PRODUCTS_DIR, + nside_coverage = NSIDE_COVERAGE, + nside = NSIDE, + script_hash = COVERAGE_HASH + threads: 1 + resources: + mem_mb = 32000, + runtime = 240 + shell: + "set -euo pipefail\n" + f"python {SCRIPTS}/coverage_map.py" + " --products-dir '{params.products}'" + " --out {output.hsp} --manifest {output.manifest}" + " --nside-coverage {params.nside_coverage} --nside {params.nside}" diff --git a/workflow/rules/exposure.smk b/workflow/rules/exposure.smk index 5845adb8d..fb6f3ecca 100644 --- a/workflow/rules/exposure.smk +++ b/workflow/rules/exposure.smk @@ -1,6 +1,6 @@ """Exposure chain — per exposure, keyed by exp base id (dedup is structural). - exp_get_images -> exp_split -> exp_psf -> exp_persist + exp_get_images -> exp_split -> exp_psf -> exp_persist -> exp_footprint Each in the exposure's own sharded work dir, chained by manifests; every config reads fixed ``$SP_RUN/output/run_sp_exp_*`` INPUT_DIRs, so nothing resolves a @@ -21,8 +21,12 @@ here, and no ``exp_mask``. packs the PSF products named by `persist_exp:` into one tar per exposure off /scratch before the purge (or clean_exposure) can take them. It is a separate rule from exp_psf precisely so that editing that list costs a re-pack and not a -four-hour refit; the full -argument is in workflow/scripts/persist_exp.py. +four-hour refit; the full argument is in workflow/scripts/persist_exp.py. +``exp_footprint`` follows it, and writes to the same root for the same reason: +it records where on the sky the CCDs that got a PSF model are, which is the +coverage mask's raw material (workflow/scripts/exp_footprint.py, +rules/coverage.smk). + NO temp() anywhere in this file, ever (D5). Exposures overlap tiles by construction (~7-10 tiles each), so their consumer set closes over the CAMPAIGN, @@ -169,6 +173,58 @@ rule exp_persist: " {params.patterns}" +# --- per-CCD sky footprints ------------------------------------------------- +# One JSON per exposure recording, for each CCD that HAS a PSF model, the four +# sky corners of that CCD. It is the raw material of the campaign's coverage mask +# (rules/coverage.smk), and it replaces the v1.x chain's VOSpace header download +# + summary-scrape subtraction with a local read of two things this workflow +# already wrote. workflow/scripts/exp_footprint.py argues both inputs and the +# CCD-index invariant tests/unit/test_exp_footprint.py pins. +# +# A LOCALRULE (declared in the Snakefile), by exp_persist's arithmetic and then +# some: one pickle load and ~160 pixel_to_world calls, milliseconds, against a +# scheduling latency of seconds and ~20k exposures at DR6 scale. +# +# ONE DECLARED INPUT, AND IT IS THE PERSIST MANIFEST — deliberately NOT +# exp_psf's as well, even though the WCS array this rule reads is written by +# exp_split and lives in the same scratch store. exp_persist already orders this +# rule after the whole PSF chain, so the second edge would buy no ordering; what +# it WOULD buy is a scratch manifest (the one clean_exposure deletes) in the +# input list of a rule whose output is durable. A persistent-root manifest +# outliving a purged scratch store is a real state, and in it that edge would +# schedule a four-hour VOS rebuild of the exposure to satisfy a few KB of +# provenance. Declaring only the durable input makes the same state a loud +# one-line failure from the script instead. +# +# WRITES TO THE PERSISTENT ROOT, beside exp_persist's manifest and for the same +# reason: the record must outlive both reclamation and the purge — a coverage +# map is campaign-cumulative, so a record written today is read by every map +# built after it. +rule exp_footprint: + input: + lambda wc: prod_exp_manifest(wc.exp, "exp_persist") + output: + manifest = f"{PROD_EXP_DIR}/manifests/exp_footprint.json" + # No `log:`, for exp_persist's reason: the failure modes are "no WCS array" + # and "the two stages disagree about the focal plane", both of which the + # script reports on stderr and neither of which has a per-CCD verdict. + params: + exp_dir = lambda wc: exp_dir(wc.exp), + persist = lambda wc: prod_exp_manifest(wc.exp, "exp_persist"), + script_hash = FOOTPRINT_HASH + threads: 1 + retries: 2 + resources: + mem_mb = 2000, + runtime = 10 + shell: + "set -euo pipefail\n" + f"python {SCRIPTS}/exp_footprint.py" + " --exp-dir '{params.exp_dir}' --exp {wildcards.exp}" + " --persist-manifest '{params.persist}'" + " --manifest {output.manifest}" + + # --- reclamation (D5) ------------------------------------------------------- # The one exception to "no reclamation in this file": clean_exposure OWNS # exposure-level deletion, and it is a real job, not temp() bookkeeping, because @@ -207,7 +263,13 @@ rule clean_exposure: # exposure's chain did not already put there. It is UNCONDITIONAL now: # exp_persist always packs the star catalogue's inputs, so there is no # keep list under which this rule has nothing to wait for. - lambda wc: [prod_exp_manifest(wc.exp, "exp_persist")] + lambda wc: [prod_exp_manifest(wc.exp, "exp_persist")], + # And the footprint, for the same ordering reason one layer further out: + # it is derived from headers-.npy, which lives in the store this job + # deletes. Reclamation must not overtake the read, and unlike the purge + # this deletion is ours to order. Unconditional for exp_persist's reason: + # the psf_validation catalogues it reads are always packed. + lambda wc: [prod_exp_manifest(wc.exp, "exp_footprint")] output: tombstone = f"{EXP_DIR}/cleaned.json" params: diff --git a/workflow/scripts/coverage_map.py b/workflow/scripts/coverage_map.py new file mode 100644 index 000000000..88039a434 --- /dev/null +++ b/workflow/scripts/coverage_map.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""Build the campaign's HealSparse coverage (nexp) map from the exposure footprints. + +Run as the shell of the in-DAG ``coverage_map`` rule, never by hand. + +WHAT THE MAP IS. Per sky pixel, the number of exposures with a VALID PSF MODEL +covering it. The CCDs of one exposure do not overlap, so stamping value 1 per CCD +polygon and accumulating counts exposures. sp_validation applies it as a +structural mask (``notebooks/demo_apply_hsp_masks.py``). + +CAMPAIGN-CUMULATIVE, AND THAT IS THE POINT. This script GLOBS every +``/exp/*/*/manifests/exp_footprint.json`` — not just the ones the +rule declared as inputs, and INCLUDING exposures whose scratch stores have been +reclaimed. The declared inputs are the in-scope, non-tombstoned footprints, which +is what buys ordering and rerun semantics without dragging out-of-scope tiles +into the DAG; the records themselves live on the persistent root and stay valid +sky forever. So appending tiles GROWS the map rather than replacing it, which is +what a survey coverage mask should do. The rule's comment says the same thing +where a reader of the DAG will meet it. + +THE STAMPING IS NOT HERE. It is ``shapepipe.utilities.coverage_map_builder. +build_map``, which is where the RA-seam guard, the pole guard and the polygon +accumulation live. This script is the JSON-to-arrays half, and the records are +raw sky: unwrapping across RA=0 happens once, inside build_map, at the moment a +polygon is stamped. + +NSIDE IS NOT DEFAULTED ANYWHERE. Both values are required arguments, carried +from `coverage:` in config.yaml, and the difference they make is invisible in +the output: nside=131072 is ~0.1"/pixel, chosen to match the UNIONS bit-mask +resolution so coverage and mask align pixel-wise. A map built coarser would look +entirely reasonable and would not align, and the consumer would not notice. +""" + +import argparse +import filecmp +import json +import sys +from pathlib import Path + +import numpy as np + +from shapepipe.utilities.coverage_map_builder import build_map + +# Where exp_footprint writes, relative to the products root. The shard and the +# exposure id are both globbed: this map is the whole campaign's, so it is +# deliberately not built from a list of units. +FOOTPRINT_GLOB = "exp/*/*/manifests/exp_footprint.json" + + +def read_footprints(products_dir): + """Every exposure footprint on the persistent root, as flat arrays. + + Returns ``(ccd_ids, ra, dec, units)``: length-M id and ``(M, 4)`` corner + arrays over all CCDs of all exposures, plus the sorted exposure ids that + contributed. Records are read in sorted path order so the polygon order — + and hence the map — does not depend on readdir order. + """ + paths = sorted(Path(products_dir).glob(FOOTPRINT_GLOB)) + if not paths: + sys.exit(f"coverage_map: no {FOOTPRINT_GLOB} under {products_dir}; " + f"there is nothing to build a map from") + + ccd_ids, ra, dec, units = [], [], [], [] + for path in paths: + body = json.loads(path.read_text()) + units.append(body["unit"]) + for ccd in body["ccds"]: + ccd_ids.append(ccd["id"]) + ra.append(ccd["ra"]) + dec.append(ccd["dec"]) + + # Records but no CCDs is the other empty map, and it is the quieter one: it + # means every exposure on the root lost every CCD, which is a broken PSF + # stage rather than a survey with no coverage. Written out it would be a + # valid, plausible-looking .hsp full of nothing, and its consumer masks + # everything. + if not ccd_ids: + sys.exit(f"coverage_map: {len(paths)} footprint record(s) under " + f"{products_dir}, and not one names a CCD with a PSF model; " + f"there is nothing to stamp") + + return (np.array(ccd_ids, dtype=str), + np.array(ra, dtype=float).reshape(-1, 4), + np.array(dec, dtype=float).reshape(-1, 4), + sorted(units)) + + +def write_stable(path, body): + """Write JSON tmp-then-``cmp``-then-``mv``; an unchanged body keeps its mtime.""" + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + ".tmp") + try: + tmp.write_text(json.dumps(body, indent=2, sort_keys=True) + "\n") + if path.exists() and filecmp.cmp(tmp, path, shallow=False): + tmp.unlink() + else: + tmp.replace(path) + finally: + tmp.unlink(missing_ok=True) + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("--products-dir", required=True, type=Path, + help="the persistent root; every exposure footprint under " + "it goes into the map") + p.add_argument("--out", required=True, type=Path, + help="the HealSparse map, /coverage/coverage.hsp") + p.add_argument("--manifest", required=True, type=Path) + p.add_argument("--nside-coverage", required=True, type=int) + p.add_argument("--nside", required=True, type=int) + p.add_argument("--verbose", action="store_true") + args = p.parse_args() + + ccd_ids, ra, dec, units = read_footprints(args.products_dir) + print(f"[coverage_map] {len(units)} exposure(s), {len(ccd_ids)} CCD " + f"footprint(s) from {args.products_dir}") + + hsp_map = build_map(ccd_ids, ra, dec, args.nside_coverage, args.nside, + verbose=args.verbose) + + args.out.parent.mkdir(parents=True, exist_ok=True) + hsp_map.write(str(args.out), clobber=True) + + # The manifest is written AFTER the map, and it is the rule's record of + # which exposures the map contains — the question a mask's consumer asks + # months later, and one nothing else on disk can answer once the campaign + # has grown past it. + write_stable(args.manifest, { + "stage": "coverage_map", "level": "campaign", "status": "complete", + "map": str(args.out), + "nside_coverage": args.nside_coverage, + "nside": args.nside, + "n_exposures": len(units), + "n_ccds": len(ccd_ids), + "exposures": units, + }) + print(f"[coverage_map] -> {args.out}") + + +if __name__ == "__main__": + main() diff --git a/workflow/scripts/exp_footprint.py b/workflow/scripts/exp_footprint.py new file mode 100644 index 000000000..ff30603be --- /dev/null +++ b/workflow/scripts/exp_footprint.py @@ -0,0 +1,215 @@ +#!/usr/bin/env python3 +"""Record ONE exposure's per-CCD sky footprint, for the CCDs that have a PSF model. + +Run as the shell of the in-DAG ``exp_footprint`` rule, never by hand. + +WHAT THIS REPLACES. The v1.x coverage chain answered "which CCDs have a valid +PSF, and where are they on the sky" by downloading ~25k exposure headers from +VOSpace (``header_downloader.py``), scraping a finished campaign's patch +directories for a ``missing_job_32_all.txt`` and SUBTRACTING it from all 40*N +candidates (``ccd_psf_handler.py``), then appending one row per CCD to a shared +``exp_ra_dec.txt``. Every one of those three moves exists because the pre- +Snakemake pipeline kept no per-unit record. The workflow keeps two, both already +on disk, so this rule is a local read of things it produced itself. + +THE TWO INPUTS, AND WHY EACH IS THE RIGHT ONE. + +1. WHICH CCDS HAVE A PSF -> ``exp_persist.json``'s ``files[].name``, the + ``validation_psf--.fits`` members of the persisted tar. This is + EXACT, not inferred: ``psfex_interp`` returns WITHOUT writing that file on + NOT_ENOUGH_STARS, BAD_CHI2 and FILE_NOT_FOUND, and writes it on success, so + the member list IS the valid-PSF set. It is also DURABLE (persistent root, + survives ``clean_exposure``) and a DECLARED RULE OUTPUT, so the DAG orders + this rule after it for free. The alternative considered and rejected was + ``exp_psf.json``'s psfex_interp entry: that is a COUNT, not a list of names, + and it lives inside the directory ``clean_exposure`` deletes wholesale. + +2. WCS -> ``headers-.npy``, written by ``split_exp`` beside the per-CCD + images. A length-N ``dtype=object`` array; element ``i`` is + ``{"WCS": WCS(h), "header": h.tostring()}``, loaded with ``allow_pickle=True`` + exactly as ``merge_headers.py`` does. + +THE LOAD-BEARING INVARIANT, PINNED BY tests/unit/test_exp_footprint.py: +array index ``i`` == the CCD index in ``image--.fits`` == ``-`` +== ``validation_psf--.fits``. ``split_exp`` writes both the image and the +array element from the same ``idx-1`` in one loop, so the alignment is +structural — but it is a cross-component contract (split_exp's numbering vs. +psfex_interp's filenames vs. this record's ids), and nothing else asserts it now +that the old ``summary.get_all_shdus`` test is gone. No index is ever re-derived +here: a CCD's id comes from its position in the array, never from parsing a name. + +WHY THE SHAPE COMES FROM THE STORED HEADER AND NOT FROM THE WCS. astropy hands +back the DECOMPRESSED header for a tile-compressed HDU, so this npy carries true +``NAXIS1/2`` and no ``ZIMAGE`` — while the old VOSpace text headers carried the +binary-table ``NAXIS`` and needed ``ZNAXIS1/2``. ``_image_shape`` handles both, +and is imported rather than reimplemented for exactly that reason. ``WCS`` drops +the ``Z*`` keywords, so it cannot answer the question either way. + +DATA AND MANIFEST IN ONE FILE. 40 rows of 8 floats is a few KB, and inodes are +what bind on /project (persist_exp.py argues the quota arithmetic). Per-exposure +JSON, never an appended shared text file: a single appended file is precisely +what cannot survive 20k parallel jobs, and it is what the v1.x chain used. +``ccds_no_psf`` is carried explicitly so the record is self-describing about +attrition — a reader can tell "this CCD is not in the map" from "this CCD was +never looked at". + +Written byte-stable, no timestamp, tmp-then-``cmp``-then-``mv``, exactly as +``persist_exp.py`` does and for the same reason: mtime is a rerun trigger, and an +unconditional rewrite would make ``clean_exposure`` look out of date once per +invocation. +""" + +import argparse +import filecmp +import json +import sys +from fnmatch import fnmatch +from pathlib import Path + +import numpy as np +from astropy.io.fits import Header + +from shapepipe.utilities.ccd_footprint import ( + _ccd_corners, + _image_shape, +) + +# Same directory; the rule invokes this file by path, so it is sys.path[0]. +import persist_exp + +# The split stage's run dir (RUN_NAME in config_exp_Sp.ini) and its module +# output dir. Hardcoded rather than passed, for persist_exp.py's reason: this +# rule reads the split stage's headers and nothing else, and a knob here would +# be a knob for "read some other stage". +HEADERS_DIR = "output/run_sp_exp_Sp/split_exp_runner/output" + +# The members this rule selects, named and resolved through the catalogue +# exp_persist packs by, so the glob has one definition. They are always there to +# find: persist_exp packs this product for every exposure whatever `persist_exp:` +# says (its ALWAYS), and fails the pack rather than writing a manifest without it. +MEMBER_PRODUCT = persist_exp.ALWAYS +MEMBER_PATTERN = persist_exp.resolve(MEMBER_PRODUCT) + +# Only the PREFIX of a member name is ours to know; the rest is the unit id and +# the CCD index, which is what makes the set a set of indices. Derived from the +# pattern so it cannot drift from it. +PSF_PREFIX = MEMBER_PATTERN.split("-", 1)[0] + + +def valid_ccds(manifest, exp): + """The CCD indices with a PSF model, read off an ``exp_persist`` manifest. + + BY PRODUCT NAME, OR FAILING THAT BY FILE NAME — merge_star_cat.is_member()'s + test, and for its reasons: persist_exp labels every member with the product + it came from, but a tar packed before that field existed carries no label, + and a keep list written as a raw glob labels its members with the glob. + + No precondition on the keep list. exp_persist packs this product for every + exposure whatever `persist_exp:` says, so an empty answer here means the + exposure genuinely lost every CCD, not that the config unpacked them. + """ + body = json.loads(Path(manifest).read_text()) + ccds = set() + for entry in body.get("files") or []: + name = entry.get("name", "") + if not (entry.get("product") == MEMBER_PRODUCT + or fnmatch(name, MEMBER_PATTERN)): + continue + if not name.startswith(f"{PSF_PREFIX}-{exp}-"): + continue + stem = name[len(f"{PSF_PREFIX}-{exp}-"):].removesuffix(".fits") + if not stem.isdigit(): + sys.exit(f"exp_footprint: {exp}: cannot read a CCD index out of " + f"tar member {name!r}") + ccds.add(int(stem)) + return ccds + + +def footprint(headers, exp, with_psf): + """One record's ``ccds`` and ``ccds_no_psf``, in array order. + + ``headers`` is the ``headers-.npy`` array; index ``i`` IS the CCD + index (see the module docstring). Corners are computed only for the CCDs in + ``with_psf`` — a CCD with no PSF model contributes nothing to a map that + counts exposures with a valid PSF, and computing its corners anyway would + invite a later reader to use them. + """ + ccds, no_psf = [], [] + for i, entry in enumerate(headers): + ccd_id = f"{exp}-{i}" + if i not in with_psf: + no_psf.append(ccd_id) + continue + # The WCS is the pickled object split_exp built; the SHAPE must come + # from the stored header text (module docstring). + shape = _image_shape(Header.fromstring(entry["header"])) + ra, dec = _ccd_corners(entry["WCS"], shape) + # float(), because _ccd_corners hands back numpy scalars and this record + # has to be byte-stable: a plain double's repr is, a numpy type's + # serialisation is json's business rather than ours. + ccds.append({"id": ccd_id, + "ra": [float(x) for x in ra], + "dec": [float(x) for x in dec]}) + return ccds, no_psf + + +def write_stable(path, body): + """Write JSON tmp-then-``cmp``-then-``mv``; an unchanged body keeps its mtime.""" + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + ".tmp") + try: + tmp.write_text(json.dumps(body, indent=2, sort_keys=True) + "\n") + if path.exists() and filecmp.cmp(tmp, path, shallow=False): + tmp.unlink() # unchanged: leave the mtime alone + else: + tmp.replace(path) + finally: + tmp.unlink(missing_ok=True) + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("--exp-dir", required=True, type=Path, + help="the exposure's scratch store") + p.add_argument("--exp", required=True) + p.add_argument("--persist-manifest", required=True, type=Path, + help="/exp///manifests/" + "exp_persist.json; names the valid-PSF CCDs") + p.add_argument("--manifest", required=True, type=Path) + args = p.parse_args() + + npy = args.exp_dir / HEADERS_DIR / f"headers-{args.exp}.npy" + if not npy.exists(): + sys.exit(f"exp_footprint: {args.exp}: no WCS array at {npy} — the " + f"split stage's store is gone or never wrote one " + f"(config_exp_Sp.ini's OUTPUT_SUFFIX must include `image`).") + headers = np.load(npy, allow_pickle=True) + + with_psf = valid_ccds(args.persist_manifest, args.exp) + # A PSF file for a CCD the split never produced means the two stages + # disagree about the focal plane — the one failure the id alignment cannot + # absorb, so it is loud rather than silently dropped. + stray = sorted(i for i in with_psf if i >= len(headers)) + if stray: + sys.exit(f"exp_footprint: {args.exp}: {args.persist_manifest} names " + f"CCD(s) {stray} but {npy} holds only {len(headers)}") + + ccds, no_psf = footprint(headers, args.exp, with_psf) + + write_stable(args.manifest, { + "stage": "exp_footprint", "level": "exp", "unit": args.exp, + "status": "complete", + "source_manifest": str(args.persist_manifest), + "n_ccd_headers": len(headers), + "n_valid_psf": len(ccds), + "ccds": ccds, + "ccds_no_psf": no_psf, + }) + + warn = f" ({len(no_psf)} without a PSF model)" if no_psf else "" + print(f"[exp_footprint] {args.exp}: {len(ccds)}/{len(headers)} CCD " + f"footprint(s){warn} -> {args.manifest}") + + +if __name__ == "__main__": + main() diff --git a/workflow/scripts/run_report.py b/workflow/scripts/run_report.py index 11cdcffe4..d1b95e7b2 100644 --- a/workflow/scripts/run_report.py +++ b/workflow/scripts/run_report.py @@ -61,12 +61,12 @@ "tile_ngmix", "tile_merge_cats", "tile_make_cat"] EXP_STAGES = ["exp_get_images", "exp_split", "exp_psf"] -# exp_persist is DELIBERATELY NOT in that list. This report disk-scans the -# scratch run_dir, and exp_persist's manifest is the one exposure manifest that -# lives on products_dir instead — that placement is what makes it survive -# clean_exposure. Listed here it would read as "not run" for every exposure in -# the campaign. Reporting on the persisted products means scanning the second -# root, which is a report this one does not yet do. +# exp_persist and exp_footprint are DELIBERATELY NOT in that list. This report +# disk-scans the scratch run_dir, and those two are the exposure manifests that +# live on products_dir instead — that placement is what makes them survive +# clean_exposure. Listed here they would read as "not run" for every exposure +# in the campaign. Reporting on the persisted products means scanning the +# second root, which is a report this one does not yet do. # The manifests clean_tile leaves on disk (workflow/scripts/clean_tile.py names # the mechanism that owns each). Their presence is therefore NOT evidence that a