suncasa.eovsa.eovsa_pipeline

Attributes

tasks
split
tclean
gencal
clearcal
applycal
gaincal
delmod
tools
qatool
iatool
mstool
tbtool
ms
tb
hostname
is_on_server
is_on_inti
base_dir
pathconfig
udbmsdir
udbmsscldir
udbmsslfcaleddir
udbdir
caltbdir
slfcaltbdir
qlookfitsdir
qlookfigdir
synopticfigdir
workdir_default
parser

Classes

Path_config

Functions

get_tdate_from_basename(vis)
get_local_day_bounds(tim)	Return the local-day bounds used by the daily EOVSA pipeline.
get_synoptic_output_info(tim)	Return the expected synoptic daily FITS products and status file for one day.
summarize_synoptic_outputs(tim)	Summarize synoptic daily FITS availability for one pipeline day.
read_pipeline_status(statusfile)	Read the per-day pipeline status JSON if it exists.
write_pipeline_status(statusfile, state, **extra)	Persist a compact per-day pipeline status JSON.
get_calibration_deadline_utc(tim[, hour, day_offset])	Return the UTC hard deadline for one observing day label.
get_calibration_readiness(tim)	Check whether the observer-written daily calibrations are ready for one day.
should_enable_smart_cal_check([enable_flag])	Resolve the cron-oriented calibration gate behavior.
getspwfreq(vis)
trange2ms([trange, doimport, verbose, doscaling, ...])	This finds all solar UDBms files within a timerange; If the UDBms file does not exist
calib_pipeline(trange[, workdir, doimport, overwrite, ...])	trange: can be 1) a single Time() object: use the entire day
mk_qlook_image(trange[, doimport, docalib, ncpu, ...])	trange: can be 1) a single Time() object: use the entire day
plt_qlook_image(imres[, figdir, verbose, synoptic])
qlook_image_pipeline(date[, twidth, ncpu, doimport, ...])	date: date string or Time object. e.g., '2017-07-15' or Time('2017-07-15')
pipeline([year, month, day, ndays, clearcache, ...])	Main pipeline for importing and calibrating EOVSA visibility data.

Module Contents

suncasa.eovsa.eovsa_pipeline.tasks

suncasa.eovsa.eovsa_pipeline.split

suncasa.eovsa.eovsa_pipeline.tclean

suncasa.eovsa.eovsa_pipeline.gencal

suncasa.eovsa.eovsa_pipeline.clearcal

suncasa.eovsa.eovsa_pipeline.applycal

suncasa.eovsa.eovsa_pipeline.gaincal

suncasa.eovsa.eovsa_pipeline.delmod

suncasa.eovsa.eovsa_pipeline.tools

suncasa.eovsa.eovsa_pipeline.qatool

suncasa.eovsa.eovsa_pipeline.iatool

suncasa.eovsa.eovsa_pipeline.mstool

suncasa.eovsa.eovsa_pipeline.tbtool

suncasa.eovsa.eovsa_pipeline.ms

suncasa.eovsa.eovsa_pipeline.tb

suncasa.eovsa.eovsa_pipeline.get_tdate_from_basename(vis)

suncasa.eovsa.eovsa_pipeline.hostname

suncasa.eovsa.eovsa_pipeline.is_on_server

suncasa.eovsa.eovsa_pipeline.is_on_inti

suncasa.eovsa.eovsa_pipeline.base_dir = '/inti/data/pipeline_mirror'

class suncasa.eovsa.eovsa_pipeline.Path_config(base_dir=base_dir)

paths

base_dir = '/inti/data/pipeline_mirror'

udbmsdir

udbmsscldir

udbmsslfcaleddir

udbdir

caltbdir

slfcaltbdir

qlookfitsdir

qlookfigdir

synopticfigdir

workdir_default

_get_env_var(env_var, default_path)

_print_summary()

suncasa.eovsa.eovsa_pipeline.pathconfig

suncasa.eovsa.eovsa_pipeline.udbmsdir

suncasa.eovsa.eovsa_pipeline.udbmsscldir

suncasa.eovsa.eovsa_pipeline.udbmsslfcaleddir

suncasa.eovsa.eovsa_pipeline.udbdir

suncasa.eovsa.eovsa_pipeline.caltbdir

suncasa.eovsa.eovsa_pipeline.slfcaltbdir

suncasa.eovsa.eovsa_pipeline.qlookfitsdir

suncasa.eovsa.eovsa_pipeline.qlookfigdir

suncasa.eovsa.eovsa_pipeline.synopticfigdir

suncasa.eovsa.eovsa_pipeline.workdir_default

suncasa.eovsa.eovsa_pipeline.get_local_day_bounds(tim)

Return the local-day bounds used by the daily EOVSA pipeline.

suncasa.eovsa.eovsa_pipeline.get_synoptic_output_info(tim)

Return the expected synoptic daily FITS products and status file for one day.

suncasa.eovsa.eovsa_pipeline.summarize_synoptic_outputs(tim)

Summarize synoptic daily FITS availability for one pipeline day.

suncasa.eovsa.eovsa_pipeline.read_pipeline_status(statusfile)

Read the per-day pipeline status JSON if it exists.

suncasa.eovsa.eovsa_pipeline.write_pipeline_status(statusfile, state, **extra)

Persist a compact per-day pipeline status JSON.

suncasa.eovsa.eovsa_pipeline.get_calibration_deadline_utc(tim, hour=4, day_offset=2)

Return the UTC hard deadline for one observing day label.

suncasa.eovsa.eovsa_pipeline.get_calibration_readiness(tim)

Check whether the observer-written daily calibrations are ready for one day.

suncasa.eovsa.eovsa_pipeline.should_enable_smart_cal_check(enable_flag=None)

Resolve the cron-oriented calibration gate behavior.

suncasa.eovsa.eovsa_pipeline.getspwfreq(vis)

Parameters:

vis

Returns:

mid frequencies in GHz of each spw in the vis

suncasa.eovsa.eovsa_pipeline.trange2ms(trange=None, doimport=False, verbose=False, doscaling=False, overwrite=True)

This finds all solar UDBms files within a timerange; If the UDBms file does not exist in EOVSAUDBMSSCL, create one by calling importeovsa Required inputs: trange - can be 1) a single string or Time() object in UTC: use the entire day, e.g., ‘2017-08-01’ or Time(‘2017-08-01’)

if just a date, find all scans withing the same date in local time. if a complete time stamp, find the local date first (which may be different from that provided,

and return all scans within that day

2. a range of Time(), e.g., Time([‘2017-08-01 00:00’,’2017-08-01 23:00’])

3. None – use current date Time.now()

doimport - Boolean. If true, call importeovsa to import UDB files that are missing from

those found in the directory specified in EOVSAUDBMSSCL. Otherwise, return a list of ms files it has found.

doscaling - Boolean. If true, scale cross-correlation amplitudes by using auto-correlations verbose - Boolean. If true, return more information

suncasa.eovsa.eovsa_pipeline.calib_pipeline(trange, workdir=None, doimport=False, overwrite=False, clearcache=False, verbose=False, pols='XX', version='v3.0', ncpu='auto', caltype=['refpha', 'phacal'], interp='nearest', force_imaging_rerun=False)

trange: can be 1) a single Time() object: use the entire day

2. a range of Time(), e.g., Time([‘2017-08-01 00:00’,’2017-08-01 23:00’])

3. a single or a list of UDBms file(s)

4. None – use current date Time.now()

suncasa.eovsa.eovsa_pipeline.mk_qlook_image(trange, doimport=False, docalib=False, ncpu=10, twidth=12, stokes=None, antenna='0~12', lowcutoff_freq=3.7, imagedir=None, spws=['1~5', '6~10', '11~15', '16~25'], toTb=True, overwrite=True, doslfcal=False, verbose=False)

trange: can be 1) a single Time() object: use the entire day

2. a range of Time(), e.g., Time([‘2017-08-01 00:00’,’2017-08-01 23:00’])

3. a single or a list of UDBms file(s)

4. None – use current date Time.now()

suncasa.eovsa.eovsa_pipeline.plt_qlook_image(imres, figdir=None, verbose=True, synoptic=False)

suncasa.eovsa.eovsa_pipeline.qlook_image_pipeline(date, twidth=10, ncpu=15, doimport=False, docalib=False, synoptic=False, overwrite=True)

date: date string or Time object. e.g., ‘2017-07-15’ or Time(‘2017-07-15’)

suncasa.eovsa.eovsa_pipeline.pipeline(year=None, month=None, day=None, ndays=1, clearcache=True, overwrite=False, doimport=True, pols='XX', version='v1.0', ncpu='auto', debugging=False, caltype=['refpha', 'phacal'], interp='nearest', smart_cal_check=None)

Main pipeline for importing and calibrating EOVSA visibility data.

Name:

eovsa_pipeline — main pipeline for importing and calibrating EOVSA visibility data.

Synopsis:

eovsa_pipeline.py [options]… [DATE_IN_YY_MM_DD]

Description:

Import and calibrate EOVSA visibility data of the date specified by DATE_IN_YY_MM_DD (or from ndays before the DATE_IN_YY_MM_DD if option –ndays/-n is provided). If DATE_IN_YY_MM_DD is omitted, it will be set to 2 days before now by default. There are no mandatory arguments in this command.

Parameters:

• year (int, optional) – The year for which data should be processed, defaults to None.

• month (int, optional) – The month for which data should be processed, defaults to None.

• day (int, optional) – The day for which data should be processed, defaults to None.

• ndays (int, optional) – Number of days before the specified date to include in the processing, defaults to 1.

• clearcache (bool, optional) – Whether to clear cache after processing, defaults to True.

• overwrite (bool, optional) – Whether to overwrite existing files, defaults to True.

• doimport (bool, optional) – Whether to perform the import step, defaults to True.

• pols (str, optional) – Polarizations to process, can be ‘XX’ or ‘XXYY’, defaults to ‘XX’.

• version (str, optional) – Version of the pipeline to use, choices are ‘v1.0’ or ‘v2.0’, defaults to ‘v1.0’.

• ncpu (str, optional) – Number of CPUs to use for processing, defaults to ‘auto’.

• debugging (bool, optional) – Whether to run the pipeline in debugging mode, defaults to False.

• caltype (list, optional) – Calibration types to use, defaults to [‘refpha’,’phacal’].

• interp (str, optional) – Interpolation method to use for calibration tables, defaults to ‘nearest’. Options are ‘nearest’, ‘linear’

• smart_cal_check (bool, optional) – When True, perform cron-oriented calibration readiness checks and track per-day status before processing, defaults to None. If None, the behavior is enabled automatically when EOVSA_PIPELINE_CRON=1. The gate waits for same observing-day calibration records until the hard deadline at 04:00 UTC two days after the observing-day label, then allows the run to proceed with the latest calibration records available in MySQL.

Raises:

ValueError – Raises an exception if the date parameters are out of the valid Gregorian calendar range.

Example:

To process data for November 24th, 2021 using version 2.0 of the pipeline, with all options enabled:

>>> python eovsa_pipeline.py --date 2021-11-24T20:00 --clearcache --overwrite --doimport --pols XX --version v2.0 --ndays 2

If you want to see the help message, you can run:

>>> python eovsa_pipeline.py -h

suncasa.eovsa.eovsa_pipeline.parser