tidy3d.web.Batch

tidy3d.web.Batch#

class Batch[source]#

Bases: WebContainer

Interface for submitting several Simulation objects to sever.

Parameters:

  • simulations (Union[dict[str, Union[Simulation, HeatChargeSimulation, HeatSimulation, EMESimulation, ModeSolver, ModeSimulation, VolumeMesher, ModalComponentModeler, TerminalComponentModeler]], tuple[Union[Simulation, HeatChargeSimulation, HeatSimulation, EMESimulation, ModeSolver, ModeSimulation, VolumeMesher, ModalComponentModeler, TerminalComponentModeler], …]]) – Mapping of task names to Simulations to run as a batch.

  • folder_name (str = default) – Name of folder to store member of each batch on web UI.

  • verbose (bool = True) – Whether to print info messages and progressbars.

  • solver_version (Optional[str] = None) – Custom solver version to use, otherwise uses default for the current front end version.

  • callback_url (Optional[str] = None) – Http PUT url to receive simulation finish event. The body content is a json file with fields {'id', 'status', 'name', 'workUnit', 'solverVersion'}.

  • simulation_type (Literal['tidy3d', 'microwave', 'tidy3d_design', 'tidy3d_autograd', 'tidy3d_autograd_async', 'autograd_fwd', 'autograd_bwd'] = tidy3d) – Type of each simulation in the batch, used internally only.

  • parent_tasks (Optional[dict[str, tuple[str, ...]]] = None) – Collection of parent task ids for each job in batch, used internally only.

  • num_workers (Optional[PositiveInt] = 10) – Number of workers for multi-threading upload and download of batch. Corresponds to max_workers argument passed to concurrent.futures.ThreadPoolExecutor. When left None, will pass the maximum number of threads available on the system.

  • reduce_simulation (Literal['auto', True, False] = auto) – Whether to reduce structures in the simulation to the simulation domain only. Note: currently only implemented for the mode solver.

  • pay_type (PayType = AUTO) – Specify the payment method.

  • jobs_cached (Optional[dict[str, Job]] = None) – Optional field to specify jobs. Only used as a workaround internally so that jobs is written when Batch.to_file() and then the proper task is loaded from Batch.from_file(). We recommend leaving unset as setting this field along with fields that were not used to create the task will cause errors.

  • lazy (bool = False) – Whether to load the actual data (lazy=False) or return a proxy that loads the data when accessed (lazy=True).

Notes

Commonly one needs to submit a batch of Simulation. The built-in Batch object is the best way to upload, start, monitor, and load a series of tasks. The batch object is like a Job, but stores task metadata for a series of simulations.

See also

tidy3d.web.api.webapi.run_async()

Submits a set of Simulation objects to server, starts running, monitors progress, downloads, and loads results as a BatchData object.

Job:

Interface for managing the running of a Simulation on server.

Notebooks

Attributes

jobs

Create a series of tasks in the Batch and upload them to server.

num_jobs

Number of jobs in the batch.

simulations

folder_name

verbose

solver_version

callback_url

simulation_type

parent_tasks

num_workers

reduce_simulation

pay_type

jobs_cached

lazy

Methods

delete()

Delete server-side data associated with each task in the batch.

download([path_dir, replace_existing])

Download results of each task.

estimate_cost([verbose])

Compute the maximum FlexCredit charge for a given Batch.

get_info()

Get information about each task in the Batch.

get_run_info()

get information about a each of the tasks in the Batch.

load([path_dir, replace_existing, skip_download])

Download results and load them into BatchData object.

monitor(*[, download_on_success, path_dir, ...])

Monitor progress of each running task.

real_cost([verbose])

Get the sum of billed costs for each task associated with this batch.

run([path_dir, priority, replace_existing])

Upload and run each simulation in Batch.

start([priority])

Start running all tasks in the Batch.

to_file(fname)

Exports Tidy3dBaseModel instance to .yaml, .json, or .hdf5 file

upload()

Upload a series of tasks associated with this Batch using multi-threading.

Inherited Common Usage

simulations#
folder_name#
verbose#
solver_version#
callback_url#
simulation_type#
parent_tasks#
num_workers#
reduce_simulation#
pay_type#
jobs_cached#
lazy#
run(path_dir='.', priority=None, replace_existing=False)[source]#

Upload and run each simulation in Batch.

Parameters:

  • path_dir (PathLike) – Base directory where data will be downloaded, by default current working directory.

  • priority (int = None) – Priority of the simulation in the Virtual GPU (vGPU) queue (1 = lowest, 10 = highest). It affects only simulations from vGPU licenses and does not impact simulations using FlexCredits.

  • replace_existing (bool = False) – Downloads the data even if path exists (overwriting the existing). Applies when downloading cached results or when download_on_success=True.

Returns:

Contains Union[SimulationData, HeatSimulationData, EMESimulationData] for each Union[Simulation, HeatSimulation, EMESimulation] in Batch.

Return type:

BatchData

Note

A typical usage might look like:

>>> from tidy3d.web.api.container import Batch
>>> custom_batch = Batch()
>>> batch_data = custom_batch.run() 
>>> for task_name, sim_data in batch_data.items(): 
...     # do something with data.

batch_data iterates over task names and loads the corresponding data from file one by one. If no file exists for that task, it downloads it. When config.batch_data_cache.enabled is True and the total size of all task files is below config.batch_data_cache.max_total_size_gb, accessed results are cached in memory to avoid repeated loads.

property jobs#

Create a series of tasks in the Batch and upload them to server.

Note

To start the simulations running, must call Batch.start() after uploaded.

to_file(fname)[source]#

Exports Tidy3dBaseModel instance to .yaml, .json, or .hdf5 file

Parameters:

fname (PathLike) – Full path to the .yaml or .json file to save the Tidy3dBaseModel to.

Example

>>> simulation.to_file(fname='folder/sim.json')
property num_jobs#

Number of jobs in the batch.

upload()[source]#

Upload a series of tasks associated with this Batch using multi-threading.

get_info()[source]#

Get information about each task in the Batch.

Returns:

Mapping of task name to data about task associated with each task.

Return type:

dict[str, TaskInfo]

start(priority=None)[source]#

Start running all tasks in the Batch.

Parameters:

priority (int = None) – Priority of the simulation in the Virtual GPU (vGPU) queue (1 = lowest, 10 = highest). It affects only simulations from vGPU licenses and does not impact simulations using FlexCredits.

Note

To monitor the running simulations, can call Batch.monitor().

get_run_info()[source]#

get information about a each of the tasks in the Batch.

Returns:

dict[str – Maps task names to run info for each task in the Batch.

Return type:

RunInfo]

monitor(*, download_on_success=False, path_dir='.', replace_existing=False)[source]#

Monitor progress of each running task.

  • Optionally downloads results as soon as a job reaches final success.

  • Rich progress bars in verbose mode; quiet polling otherwise.

Parameters:

  • download_on_success (bool = False) – If True, automatically start downloading the results for a job as soon as it reaches success.

  • path_dir (PathLike = './') – Base directory where data will be downloaded, by default the current working directory. Only used when download_on_success is True.

  • replace_existing (bool = False) – Downloads the data even if path exists (overwriting the existing). Only used when download_on_success is True.

download(path_dir='.', replace_existing=False)[source]#

Download results of each task.

Parameters:

  • path_dir (PathLike = './') – Base directory where data will be downloaded, by default the current working directory.

  • replace_existing (bool = False) – Downloads the data even if path exists (overwriting the existing).

Note

To load and iterate through the data, use Batch.items().

The data for each task will be named as {path_dir}/{task_id}.hdf5. The Batch hdf5 file will be automatically saved as {path_dir}/batch.hdf5, allowing one to load this Batch later using batch = Batch.from_file().

load(path_dir='.', replace_existing=False, skip_download=False)[source]#

Download results and load them into BatchData object.

Parameters:

  • path_dir (PathLike = './') – Base directory where data will be downloaded, by default current working directory.

  • replace_existing (bool = False) – Downloads the data even if path exists (overwriting the existing).

  • skip_download (bool = False) – Does not trigger download. Should be True if already downloaded.

Returns:

delete()[source]#

Delete server-side data associated with each task in the batch.

real_cost(verbose=True)[source]#

Get the sum of billed costs for each task associated with this batch.

Parameters:

verbose (bool = True) – Whether to log the cost and helpful messages.

Returns:

Billed cost for the entire Batch.

Return type:

float

estimate_cost(verbose=True)[source]#

Compute the maximum FlexCredit charge for a given Batch.

Parameters:

verbose (bool = True) – Whether to log the cost and helpful messages.

Note

Cost is calculated assuming the simulation runs for the full run_time. If early shut-off is triggered, the cost is adjusted proportionately.

Returns:

Estimated total cost of the tasks in FlexCredits.

Return type:

float