tidy3d.web.Batch

tidy3d.web.Batch#

class Batch[source]#

Bases: WebContainer

Interface for submitting several Simulation objects to sever.

Parameters:

  • simulations (Union[dict[str, Union[Simulation, HeatChargeSimulation, HeatSimulation, EMESimulation, ModeSolver, ModeSimulation, VolumeMesher, ModalComponentModeler, TerminalComponentModeler]], tuple[Union[Simulation, HeatChargeSimulation, HeatSimulation, EMESimulation, ModeSolver, ModeSimulation, VolumeMesher, ModalComponentModeler, TerminalComponentModeler], …]]) – Mapping of task names to Simulations to run as a batch.

  • folder_name (str = default) – Name of folder to store member of each batch on web UI.

  • verbose (bool = True) – Whether to print info messages and progressbars.

  • solver_version (Optional[str] = None) – Custom solver version to use, otherwise uses default for the current front end version.

  • callback_url (Optional[str] = None) – Http PUT url to receive simulation finish event. The body content is a json file with fields {'id', 'status', 'name', 'workUnit', 'solverVersion'}.

  • simulation_type (Literal['tidy3d', 'microwave', 'tidy3d_design', 'tidy3d_autograd', 'tidy3d_autograd_async', 'autograd_fwd', 'autograd_bwd'] = tidy3d) – Type of each simulation in the batch, used internally only.

  • parent_tasks (Optional[dict[str, tuple[str, ...]]] = None) – Collection of parent task ids for each job in batch, used internally only.

  • num_workers (Optional[PositiveInt] = 10) – Number of workers for batch multi-threading where configurable. Corresponds to max_workers argument passed to concurrent.futures.ThreadPoolExecutor. Upload/start use a fixed concurrency of 64. Defaults to config.web.default_num_workers.

  • reduce_simulation (Literal['auto', True, False] = auto) – Whether to reduce structures in the simulation to the simulation domain only. Note: currently only implemented for the mode solver.

  • pay_type (PayType = AUTO) – Specify the payment method.

  • jobs_cached (Optional[dict[str, Job]] = None) – Optional field to specify jobs. Only used as a workaround internally so that jobs is written when Batch.to_file() and then the proper task is loaded from Batch.from_file(). We recommend leaving unset as setting this field along with fields that were not used to create the task will cause errors.

  • lazy (bool = False) – Whether to load the actual data (lazy=False) or return a proxy that loads the data when accessed (lazy=True).

Notes

Commonly one needs to submit a batch of Simulation. The built-in Batch object is the best way to upload, start, monitor, and load a series of tasks. The batch object is like a Job, but stores task metadata for a series of simulations.

See also

tidy3d.web.api.webapi.run_async()

Submits a set of Simulation objects to server, starts running, monitors progress, downloads, and loads results as a BatchData object.

Job:

Interface for managing the running of a Simulation on server.

Notebooks

Attributes

jobs

Create a series of tasks in the Batch and upload them to server.

num_jobs

Number of jobs in the batch.

simulations

folder_name

verbose

solver_version

callback_url

simulation_type

parent_tasks

num_workers

reduce_simulation

pay_type

jobs_cached

lazy

Methods

delete()

Delete server-side data associated with each task in the batch.

download([path_dir, replace_existing])

Download results of each task.

estimate_cost([verbose])

Compute the maximum FlexCredit charge for a given Batch.

from_file(fname[, group_path, lazy, on_load])

Load a Batch from file.

get_info()

Get information about each task in the Batch.

get_run_info()

get information about a each of the tasks in the Batch.

load([path_dir, replace_existing, skip_download])

Download results and load them into BatchData object.

monitor(*[, download_on_success, path_dir, ...])

Monitor progress of each running task.

real_cost([verbose])

Get the sum of billed costs for each task associated with this batch.

run([path_dir, priority, replace_existing, ...])

Upload and run each simulation in Batch.

start([priority, vgpu_allocation, ...])

Start running all tasks in the Batch.

to_file(fname)

Exports Tidy3dBaseModel instance to .yaml, .json, or .hdf5 file

upload()

Upload a series of tasks associated with this Batch using multi-threading.
simulations#
folder_name#
verbose#
solver_version#
callback_url#
simulation_type#
parent_tasks#
num_workers#
reduce_simulation#
pay_type#
jobs_cached#
lazy#
run(path_dir='.', priority=None, replace_existing=False, vgpu_allocation=None, ignore_memory_limit=None)[source]#

Upload and run each simulation in Batch.

Parameters:

  • path_dir (PathLike) – Base directory where data will be downloaded, by default current working directory.

  • priority (int = None) – Priority of the simulation in the Virtual GPU (vGPU) queue (1 = lowest, 10 = highest). It affects only simulations from vGPU licenses and does not impact simulations using FlexCredits.

  • replace_existing (bool = False) – Downloads the data even if path exists (overwriting the existing). Applies when downloading cached results or when download_on_success=True.

  • vgpu_allocation (int = None) – Number of virtual GPUs to allocate for the simulation (1, 2, 4, or 8). Only applies to vGPU license users. If not specified, the system automatically determines the optimal GPU count.

  • ignore_memory_limit (Optional[bool] = None) – If True, allows the simulation to run even when estimated vGPU memory exceeds the allocation limit (up to 2x the limit). Only applies to vGPU license users. Default None leaves the server behaviour unchanged.

Returns:

Contains Union[SimulationData, HeatSimulationData, EMESimulationData] for each Union[Simulation, HeatSimulation, EMESimulation] in Batch.

Return type:

BatchData

Note

A typical usage might look like:

>>> from tidy3d.web.api.container import Batch
>>> custom_batch = Batch()
>>> batch_data = custom_batch.run() 
>>> for task_name, sim_data in batch_data.items(): 
...     # do something with data.

batch_data iterates over task names and loads the corresponding data from file one by one. If no file exists for that task, it downloads it. When config.batch_data_cache.enabled is True and the total size of all task files is below config.batch_data_cache.max_total_size_gb, accessed results are cached in memory to avoid repeated loads.

property jobs#

Create a series of tasks in the Batch and upload them to server.

Note

To start the simulations running, must call Batch.start() after uploaded.

to_file(fname)[source]#

Exports Tidy3dBaseModel instance to .yaml, .json, or .hdf5 file

Parameters:

fname (PathLike) – Full path to the .yaml or .json file to save the Tidy3dBaseModel to.

Example

>>> simulation.to_file(fname='folder/sim.json')
classmethod from_file(fname, group_path=None, lazy=False, on_load=None, **parse_obj_kwargs)[source]#

Load a Batch from file.

Notes

For Batch, lazy=True also configures per-task data loading behavior used by Batch.load().

property num_jobs#

Number of jobs in the batch.

upload()[source]#

Upload a series of tasks associated with this Batch using multi-threading.

get_info()[source]#

Get information about each task in the Batch.

Returns:

Mapping of task name to data about task associated with each task.

Return type:

dict[str, TaskInfo]

start(priority=None, vgpu_allocation=None, ignore_memory_limit=None)[source]#

Start running all tasks in the Batch.

Parameters:

  • priority (int = None) – Priority of the simulation in the Virtual GPU (vGPU) queue (1 = lowest, 10 = highest). It affects only simulations from vGPU licenses and does not impact simulations using FlexCredits.

  • vgpu_allocation (int = None) – Number of virtual GPUs to allocate for the simulation (1, 2, 4, or 8). Only applies to vGPU license users. If not specified, the system automatically determines the optimal GPU count.

  • ignore_memory_limit (Optional[bool] = None) – If True, allows the simulation to run even when estimated vGPU memory exceeds the allocation limit (up to 2x the limit). Only applies to vGPU license users. Default None leaves the server behaviour unchanged.

Note

To monitor the running simulations, can call Batch.monitor().

get_run_info()[source]#

get information about a each of the tasks in the Batch.

Returns:

dict[str – Maps task names to run info for each task in the Batch.

Return type:

RunInfo]

monitor(*, download_on_success=False, path_dir='.', replace_existing=False)[source]#

Monitor progress of each running task.

  • Optionally downloads results as soon as a job reaches final success.

  • Rich progress bars in verbose mode; quiet polling otherwise.

Parameters:

  • download_on_success (bool = False) – If True, automatically start downloading the results for a job as soon as it reaches success.

  • path_dir (PathLike = './') – Base directory where data will be downloaded, by default the current working directory. Only used when download_on_success is True.

  • replace_existing (bool = False) – Downloads the data even if path exists (overwriting the existing). Only used when download_on_success is True.

download(path_dir='.', replace_existing=False)[source]#

Download results of each task.

Parameters:

  • path_dir (PathLike = './') – Base directory where data will be downloaded, by default the current working directory.

  • replace_existing (bool = False) – Downloads the data even if path exists (overwriting the existing).

Note

To load and iterate through the data, use Batch.items().

The data for each task will be named as {path_dir}/{task_id}.hdf5. The Batch hdf5 file will be automatically saved as {path_dir}/batch.hdf5, allowing one to load this Batch later using batch = Batch.from_file().

load(path_dir='.', replace_existing=False, skip_download=False)[source]#

Download results and load them into BatchData object.

Parameters:

  • path_dir (PathLike = './') – Base directory where data will be downloaded, by default current working directory.

  • replace_existing (bool = False) – Downloads the data even if path exists (overwriting the existing).

  • skip_download (bool = False) – Does not trigger download. Should be True if already downloaded.

Returns:

delete()[source]#

Delete server-side data associated with each task in the batch.

real_cost(verbose=True)[source]#

Get the sum of billed costs for each task associated with this batch.

Parameters:

verbose (bool = True) – Whether to log the cost and helpful messages.

Returns:

Billed cost for the entire Batch.

Return type:

float

estimate_cost(verbose=True)[source]#

Compute the maximum FlexCredit charge for a given Batch.

Parameters:

verbose (bool = True) – Whether to log the cost and helpful messages.

Note

Cost is calculated assuming the simulation runs for the full run_time. If early shut-off is triggered, the cost is adjusted proportionately.

Returns:

Estimated total cost of the tasks in FlexCredits.

Return type:

float