Web Workflow

This page explains how to submit and manage web jobs.

Basic Usage

The web management functions are located in the web module of flex_rf.

# Import the web module
import flex_rf.web as web

For basic usage, the web.run() method is a one-stop-shop that submits a job, monitors progress, reports FlexCredit (FC) cost, and downloads the result data in one function call.

# Submit a job using web.run()
my_results = web.run(
    my_simulation,
    task_name='my simulation job',
    path='/my/local/download/path'
)

Any supported simulation type (TerminalComponentModeler or ModeSolver) can be submitted using web.run(). The return type of the result data will correspond to the type of the submitted simulation. If path is omitted, the results download to the current working directory.

Dictionaries and lists of simulations are also supported. This makes it very easy to submit batch jobs.

# Define a dict of simulations
my_batch = {
    "parameter set 1": my_tcm_1,
    "parameter set 2": my_tcm_2,
    "parameter set 3": my_tcm_3,
}

# Submit using one web.run() call
my_batch_results = web.run(my_batch, task_name='my batch job')

The returned shape of the result will match the shape of the input Python dict or list.

# The returned results match the shape of the input batch dict
my_tcm_result_1 = my_batch_results['parameter set 1']

Job Management

The web.run() workflow can be broken down into four individual steps, when more granular control is desired. Common use cases include

• Non-blocking (“fire-and-forget”) job submission, e.g. from an automated script or AI agent
• Loss of internet connection

The first step is to upload a simulation.

# Upload a job and obtain a task_id
task_id = web.upload(my_simulation, task_name='my simulation job')

This does not run the simulation automatically. Instead, it uploads the simulation to the cloud server and returns an auto-generated task_id that functions as a tracking label for the job. The estimated FC cost for the job is also reported.

Tip

Should the task_id be lost, e.g. due to a kernel crash or accidental override, it can also be found on the online Flexcompute workspace. Additionally, when using web.run() or web.upload(), the task_id (also referred to as the resource_id) is included in the printed terminal output.

The next step is to start the job.

# Start execution of an uploaded job
web.start(task_id)

Unlike web.run(), the web.start() method is non-blocking. This allows an automation script or AI agent to launch many jobs simultaneously.

The next step is to check on the running job.

# Monitor progress of a started job
web.monitor(task_id)

Finally, when the job is done, the results can be downloaded.

# Download results from a completed job
my_results = web.load(task_id)

Use the web.abort() method to abort a task in progress.

# Abort a task in progress
web.abort(task_id)

FlexCredit Cost

The estimated FC cost is calculated based on the total number of FDTD grid cells and time steps. It can be interpreted as the maximum cost for a given run. However, it is possible for the simulation to complete early due to the energy shutoff condition. In that case, the real FC cost will be a fraction of the estimated cost, reduced proportionately according to the number of actual simulated time steps.

When the 3D simulation includes wave ports, additional cost will be incurred for mode solver calculations necessary for each port. This is usually a very small fraction of the total FC cost.

Estimated and real FC costs are reported by web.run() in the terminal output. It is also possible to query the cost information explicitly.

# Obtain estimated cost
cost_estimated = web.estimate_cost(task_id)

# Obtain real cost (after job completion)
cost_real = web.real_cost(task_id)