Development Guide 🛠️

Welecome to the tidy3d developers guide! These are just some recommendations I’ve compiled, but we can change anything as we think might help the development cycle more.

Project Structure

As of tidy3d>=2.6, the frontend has been restructured to improve the development cycle. The project directories follow the following structure, which is derived from some recommended Python project architecture guides. This is a handy structure because many tools, such as sphinx, integrate quite well with this type of project layout.

docs/
    # sphinx rst files
    ...
    notebooks/
        # Git submodule repository
        # Checks out github.com/flexcompute/tidy3d-notebooks
tests/
    # pytest source and docs
    # pytest notebooks
scripts/
    # useful handy scripts
tidy3d/
    # python source code
...
pyproject.toml # python packaging
poetry.lock # environment management

It is important to note the new tools we are using to manage our development environment and workflow.

• poetry

• pipx

Installation

The Fast Lane

Maybe you already have tidy3d installed in some form. After installing version tidy3d>=2.6, you can use a few terminal commands to set you up on the correct environment and perform common development tasks. Just run in your terminal, tidy3d develop to get the latest list of commands.

It does not matter how you have installed tidy3d before as long as you have any form of tidy3d>=2.6` in your environment. This can help you transition from a standard user installation to a development environment installation.

Quick Start

Instructions for anyone who wants to test the new development flow before it gets included as part of the pre-release:

For ubuntu:

git clone https://github.com/flexcompute/tidy3d.git
cd tidy3d
# Make sure you're in a branch > pre/2.6 and you can `import tidy3d` in python
pip install -e . # or whatever local installation works for you
tidy3d develop # Read all the new development helper commands
# tidy3d develop uninstall-dev-envrionment # in case you need to reset your environment
tidy3d develop install-dev-environment # install all requirements that you don't have and verify the exisiting ones
poetry run tidy3d develop verify-dev-environment # reproducibly verify development envrionment
# poetry run tidy3d develop build-docs # eg. reproducibly build documentation

Now you can run the following tidy3d cli commands to test them.

Automatic Environment Installation (Beta)

If you are transitioning from the old development flow, to this new one, there are a list of commands you can run to make your life easier and set you up well:

# Automatically check and install requirements like pipx, poetry, pandoc
tidy3d develop install-dev-environment

Note that this is just a automatic script implementation of the The Detailed Lane instructions. It is intended to help you and raise warnings with suggestions of how to fix an environment setup issue. You do not have to use this helper function and can just follow the instructions in The Detailed Lane. All commands are echo-ed in the terminal so you will be able to observe and reproduce what is failing if you desire.

The way this command works is dependent on the operating system you are running. There are some prerequisites for each platform, but the command line tool will help you identify and install the tools it requires. You should rerun the command after you have installed any prerequisite as it will just progress with the rest of the tools installation. If you already have the tool installed, it will verify that it conforms to the supported versions.

This command will first check if you already have installed the development requirements, and if not, it will run the installation scripts for pipx, poetry, and ask you to install the required version of pandoc. It will also install the development requirements and tidy3d package in a specific poetry environment.

Environment Verification

If you rather install poetry, pipx and pandoc yourself, you can run the following command to verify that your environment conforms to the reproducible development environment which would be equivalent to the one installed automatically above and described in The Detailed Lane.

tidy3d develop verify-dev-environment

The Detailed Lane

If you do not have any of the above tools already installed and want to install them manually, let’s go through the process of setting things up from scratch:

Environment Requirements

Make sure you have installed pipx. We provide common installation flows below:

Ubuntu 22.04macOSWindows

This installation flow requires a python3 installation. Depending how you have installed python3, you may have to edit this command to run on your target installation. Further instructions by pipx here

python3 -m pip install --user pipx
python3 -m pipx ensurepath

Then install poetry:

Ubuntu 22.04macOSWindows

Further instructions in the poetry installation instructions

python3 -m pipx install poetry

After restarting the bash terminal, you should be able to find poetry in your PATH if it has been installed correctly:

poetry --version
poetry # prints all commands

If you want to locally build documentation, then it is required to install pandoc<3.

Ubuntu 22.04macOSWindows

Further instructions in the pandoc installation instructions. Note you will need permissions to do this.

sudo apt-get update
sudo apt-get install pandoc

Now you need to install the package in the reproducible poetry environment in development mode:

poetry install -E dev

Congratulations! Now you have all the required tools installed, you can now use all the poetry run tidy3d develop commands reproducibly.

If you want to contribute to the project, read the following section:

More Contribution Requirements

If you want to contribute to the development of tidy3d, you can follow the instructions below to set up your development environment. This will allow you to run the tests, build the documentation, and run the examples. Another thing you need to do before committing to the project is to install the pre-commit hooks. This will ensure that your code is formatted correctly and passes the tests before you commit it. To do this, run the following command:

poetry run pre-commit install

This will run a few file checks on your code before you commit it. After this whenever you commit, the pre-commit hooks will run automatically. If any of the checks fail, you will have to fix the issues before you can commit. If for some reason, it’s a check you want to waive, you can follow the instructions of the tool to automatically waive them or you can run the following command to skip the checks only on minimal circumstances:

git commit --no-verify

You can also run the checks manually on all files by running the following command:

poetry run pre-commit run --all-files

Packaging Equivalent Functionality

This package installation process should be approximately equivalent to the previous setup.py installation flow. Independent of the poetry development flow, it is possible to run any of the following commands in any particular virtual environment you have configured:

pip install tidy3d[dev]
pip install tidy3d[docs]
pip install tidy3d[web]
...
pip install tidy3d[jax]

All these options can be found inside the pyproject.toml tool.poetry.extras section. Each has a corresponding list of dependencies whose versions are defined on the tool.poetry.dependencies section of the file.

Using the Development Flow

Developing tidy3d with poetry

poetry is an incredibly powerful tool for reproducible package development environments and dependency management. TODO add link.

If you are developing tidy3d, we recommend you work within the configured poetry environment defined by poetry.lock. The way to install this environment is simple:

cd tidy3d/
poetry install -E dev

This function will install the package with all the development dependencies automatically. This means you should be able to run any functionality that is possible with tidy3d reproducibly.

It is important to note the function above is equivalent to pip install tidy3d[dev], but by using poetry there is a guarantee of using the reproducible locked environment.

It is recommended to use poetry for package development. However, there are some cases where you might need to use an external virtual environment for some operations. There are a few workarounds where you can leverage the reproducibility of the poetry managed environment with the freedom of a standard virtual environment. There are a few more instructions and explanations in the poetry env docs . F See the following example:

mamba create -n tidy3denv python==3.10 # create venv with mamba
mamba activate tidy3denv # activate the venv
poetry env use python # using the mamba venv python now
poetry env info # verify the venvs used by poetry and mamba
cd anywhere
# you can use the python activated venv anywhere.

There are also other methodologies of implementing common dependencies management.

Common Utilities

There are a range of handy development functions that you might want to use to streamline your development experience.

Use Cases

Description	Caveats	Command
Benchmark timing import of tidy3d	Verify the available timing tests by running the command without any arguments.	poetry run tidy3d develop benchmark-timing-operations -c <timing_command>
Build documentation on reproducible environment		poetry run tidy3d develop build-docs
Build documentation with latest remote notebooks	It is defaulted to the develop branch of the tidy3d-notebooks repository.	poetry run tidy3d develop build-docs-remote-notebooks
Complete notebooks + base testing of the tidy3d	Make sure you have the notebooks downloaded.	poetry run tidy3d develop test-all
Dual snapshot between the tidy3d and notebooks source and submodule repository.	Make sure you are on the correct git branches you wish to commit to on both repositories, and all non-git-ignored files will be added to the commit.	tidy3d develop commit <your message>
Interactively convert all markdown files to rst (replacement for m2r2)		poetry run tidy3d develop convert-all-markdown-to-rst
Running pytest commands inside the poetry environment.	Make sure you have already installed tidy3d in poetry and you are in the root directory.	poetry run pytest
Run coverage testing from the poetry environment.		poetry run coverage run -m pytest
Standard testing of the tidy3d frontend	Make sure you have already installed tidy3d in poetry and you are in the root directory.	poetry run tidy3d develop test-base
Using tidy3d develop commands inside the poetry environment.	Make sure you have already installed tidy3d in poetry	poetry run tidy3d develop <your command>
Update lockfile after updating a dependency in pyproject.toml	Remember to install after this command.	poetry lock
Update and replace all the docstrings in the codebase between versions		poetry run tidy3d develop replace-in-files

Documentation

Assuming you already have poetry and the tidy3d develop commands installed (see the instructions if not), then building the documentation is easy:

poetry run tidy3d develop build-docs

The output of the build will be in _docs/ and you can view it by opening _docs/index.html in your browser. You might just have to click the index.html file to open it in your browser within a File Explorer.

Under docs/_static/css we can find custom.css which the color themes custom to Flexcompute can be found.

This process is self-contained in tidy3d-notebooks.

Make sure to add a link to the notebook in tidy3d-notebooks/docs/* directory in a relevant file.

Then you have to commit to either the develop branch or your custom one. However, the important thing to understand is that the submodule in docs/notebooks has a state that is also committed. This means that when you or any tool clones this directory, then the state and mapped branch/commit of the submodule will be the one that was committed. However, you have to be careful that when your commit gets merged the commit of the tidy3d-notebooks submodule is also pointing to the latest develop branch and not any local branch in which you have been developing. Otherwise, the documentation will be built with your local branch, and not the published branch.

This submodule commit process can be done by running git add docs/notebooks and then committing the change.

If you want to locally develop notebooks in tidy3d/docs/notebooks then just use that submodule as your main development repository and commit to your local branch. Then when you are ready to publish, just make sure to commit the submodule to the latest develop branch. You can then build the documentation locally easily using this approach before it is published.

The tidy3d develop suite includes a utility command replace-in-files, which is designed to recursively find and replace strings in files within a specified directory. This functionality is particularly useful for updating docstrings across the codebase when there are changes in the API, ensuring that the documentation remains consistent with multiple version updates. This is useful when updating the API and you want to update the docstrings to reflect the changes from multiple versions.

Example usage:

poetry run tidy3d develop replace-in-files -d ./ -j ./docs/versions/test_replace_in_files.json -v 0.18.0 --dry-run True

Command Details

• Name: replace-in-files

• Description: Recursively finds and replaces strings in files based on a JSON configuration.

• Options: - --directory or -d: Specifies the directory to process. Defaults to the current directory if not provided. - --json-dictionary or -j: Path to a JSON file containing the mapping of strings to be replaced. - --selected-version or -v: Specifies the version to select from the JSON file. - --dry-run: Executes the command in a dry run mode without making actual changes.

The JSON file should contain a dictionary where keys are version numbers and values are dictionaries of strings to find and their replacements.

Example JSON structure:

{
  "0.18.0": {
    "tidy3d.someuniquestringa": "tidy3d.someuniquestring2",
    "tidy3d.someuniquestringb": "tidy3d.someuniquestring2",
    "tidy3d.someuniquestringc": "tidy3d.someuniquestring2"
  }
}

The command can be executed using the poetry run command. It requires specifying the directory, JSON dictionary, and the selected version. The --dry-run option allows you to preview changes without applying them.

Example Command

poetry run tidy3d develop replace-in-files -d ./ -j ./docs/versions/test_replace_in_files.json -v 0.18.0 --dry-run True

This example will process files in the current directory (./), using the replacement rules specified in test_replace_in_files.json for version 0.18.0. The --dry-run flag set to True ensures that changes are not actually applied, allowing for a safe preview of potential modifications.

• The sphinx warnings are OK as long as the build occurs, errors will cause the crash the build.

• Make sure all your internal API references start with tidy3d.<your_reference>

• In notebooks, always have absolute links, otherwise the links will break when the user downloads them.

Recommendations

tidy3d is a pretty big project already, and will get bigger. We want to optimise the performance of the codebase throughout the multiple operations that we perform.

We want to improve the speed of the project import and there are a few techniques to do this which are inherent to the way we write our code.

• For profiling the package distribution in the import

• In terms of speeding our operations

We have already begun facing these type of code-speed issues as first raised here, here

So when we import dependencies inside our code-base in particular where these are used, we will try to do the following:

from mypackage import just_what_I_need

instead of

import mypackage

This is because the latter will import the entire package, which is not necessary and will slow down the code.

If you look within pyproject.toml, it is possible to see that we have different packages relating to different functionalities that are optional.

Some examples from these are [vtk, jax, trimesh, gdstk, gdspy] etc. What we want to do is improve the import speed of the core-package in order to minimise small core operations. As we scale into a bigger package, decoupling these type of imports from the total pacakge import is essential.

We want to make the tidy3d package be as light as possible for a given set of operations. As such, it is important to understand exactly where a given set of operations is expending computational power.

We have a set of utilties to verify this.

poetry run tidy3d develop benchmark-timing-operations

Release Flow

Currently most of our release development flow is made under the latest pre/* branch under the main frontend tidy3d repository. You want to fork from this latest branch to develop your feature in order for it to be included under that release.

We are using a variation of the gitflow workflow - so this is the first thing to familiarize yourselves with. The splitting of branches into main, develop and separate feature branches is as explained there. Most importantly, all contributions should happen through a PR from a feature branch into the ``develop`` branch.

The extra step that we have in our workflow is to always rebase and merge instead of simply merge branches. This has the advantage of avoiding a mess of crossing paths and keeps the history clean, but it does require a little more work. As an extra advantage, once you get the hang of rebasing it also becomes a very useful tool to prune your commits and write more meaningful commit messages when you’re done with the work. The main purpose of this page is to give an example of the workflow.

for more information on the difference between rebasing vs merging, see this article.

The first thing to do when starting a new batch of work is to start from a clean branch on your machine.

 # from the main tidy3d frontend repo
git checkout develop
git pull origin develop
git checkout -b my_name/new_feature

Before rebasing, you should make sure you have the latest version of develop, in case other work has been merged meanwhile.

git checkout develop
git pull origin develop
git checkout my_name/new_feature
git rebase -i develop

This will now open an editor that will allow you to edit the commits in the feature branch. There is plenty of explanations of the various things you can do.

Most probably, you just want to squash some of your commits. The first commit cannot be squashed - later commits get squashed into previous commits.

Once you save the file and close it, a new file will open giving you a chance to edit the commit message of the new, squashed commit, to your liking. Once you save that file too and close it, the rebasing should happen.

NB: The rebase may not work if there were conflicts with current develop. Ideally we should avoid that by making sure that two people are never working on the same part of the code. When it happens, you can try to resolve the conflicts, or git rebase --abort if you want to take a step back and think about it.

Finally, you now need to force push your branch to origin, since the rebasing has changed its history.

git push -f origin my_name/new_feature

After this, you can notify Momchil that the branch is ready to to be merged. In the comment you can optionally also say things like “Fixes #34”. This will then automatically link that PR to the particular issue, and automatically close the issue.

This can be repeated as often as needed. In the end, you may end up with a number of commits. We don’t enforce a single commit per feature, but it makes the most sense if the feature is small. If the feature is big and contains multiple meaningful commits, that is OK. In any case, rebasing allows you to clean everything up.

NB: Only do this once you feel like you are fully done with that feature, i.e. all PR comments have been addressed, etc. This is not critical, but is nicer to only rebase in the end so as not to muddle up the PR discussion when you force push the new branch (see below).

Documentation

When a new release is created, it is necessary to create a “mirror” branch under the tidy3d-docs repository in order for it to build. If this is not there an error such as the following might appear under the sync-readthedocs-repo Github action. You will also need to do this for full releases.

/usr/bin/git worktree remove github-pages-deploy-action-temp-deployment-folder --force
Error: The deploy step encountered an error: There was an error creating the worktree: The process '/usr/bin/git' failed with exit code 128 ❌ ❌
Notice: Deployment failed! ❌

Coverage

All notebooks are now developed under the tidy3d-notebooks, and you can also develop this submodule under tidy3d/notebooks. Note that the submodule is linked to the develop branch of tidy3d-notebooks.

Say, you have done some changes onto the repository in tidy3d-notebooks and propagated them to the remote branch, you can run the following command:

poetry run tidy3d develop build-docs-from-remote-notebooks

This command will pull the latest changes onto your notebook submodule and build the documentation.