An Introduction to the Development Flow

An Introduction to the Development Flow#

This page hopefully will get you started to develop Tidy3D.

TLDR:

Branch off of the target branch (usually develop or pre/x.x), work on your branch, and submit a PR when ready.
Use isolated development environments with uv.
Use ruff to lint and format code, and install the pre-commit hook via pre-commit install to automate this.
Document code using NumPy-style docstrings.
Write unit tests for new features and try to maintain high test coverage.

Understanding Virtual Environments#

Introduction#

In larger projects, it’s crucial to have a separate Python environment for each feature or branch you work on. This practice ensures isolation and reproducibility, simplifying testing and debugging by allowing issues to be traced back to specific environments. It also facilitates smoother integration and deployment processes, ensuring controlled and consistent development. Managing multiple environments might seem daunting, but it’s straightforward with the right tools. Follow the steps below to set up and manage your environments efficiently.

Benefits#

Isolation: Avoids conflicts between dependencies of different features.
Reproducibility: Each environment can be easily replicated.
Simplified Testing: Issues are contained within their respective environments.
Smooth Integration: Ensures features are developed in a consistent setting.

Prerequisites#

Make sure that you have uv installed. This can be done system-wide with pipx or within a conda environment. Note that we use conda only for setting up the interpreter (Python version) and uv, not for managing dependencies. Refer to the official development guide for detailed instructions:

https://docs.flexcompute.com/projects/tidy3d/en/stable/development/index.html#installation

Setting Up a New Environment#

Check out the branch:
```
git checkout branch
```

Set up the environment with conda (skip this step if you don’t use conda):

conda create -n branch_env python=3.11
conda activate branch_env
python --version

Install dependencies with uv:

uv sync --active --frozen --extra dev
uv run pre-commit install

Update the environment when switching to a different branch:
```
uv sync --active --frozen --extra dev
```

Multiple Folders or Worktrees#

If you have multiple folders (e.g., multiple clones or git worktrees), you will need to repeat the environment setup for each folder. Ensure that each folder has its own isolated environment.

By following these steps, you can maintain isolated and reproducible environments for each branch and feature, leading to a more efficient and error-free development process.

Using uv for package management#

What is uv#

uv is the package and environment tool used in this repository.

We use it to:

Resolve and lock dependencies.
Create reproducible development environments.
Run commands in the project environment.
Build distribution artifacts.

Why we use it in `tidy3d`#

Dependency metadata lives in one place: pyproject.toml.
uv.lock captures exact resolved versions for reproducible installs.
uv sync --frozen and uv run --frozen keep local and CI behavior aligned.

How to install it#

See the development installation guide in this documentation, or follow the official uv docs for installation and project workflows.

Project workflow in this repo#

Create/update the development environment:

uv sync --frozen --extra dev

Run commands inside the managed environment:

uv run --frozen pytest
uv run --frozen pre-commit run --all-files

Update locked dependencies after changing dependency metadata:

uv lock

Dependency metadata#

Dependencies are defined in pyproject.toml:

Runtime dependencies under [project.dependencies].
Optional dependency sets under [project.optional-dependencies].

Publishing/building#

Build local distribution artifacts:

uv build --sdist --wheel

For publish configuration and authentication details, refer to the uv publishing guide.

Code Quality Principles#

When writing a code snippet, remember the saying: “code is read more than written”. We want to maintain our code maintainable, readable and high quality.

Linting & Formatting#

To maintain code quality, we use Ruff as a linter and code formatter. A linter analyzes code to identify and flag potential errors, stylistic issues, and code that doesn’t adhere to defined standards (such as PEP8). A code formatter automatically restructures the code to ensure it is consistently styled and properly formatted, making it consistent across the code base.

Run ruff format to format all Python files:

uv run ruff format .

Run ruff check to check for style and other issues. Many common warnings can be automatically fixed with the --fix flag:

uv run ruff check tidy3d --fix

The configuration defining what ruff will correct lives in pyproject.toml under the [tool.ruff] section.

When submitting code, for tests to pass, ruff should give no warnings.

Documentation#

Document all code you write using NumPy-style docstrings.

Testing#

Here we will discuss how tests are defined and run in Tidy3d.

Unit Testing#

The tests live in tests/ directory.

We use pytest package for our testing.

To run all of the tests, call:

uv run pytest -rA tests

This command will trigger pytest to go through each file in tests/ called test*.py and run each function in that file with a name starting with test.

If all of these functions run without any exceptions being raised, the tests pass!

The specific configuration we use for pytest lives in the [tool.pytest.ini_options] section of pyproject.toml.

These tests are automatically run when code is submitted using GitHub Actions, which tests on Python 3.9 through 3.12 running on Ubuntu, MacOS, and Windows operating systems, as well as Flexcompute’s servers.

Note: The -rA flag is optional but produces output that is easily readable.

Note: You may notice warnings and errors in the pytest output, this is because many of the tests intentionally trigger these warnings and errors to ensure they occur in certain situations. The important information about the success of the test is printed out at the bottom of the pytest output for each file.

To get a code coverage report:

pip install pytest-cov

if not already installed

To run coverage tests with results printed to STDOUT:

pytest tests --cov-report term-missing --cov=tidy3d

To run coverage tests and get output as .html (more intuitive):

pytest tests --cov-report=html --cov=tidy3d
open htmlcov/index.html

Automated Testing#

We use GitHub Actions to perform these tests automatically and across different operating systems.

On commits, each of the pytest tests are run using Python (minimum supported version up to maximum supported version) installed on Ubuntu, MacOS, and Windows operating systems.

See the “actions” tab for details on previous tests and .github/workflows/run_tests.yml for the configuration and to see the specific tests run.

See this for more explanation.

Other Tests#

There are additional tests in both the documentation and our private backend code. The same practices outlined here apply to those tests.

More Resources on Testing#

A useful explanation for those curious to learn more about the reasoning behind these decisions:

https://www.youtube.com/watch?v=DhUpxWjOhME <https://www.youtube.com/watch?v=DhUpxWjOhME>

`tidy3d` Project Structure#

As of tidy3d>=2.6, the frontend has been restructured to improve the development cycle. The project directories follow the following structure, which is derived from some recommended Python project architecture guides. This is a handy structure because many tools, such as sphinx, integrate quite well with this type of project layout.

docs/
    # sphinx rst files
    ...
    notebooks/
        # Git submodule repository
        # Checks out github.com/flexcompute/tidy3d-notebooks
    faq/
        # Git submodule repository
        # Checks out github.com/flexcompute/tidy3d-faq
tests/
    # pytest source and docs
    # pytest notebooks
scripts/
    # useful handy scripts
tidy3d/
    # python source code
...
pyproject.toml # python packaging
uv.lock # environment management

It is important to note the new tools we are using to manage our development environment and workflow.

uv
pipx

Important Branches#

We currently have three main branches that have to be kept track of when creating a release, each with different functionality.

Project Branches#
Name	Description	Caveats
`latest`	Contains the latest version of the docs. Version release tags are created from this branch.	Feature PRs should not be made to this branch as will cause divergence. Only in important documentation patches.
`develop`	Contains the “staging” version of the project. Patch versions and development occurs from these branches.	Docs PRs that are non-crucial for the current version should be made to this branch.
`pre/^*`	Contains the next version of the project.	Documentation and source code that will only go live in the next version should be updated here.

Sometimes, hopefully infrequently, the latest and develop branches might diverge. It is important to bring them back together. However, what happens if we rebase develop into latest?

It could be argued that all the commits in the latest branch should have constructed within the develop branch. Then, there is the question if we want to maintain the commit history accordingly. If we just want to maintain the content, then rebasing and fixing up all the branches works fine. The problem with a merge commit is that it inserts the commits at the historical period in which they were made, rather than the commit period in which we desire to add them. Hence, it makes sense to merge the develop and latest branches in order to maintain the same history, assuming the commits should in theory have been in both branches.