Docker Development Environment

Docker Development Environment#

This guide will get you up and running with Tidy3D Python client development using Docker. The Docker workflow provides an isolated, reproducible development environment without affecting your system packages.

Overview #

The Docker development environment provides:

Isolation: Clean environment separate from your system
Reproducibility: Everyone uses the same base environment
Simplicity: No need to manage system dependencies
Flexibility: Easy to reset and start fresh

What’s included:

Python 3.11 with uv package manager
Poetry for dependency management
AWS CLI for CodeArtifact access
Neovim, git, pandoc for documentation
Jupyter Lab for notebook development

Prerequisites #

System Requirements #

Operating System: Ubuntu 22.04+, macOS, or Windows with WSL2
Docker Engine: Installed and running
Git: With SSH keys configured for GitHub
Disk Space: ~2GB for Docker image + container

Quick Check#

# Check Docker is installed and running
docker --version
docker ps

# Check Git SSH access
ssh -T git@github.com

Installing Docker #

If you don’t have Docker installed:

Ubuntu

# Install Docker Engine (NOT from snap)
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Add your user to docker group
sudo usermod -aG docker $USER

# Reboot to apply group changes
sudo reboot

# Verify installation
docker run hello-world

See: Docker Ubuntu Install Guide

macOS

Download and install Docker Desktop from docker.com

Windows

Install Docker Desktop with WSL2 backend from docker.com

Access Requirements #

GitHub: Access to flexcompute/tidy3d
Tidy3D Account: Sign up at tidy3d.simulation.cloud
API Key: Get yours from account page

Quick Start Guide #

This guide is for anyone setting up the development environment for the first time or migrating from a version after 2.9.0 when the dev.Dockerfile image was enabled.

Step 1: Clone the Repository #

# Clone the repository and navigate into it
git clone https://github.com/flexcompute/tidy3d.git
cd tidy3d

# Checkout develop branch or your feature branch
git checkout develop

Step 2: Build the Docker Image #

Build the development Docker image (~5 minutes):

# Build the development Docker image
docker build -t tidy3d-python-client-dev -f dev.Dockerfile .

The image includes:

Debian base with Python 3.11
uv for fast package installation
Poetry for dependency management
AWS CLI, git, pandoc, neovim
flexdaemon user (UID 1000, GID 1000)

Step 3: Create the Development Container #

Create a persistent container with your local code mounted:

# Create a persistent container with your local code mounted
# This also maps port 8888 for tools like Jupyter Lab
docker container create --name=tidy3d_python_client_dev \
    --userns=host \
    -p 8888:8888 \
    -v .:/home/flexdaemon/tidy3d \
    tidy3d-python-client-dev

Flags explained:

--name: Container name for easy reference
--userns=host: Use host user namespace (prevents permission issues)
-p 8888:8888: Expose port 8888 for Jupyter Lab
-v .:/home/flexdaemon/tidy3d: Mount current directory into container

Step 4: Start the Container #

# Start the container
docker start tidy3d_python_client_dev

Step 5: Open an Interactive Shell #

# Open an interactive shell inside the running container
docker exec -it tidy3d_python_client_dev /bin/bash

You’re now inside the container at /home/flexdaemon!

Step 6: Set Up Python Environment #

Inside the container, create a virtual environment and install the package:

# (Inside Container) Navigate to the mounted tidy3d directory
cd tidy3d

# (Inside Container) Create a virtual environment and activate it
uv venv -p 3.11
source .venv/bin/activate

# (Inside Container) Install the project in editable mode with dev dependencies
uv pip install -e .[dev]

Note

The virtual environment is created inside the mounted directory, so it persists between container restarts.

Step 7: Configure API Key and Run Tests #

Configure your API key and verify everything is working:

# (Inside Container) Configure your API key
tidy3d configure --apikey=YOUR_API_KEY_HERE

# (Inside Container) Test the configuration
python -c "import tidy3d.web as web; web.test()"

# (Inside Container) Run tests
pytest tests/ -v

If tests pass, you’re ready to develop!

Development Workflow #

Daily Workflow #

Start the container (if not running):

# On host
docker start tidy3d_python_client_dev
docker exec -it tidy3d_python_client_dev bash

Activate virtual environment (inside container):
```
cd tidy3d
source .venv/bin/activate
```
Make changes to the code on your host machine using your favorite editor (VSCode, PyCharm, etc.)

Changes are immediately reflected in the container since the directory is mounted.

Run tests (inside container):

# Run all tests
pytest

# Run specific test file
pytest tests/test_simulation.py

# Run with verbose output
pytest -v -s

Commit changes (on host or in container):

git add .
git commit -m "Your descriptive message"
git push origin your-feature-branch

Multiple Terminal Sessions #

You can open multiple terminals into the same container:

# Terminal 1 (host)
docker exec -it tidy3d_python_client_dev bash

# Terminal 2 (host)
docker exec -it tidy3d_python_client_dev bash

# Terminal 3 (host)
docker exec -it tidy3d_python_client_dev bash

Each terminal shares the same filesystem and running processes.

Running Tests #

# Inside container with venv activated

# Run all tests
pytest

# Run tests in parallel (faster)
pytest -n auto

# Run specific test module
pytest tests/test_components.py

# Run tests matching pattern
pytest -k "test_simulation"

# Run with coverage
pytest --cov=tidy3d --cov-report=html
# View coverage: python -m http.server 8000 -d htmlcov

Code Quality Checks #

# Inside container with venv activated

# Format code with ruff
ruff format .

# Check for linting issues
ruff check .

# Auto-fix linting issues
ruff check --fix .

# Run type checking
mypy tidy3d

Building Documentation #

# Inside container with venv activated
cd docs

# Build HTML documentation
make clean
make html

# View the docs
cd _build/html
python -m http.server 8000
# Open http://localhost:8000 in your host browser

Managing the Container #

Start and Stop #

# Start container
docker start tidy3d_python_client_dev

# Stop container (preserves state)
docker stop tidy3d_python_client_dev

# Restart container
docker restart tidy3d_python_client_dev

Checking Status #

# Check if container is running
docker ps

# Check all containers (including stopped)
docker ps -a

# View container logs
docker logs tidy3d_python_client_dev

# Follow container logs (live)
docker logs -f tidy3d_python_client_dev

Removing and Recreating #

# Stop and remove container
docker stop tidy3d_python_client_dev
docker container rm tidy3d_python_client_dev

# Recreate container (preserves code on host)
docker container create --name=tidy3d_python_client_dev \
    --userns=host \
    -p 8888:8888 \
    -v .:/home/flexdaemon/tidy3d \
    tidy3d-python-client-dev

Note

Your code changes are safe! They’re stored on your host in the mounted directory.

Rebuilding the Image #

If dev.Dockerfile changes or you want to update base dependencies:

# Rebuild the image
docker build -t tidy3d-python-client-dev -f dev.Dockerfile .

# Remove old container and create new one
docker stop tidy3d_python_client_dev
docker container rm tidy3d_python_client_dev
docker container create --name=tidy3d_python_client_dev \
    --userns=host \
    -p 8888:8888 \
    -v .:/home/flexdaemon/tidy3d \
    tidy3d-python-client-dev

Advanced Usage #

Running Jupyter Lab #

Start Jupyter Lab inside the container to work with notebooks:

# Inside container with venv activated
cd tidy3d
jupyter lab . --ip=0.0.0.0 --port=8888

Jupyter Lab will print a URL like:

http://127.0.0.1:8888/lab?token=abc123def456...

Copy this complete URL and paste it into your web browser (like Firefox) on your host machine to begin your session.

Note

Port 8888 is forwarded to your host via the -p 8888:8888 flag used when creating the container.

Running GitHub Actions Locally #

To run GitHub Actions locally, install the act extension:

gh extension install nektos/gh-act

You can then run remote-tests locally using your dev container image:

gh act --pull=false \
    -W .github/workflows/tidy3d-python-client-tests.yml \
    -P "ubuntu-latest=tidy3d-python-client-dev:latest" \
    --input remote_tests=true \
    "workflow_dispatch"

Note

The image name must match the one you built: tidy3d-python-client-dev:latest

VSCode Integration #

Attach VSCode to the running container for a full IDE experience:

Install Extensions

In VSCode, install:

Remote - Containers (ms-vscode-remote.remote-containers)
Docker (ms-azuretools.vscode-docker)

Attach to Container

Start your container:
```
docker start tidy3d_python_client_dev
```
In VSCode:
- Open Command Palette (Cmd/Ctrl+Shift+P)
- Type “Remote-Containers: Attach to Running Container”
- Select tidy3d_python_client_dev
VSCode will open a new window connected to the container
Open folder: /home/flexdaemon/tidy3d
Select Python interpreter: .venv/bin/python

Now you can:

Edit code with full IntelliSense
Run tests from VSCode test explorer
Debug Python code with breakpoints
Use integrated terminal (already inside container)

Optional: AWS CodeArtifact Access (Internal Developers Only)#

Note

Who needs this?

CodeArtifact hosts internal packages like tidy3d-extras needed for:

Integration tests in CI/CD
Backend-dependent development work
Full-stack Tidy3D development

External contributors: You can skip this entire section. The open-source tidy3d client works without CodeArtifact.

Warning

Security Policy: We use AWS SSO for authentication, which requires daily login for security compliance. This is a simple 2-step process per development session:

aws sso login - Opens browser for authentication (~30 seconds)
poetry aws-login - Automatic token management

Why CodeArtifact?#

Q: Why can’t we use PyPI?

A: Internal packages like tidy3d-extras contain proprietary code that cannot be published publicly.

Q: Why SSO with daily expiration?

A: Flexcompute security policy requires time-limited credentials. Long-lived tokens pose security risks.

Q: What if I don’t want to deal with this?

A: Use the Docker development environment, which matches our CI setup exactly. Or if you’re only working on the open-source client, you don’t need CodeArtifact at all.

Supported Path: Poetry with AWS Login Plugin #

The development container includes the poetry-codeartifact-login plugin for simplified authentication.

Note

For non-Docker users: If you’re not using the Docker container, you’ll need to install the plugin first:

# Install the Poetry plugin (one-time, outside Docker)
poetry self add poetry-codeartifact-login

# Also ensure AWS CLI v2 is installed on your system
aws --version  # Should show version 2.x

Docker users can skip this step - the plugin is pre-installed in the dev container.

One-Time Setup

Inside the container (or on your local machine if not using Docker), configure AWS SSO (only needed once):

# Configure AWS SSO
aws configure sso

Enter the following when prompted:

Prompt	Value
SSO session name	`flexcompute-pypi`
SSO start URL	`https://d-9067bfae6e.awsapps.com/start/#`
SSO region	`us-east-1`
Account	`625554095313`
Role	`codeartifact-readonly`
Profile name	`flexcompute-pypi`

Daily Authentication (each development session)

Each day when you start development, run these two commands:

# Step 1: Login to AWS SSO (opens browser on host for authentication)
aws sso login --profile flexcompute-pypi

# Step 2: Configure Poetry with CodeArtifact
poetry aws-login codeartifact --profile flexcompute-pypi

# That's it! Now you can install internal packages
poetry install -E extras

Note

How it works: The poetry aws-login command automatically:

Generates a short-lived CodeArtifact authentication token
Configures Poetry to use the token for the codeartifact source
Stores credentials securely (no manual token passing required)

The plugin handles all token management internally, so you never see or copy tokens manually.

Persisting AWS Credentials #

To avoid re-configuring AWS SSO each time you recreate the container, mount your AWS credentials:

# Recreate container with AWS credentials mounted
docker container create --name=tidy3d_python_client_dev \
    --userns=host \
    -p 8888:8888 \
    -v .:/home/flexdaemon/tidy3d \
    -v ~/.aws:/home/flexdaemon/.aws \
    tidy3d-python-client-dev

This way, you only need to run aws configure sso once on your host machine.

Alternative Package Managers (Self-Support)#

If you prefer pip, uv, or other package managers instead of Poetry, you’ll need to configure CodeArtifact authentication yourself.

The officially supported development path uses Poetry. For other package managers:

See AWS CodeArtifact pip documentation
Use aws codeartifact login --tool pip for manual setup
You are responsible for configuration and troubleshooting

Alternatively, use the Docker development environment which has everything pre-configured.

Quick Reference #

Command Cheatsheet #

Container Management:

# Build image
docker build -t tidy3d-python-client-dev -f dev.Dockerfile .

# Create container
docker container create --name=tidy3d_python_client_dev \
    --userns=host -p 8888:8888 \
    -v .:/home/flexdaemon/tidy3d \
    tidy3d-python-client-dev

# Start/stop container
docker start tidy3d_python_client_dev
docker stop tidy3d_python_client_dev

# Enter container
docker exec -it tidy3d_python_client_dev bash

Inside Container:

# Setup environment
cd tidy3d
uv venv -p 3.11
source .venv/bin/activate
uv pip install -e .[dev]

# Testing
pytest                          # All tests
pytest -n auto                  # Parallel tests
pytest tests/test_file.py       # Specific file

# Code quality
ruff format .                   # Format code
ruff check .                    # Lint code
mypy tidy3d                     # Type check

# Documentation
cd docs && make html            # Build docs

# Jupyter
jupyter lab . --ip=0.0.0.0      # Start Jupyter Lab

# CodeArtifact (internal developers only)
aws sso login --profile flexcompute-pypi                    # Daily: Login to AWS
poetry aws-login codeartifact --profile flexcompute-pypi    # Daily: Configure Poetry

Directory Structure #

Host:
~/flexcompute/tidy3d/
├── tidy3d/              # Source code (mounted to container)
├── tests/               # Test suite (mounted)
├── docs/                # Documentation (mounted)
├── .venv/               # Virtual env (created inside, persists)
├── pyproject.toml       # Poetry config (mounted)
└── dev.Dockerfile       # Docker image definition

Container:
/home/flexdaemon/
├── tidy3d/              # Mounted from host
│   ├── .venv/           # Your virtual environment
│   ├── tidy3d/          # Package source
│   └── tests/           # Tests
└── .aws/                # AWS credentials (if mounted)

Troubleshooting #

Container Issues #

Problem: docker: permission denied

Solution: Add your user to the docker group:

sudo usermod -aG docker $USER
# Logout and login (or reboot)

Problem: Container exits immediately

Solution: Check logs for errors:

docker logs tidy3d_python_client_dev

# Try running interactively to debug
docker run --rm -it tidy3d-python-client-dev bash

Problem: Port 8888 already in use

Solution: Use a different port:

docker container create --name=tidy3d_python_client_dev \
    --userns=host \
    -p 8889:8888 \
    -v .:/home/flexdaemon/tidy3d \
    tidy3d-python-client-dev

# Access Jupyter at http://localhost:8889

Permission Issues #

Problem: Cannot edit files, permission denied

Solution: Ensure container uses host user namespace:

# Container must be created with --userns=host flag
docker container create --name=tidy3d_python_client_dev \
    --userns=host \
    -v .:/home/flexdaemon/tidy3d \
    tidy3d-python-client-dev

Problem: flexdaemon user has wrong UID

Solution: The dev.Dockerfile creates user with UID 1000. If your host user has a different UID, files may have permission issues. Check your host UID:

# On host
id -u  # Should output 1000 for best compatibility

# If different, you can modify the Dockerfile or change file ownership

Virtual Environment Issues #

Problem: uv venv command not found

Solution: uv is pre-installed in the image. Verify:

which uv
uv --version

# If missing, rebuild the image

Problem: Virtual environment activation fails

Solution: Make sure you’re in the right directory:

cd /home/flexdaemon/tidy3d
ls .venv  # Should exist after 'uv venv'
source .venv/bin/activate

Test Failures #

Problem: Tests fail with ModuleNotFoundError

Solution: Reinstall in editable mode:

# Inside container with venv activated
cd /home/flexdaemon/tidy3d
uv pip install -e .[dev]

Problem: Import errors for optional dependencies

Solution: Install all extras:

uv pip install -e .[dev,jax,vtk,trimesh,gdstk]

CodeArtifact Authentication Issues #

Problem: poetry install -E extras fails with “Unable to find tidy3d-extras” or authentication error

Solution: Verify SSO session is active and re-authenticate:

# Check if SSO session is valid
aws sts get-caller-identity --profile flexcompute-pypi

# If expired or fails, re-authenticate (2-step process)
aws sso login --profile flexcompute-pypi
poetry aws-login codeartifact --profile flexcompute-pypi

# Retry installation
poetry install -E extras

Problem: aws sso login doesn’t work, times out, or fails

Solution: This is an AWS account access issue. Contact IT support or your team lead to verify:

Your AWS account is active
You have access to the correct AWS organization
Your SSO credentials are correct
You have the codeartifact-readonly role assigned

Problem: poetry aws-login command not found

Solution: The plugin is pre-installed in the dev container. If missing:

# Check if plugin is installed
poetry self show plugins

# Reinstall if needed
poetry self add poetry-codeartifact-login

Problem: I use pip/uv, not Poetry. How do I authenticate?

Solution: The officially supported path is Poetry with the poetry aws-login command.

For pip/uv users, you must self-support using AWS documentation.

Option 1 (for pip users):

aws sso login --profile flexcompute-pypi
aws codeartifact login --tool pip \
    --repository pypi-releases \
    --domain flexcompute \
    --domain-owner 625554095313 \
    --region us-east-1 \
    --profile flexcompute-pypi

Option 2 (not recommended):

CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token \
    --domain flexcompute \
    --domain-owner 625554095313 \
    --query authorizationToken \
    --output text \
    --profile flexcompute-pypi`

pip config set site.extra-index-url \
    https://aws:$CODEARTIFACT_AUTH_TOKEN@flexcompute-625554095313.d.codeartifact.us-east-1.amazonaws.com/pypi/pypi-releases/simple/

Note: Tokens expire after 12 hours, requiring daily re-authentication.

Alternatively, use the Docker development environment.

Problem: poetry lock or poetry update hangs

Solution: Clear Poetry cache and retry:

# Clear Poetry cache
poetry cache clear pypi --all
poetry cache clear codeartifact --all

# Retry operation
poetry update --lock

Jupyter Issues #

Problem: Cannot access Jupyter Lab from host browser

Solution: Ensure Jupyter is binding to 0.0.0.0:

# Inside container
jupyter lab . --ip=0.0.0.0 --port=8888

# Port must be forwarded when creating container
# Check with: docker inspect tidy3d_python_client_dev | grep PortBindings

Problem: Jupyter kernel dies or doesn’t start

Solution: Check virtual environment has ipykernel:

uv pip install ipykernel
python -m ipykernel install --user --name=tidy3d

Next Steps #

After completing this guide, you should have:

✅ Docker development container running ✅ Python environment set up with dependencies ✅ Tests passing successfully ✅ (Optional) Jupyter Lab accessible ✅ (Optional) VSCode attached to container ✅ (Optional) AWS CodeArtifact configured

Continue learning:

Using the Development Flow - Advanced development workflows
Documentation - Building and writing documentation
Recommendations - Best practices and code style
Release Flow - Release process and versioning

More Docker resources:

Welcome to Docker-based Tidy3D development!

Docker Development Environment

Contents

Docker Development Environment#

Quick Check#