Visitar URL original
Add PyTorch🔥 documentation by n-poulsen · Pull Request #2605 · DeepLabCut/DeepLabCut · GitHub
Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
9aa9647
added initial pytorch docs
Jun 7, 2024
76cd889
Update _toc.yml
MMathisLab Jun 7, 2024
f15a09a
Update user_guide.md
MMathisLab Jun 7, 2024
f448dc9
Update architectures.md
MMathisLab Jun 7, 2024
d598a56
Update labelling.md
MMathisLab Jun 7, 2024
b9195b8
Update labelling.md
MMathisLab Jun 7, 2024
4bc8aa1
Update codespell.yml
MMathisLab Jun 7, 2024
fecf7e5
Merge branch 'main' into pytorch_docs
MMathisLab Jun 7, 2024
0a2a1c5
Update README.md
MMathisLab Jun 7, 2024
a4e050f
Update installation.md
MMathisLab Jun 7, 2024
9cdb9f1
Delete conda-environments/DEEPLABCUT_M1.yaml
MMathisLab Jun 7, 2024
198a34e
Update DEEPLABCUT.yaml
MMathisLab Jun 7, 2024
e3a5dfa
Update installation.md
MMathisLab Jun 7, 2024
5a774b9
Update maDLC_UserGuide.md
MMathisLab Jun 7, 2024
9d71c9b
Update ModelZoo.md
MMathisLab Jun 8, 2024
3a5a5a1
Update standardDeepLabCut_UserGuide.md
MMathisLab Jun 8, 2024
a7c49eb
Update standardDeepLabCut_UserGuide.md
MMathisLab Jun 8, 2024
0aa9e28
Update standardDeepLabCut_UserGuide.md
MMathisLab Jun 8, 2024
4bf88d2
Update Overviewof3D.md
MMathisLab Jun 8, 2024
665f2df
Update PROJECT_GUI.md
MMathisLab Jun 8, 2024
a388d28
Update UseOverviewGuide.md
MMathisLab Jun 8, 2024
c0028c0
Update beginners-guide.md
MMathisLab Jun 8, 2024
329ce71
Update intro.md
MMathisLab Jun 8, 2024
c1aaa84
Update architectures.md
MMathisLab Jun 8, 2024
16e4386
Update architectures.md
MMathisLab Jun 8, 2024
894e3b4
Merge branch 'main' into pytorch_docs
MMathisLab Jun 8, 2024
680f353
Create course.md
MMathisLab Jun 8, 2024
09bd63e
Update _toc.yml
MMathisLab Jun 8, 2024
d29e7c4
Update manage-project.md
MMathisLab Jun 8, 2024
34a7b5a
Update beginners-guide.md
MMathisLab Jun 8, 2024
20d0bca
Update manage-project.md
MMathisLab Jun 8, 2024
f2336cc
Update _toc.yml
MMathisLab Jun 8, 2024
7a3e0ea
Rename labelling.md to labeling.md
MMathisLab Jun 8, 2024
4802d4f
Update Training-Evaluation.md
MMathisLab Jun 8, 2024
1de7232
Update video-analysis.md
MMathisLab Jun 8, 2024
77ea994
Update architectures.md
MMathisLab Jun 8, 2024
73cd31a
Update architectures.md
MMathisLab Jun 8, 2024
8483175
Update AUTHORS
MMathisLab Jun 8, 2024
518909a
Update architectures.md
MMathisLab Jun 8, 2024
82132e1
doc clean up, tested local build
MMathisLab Jun 8, 2024
0197ba4
Update beginners-guide.md
MMathisLab Jun 8, 2024
e91ee09
Update ModelZoo.md
MMathisLab Jun 8, 2024
daf2f87
Update ModelZoo.md
MMathisLab Jun 8, 2024
2a8949b
Update ModelZoo.md
MMathisLab Jun 8, 2024
9a1f28a
Update ModelZoo.md
MMathisLab Jun 8, 2024
02d71df
Update README.md
MMathisLab Jun 8, 2024
0499825
Update ModelZoo.md
MMathisLab Jun 8, 2024
83db9af
Update ModelZoo.md
MMathisLab Jun 8, 2024
14bbee0
Update AUTHORS
MMathisLab Jun 10, 2024
aa3c82d
Update README.md
MMathisLab Jun 11, 2024
fec76d8
Merge branch 'main' into pytorch_docs
MMathisLab Jun 11, 2024
5927e26
Update installation.md
MMathisLab Jun 12, 2024
3cfb0a8
Update DEEPLABCUT.yaml
MMathisLab Jun 12, 2024
bd212aa
Update DEEPLABCUT.yaml
MMathisLab Jun 12, 2024
a6c9318
Update AUTHORS
MMathisLab Jun 13, 2024
63898d7
fixed typos
Jun 13, 2024
7a33b74
Update python-package.yml
MMathisLab Jun 13, 2024
62ca76b
Update README.md
MMathisLab Jun 14, 2024
7c0a5c1
Update beginners-guide.md
MMathisLab Jun 14, 2024
8dfd1e6
Update user_guide.md
MMathisLab Jun 14, 2024
f6ba9d3
m1 for tf only
MMathisLab Jun 14, 2024
093c424
Merge branch 'main' into pytorch_docs
MMathisLab Jun 14, 2024
aeccccb
Update README.md
MMathisLab Jun 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update user_guide.md
  • Loading branch information
MMathisLab authored Jun 7, 2024
commit f15a09a32d76839c515a284ae44cbb21d1954b3d
61 changes: 33 additions & 28 deletions docs/pytorch/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,16 @@

## Using DeepLabCut 3.0

PyTorch models can be trained on any DeepLabCut project. Simply add a new key to your
**DeepLabCut 3.0 keeps the same high-level API that you know, but has a full new PyTorch backend.
Moreover, it is a rewrite that more developer friendly, more powerful, and built for modern deep
learning-based computer vision applications.**

**NOTE**🔥: We suggest that if you're just starting with DeepLabCut you start with the PyTorch backend.
You will easily know which "engine" you are using by looking at the main `config.yaml` file, or top right corner in the GUI.
If you have DeepLabCut projects in TensorFlow, we've got you covered too: you can seamlessly switch to train your already labeled data
by simply switching the engine (and thereby also compare performance). In short, expect a boost 🔥.

In short, PyTorch models can be trained in any DeepLabCut project. If you have a project already made, simply add a new key to your
project `config.yaml` file specifying `engine: pytorch`. Then any new training dataset
that will be created will be a PyTorch model (see
[Creating Shuffles and Model Configuration](#Creating-Shuffles-and-Model-Configuration))
Expand All @@ -14,33 +23,46 @@ set `engine: tensorflow`.

You can use the GUI to train DeepLabCut projects. However, you cannot switch between
PyTorch and Tensorflow models while using the GUI. If you have set your engine to
`pytorch`, then the GUI will only offer the creation of Pytorch shuffles.
`pytorch`, then the GUI will only offer the creation of PyTorch shuffles.

You can create `tensorflow` shuffles and train them again by setting the
`engine: tensorflow` key and either re-loading the project or closing the GUI entirely
and opening the project again.
`engine: tensorflow` in the top right corner of the GUI.

## Major changes

### From iterations to epochs

Pytorch models in DeepLabCut 3.0 are trained for a set number of epochs, instead of a
maximum number of iterations. An epoch is a single pass through the training dataset,
Pytorch models in DeepLabCut 3.0 are trained for a set number of `epochs`, instead of a
maximum number of `iterations`. An epoch is a single pass through the training dataset,
which means your model has seen each training image exactly once.

So if you have 64 training images for your network, an epoch is 64 iterations with batch
- So if you have 64 training images for your network, an epoch is 64 iterations with batch
size 1 (or 32 iterations with batch size 2, 16 with batch size 4, etc.).

## API

### Development state
### Creating Shuffles and Model Configuration

You can configure models using the `pytorch_config.yaml` file, as described
[here](dlc3-pytorch-config). You can use the same methods to create new shuffles in
DeepLabCut 3.0 as you did for Tensorflow models (`deeplabcut.create_training_dataset`
and `deeplabcut.create_training_model_comparison`).

More information about the different PyTorch model architectures available in DeepLabCut
is available [here](dlc3-pytorch-config). You can see a list of supported
architectures/variants by using:

```python
from deeplabcut.pose_estimation_pytorch import available_models
print(available_models())
```

### Development State and Road Map 🚧

The table below describes the DeepLabCut API methods that have been implemented,
as well as indications which options are not yet implemented, and which parameters
are not valid for the DLC 3.0 API.

You can find all DLC 3.0 API methods and the parameters they can be called with in


| API Method | Implemented | Parameters not yet implemented | Parameters invalid for pytorch |
|--------------------------------|:-----------:|-------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------|
Expand All @@ -50,28 +72,11 @@ You can find all DLC 3.0 API methods and the parameters they can be called with
| `return_evaluate_network_data` | 🔴 | | `TFGPUinference`, `allow_growth` |
| `analyze_videos` | 🟢 | `use_shelve`, `save_as_csv`, `in_random_order`, `batchsize`, `cropping`, `dynamic`, `robust_nframes`, `n_tracks`, `calibrate` | |
| `create_tracking_dataset` | 🔴 | | |
| `analyze_time_lapse_frames` | 🔴 | | |
| `analyze_time_lapse_frames` | 🟠 | the name has changed to `analyze_images` to better reflect what it actually does (no video needed) | |
| `convert_detections2tracklets` | 🟢 | `greedy`, `calibrate`, `window_size` | |
| `extract_maps` | 🔴 | | |
| `visualize_scoremaps` | 🔴 | | |
| `visualize_locrefs` | 🔴 | | |
| `visualize_paf` | 🔴 | | |
| `extract_save_all_maps` | 🔴 | | |
| `export_model` | 🔴 | | |


### Creating Shuffles and Model Configuration

You can configure models using the `pytorch_config.yaml` file, as described
[here](dlc3-pytorch-config). You can use the same methods to create new shuffles in
DeepLabCut 3.0 as you did for Tensorflow models (`deeplabcut.create_training_dataset`
and `deeplabcut.create_training_model_comparison`).

More information about the different PyTorch model architectures available in DeepLabCut
is available [here](dlc3-pytorch-config). You can see a list of supported
architectures/variants by using:

```python
from deeplabcut.pose_estimation_pytorch import available_models
print(available_models())
```