Visitar URL original
Add PyTorch🔥 documentation by n-poulsen · Pull Request #2605 · DeepLabCut/DeepLabCut · GitHub
Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
9aa9647
added initial pytorch docs
Jun 7, 2024
76cd889
Update _toc.yml
MMathisLab Jun 7, 2024
f15a09a
Update user_guide.md
MMathisLab Jun 7, 2024
f448dc9
Update architectures.md
MMathisLab Jun 7, 2024
d598a56
Update labelling.md
MMathisLab Jun 7, 2024
b9195b8
Update labelling.md
MMathisLab Jun 7, 2024
4bc8aa1
Update codespell.yml
MMathisLab Jun 7, 2024
fecf7e5
Merge branch 'main' into pytorch_docs
MMathisLab Jun 7, 2024
0a2a1c5
Update README.md
MMathisLab Jun 7, 2024
a4e050f
Update installation.md
MMathisLab Jun 7, 2024
9cdb9f1
Delete conda-environments/DEEPLABCUT_M1.yaml
MMathisLab Jun 7, 2024
198a34e
Update DEEPLABCUT.yaml
MMathisLab Jun 7, 2024
e3a5dfa
Update installation.md
MMathisLab Jun 7, 2024
5a774b9
Update maDLC_UserGuide.md
MMathisLab Jun 7, 2024
9d71c9b
Update ModelZoo.md
MMathisLab Jun 8, 2024
3a5a5a1
Update standardDeepLabCut_UserGuide.md
MMathisLab Jun 8, 2024
a7c49eb
Update standardDeepLabCut_UserGuide.md
MMathisLab Jun 8, 2024
0aa9e28
Update standardDeepLabCut_UserGuide.md
MMathisLab Jun 8, 2024
4bf88d2
Update Overviewof3D.md
MMathisLab Jun 8, 2024
665f2df
Update PROJECT_GUI.md
MMathisLab Jun 8, 2024
a388d28
Update UseOverviewGuide.md
MMathisLab Jun 8, 2024
c0028c0
Update beginners-guide.md
MMathisLab Jun 8, 2024
329ce71
Update intro.md
MMathisLab Jun 8, 2024
c1aaa84
Update architectures.md
MMathisLab Jun 8, 2024
16e4386
Update architectures.md
MMathisLab Jun 8, 2024
894e3b4
Merge branch 'main' into pytorch_docs
MMathisLab Jun 8, 2024
680f353
Create course.md
MMathisLab Jun 8, 2024
09bd63e
Update _toc.yml
MMathisLab Jun 8, 2024
d29e7c4
Update manage-project.md
MMathisLab Jun 8, 2024
34a7b5a
Update beginners-guide.md
MMathisLab Jun 8, 2024
20d0bca
Update manage-project.md
MMathisLab Jun 8, 2024
f2336cc
Update _toc.yml
MMathisLab Jun 8, 2024
7a3e0ea
Rename labelling.md to labeling.md
MMathisLab Jun 8, 2024
4802d4f
Update Training-Evaluation.md
MMathisLab Jun 8, 2024
1de7232
Update video-analysis.md
MMathisLab Jun 8, 2024
77ea994
Update architectures.md
MMathisLab Jun 8, 2024
73cd31a
Update architectures.md
MMathisLab Jun 8, 2024
8483175
Update AUTHORS
MMathisLab Jun 8, 2024
518909a
Update architectures.md
MMathisLab Jun 8, 2024
82132e1
doc clean up, tested local build
MMathisLab Jun 8, 2024
0197ba4
Update beginners-guide.md
MMathisLab Jun 8, 2024
e91ee09
Update ModelZoo.md
MMathisLab Jun 8, 2024
daf2f87
Update ModelZoo.md
MMathisLab Jun 8, 2024
2a8949b
Update ModelZoo.md
MMathisLab Jun 8, 2024
9a1f28a
Update ModelZoo.md
MMathisLab Jun 8, 2024
02d71df
Update README.md
MMathisLab Jun 8, 2024
0499825
Update ModelZoo.md
MMathisLab Jun 8, 2024
83db9af
Update ModelZoo.md
MMathisLab Jun 8, 2024
14bbee0
Update AUTHORS
MMathisLab Jun 10, 2024
aa3c82d
Update README.md
MMathisLab Jun 11, 2024
fec76d8
Merge branch 'main' into pytorch_docs
MMathisLab Jun 11, 2024
5927e26
Update installation.md
MMathisLab Jun 12, 2024
3cfb0a8
Update DEEPLABCUT.yaml
MMathisLab Jun 12, 2024
bd212aa
Update DEEPLABCUT.yaml
MMathisLab Jun 12, 2024
a6c9318
Update AUTHORS
MMathisLab Jun 13, 2024
63898d7
fixed typos
Jun 13, 2024
7a33b74
Update python-package.yml
MMathisLab Jun 13, 2024
62ca76b
Update README.md
MMathisLab Jun 14, 2024
7c0a5c1
Update beginners-guide.md
MMathisLab Jun 14, 2024
8dfd1e6
Update user_guide.md
MMathisLab Jun 14, 2024
f6ba9d3
m1 for tf only
MMathisLab Jun 14, 2024
093c424
Merge branch 'main' into pytorch_docs
MMathisLab Jun 14, 2024
aeccccb
Update README.md
MMathisLab Jun 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update ModelZoo.md
  • Loading branch information
MMathisLab authored Jun 8, 2024
commit 9d71c9b083b74907f7f6f4b0edc090916c9ddb35
42 changes: 21 additions & 21 deletions docs/ModelZoo.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
# The DeepLabCut Model Zoo!

🦒 🐈 🐕‍🦺 🐀 🐁 🦡 🦦 🐏 🐫 🐆 🦓 🐖 🐄 🐂 🦖 🐿 🦍 🦥
![image](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/8957c690-4f27-4430-8581-4161fd58d052/68747470733a2f2f696d616765732e73717561726573706163652d63646e2e636f6d2f636f6e74656e742f76312f3537663664353163396637343536366635356563663237312f313631363439323337333730302d50474f41433732494f4236415545343756544a582f6b6531375a77644742546f646449.png?format=450w)


## 🏠 [Home page](http://modelzoo.deeplabcut.org/)

Started in 2020 and expanded in 2022, the model zoo is four things:
Started in 2020, expanded in 2022, and the first Foundation Models published in 2024, the Model Zoo is four things:

- (1) a collection of models that are trained on diverse data across (typically) large datasets, which means you do not need to train models yourself
- (2) a contribution website for community crowd sourcing of expertly labeled keypoints to improve models in part 1!
- (2) a contribution website for community crowd sourcing of expertly labeled keypoints to improve models! You can get involved here: [contrib.deeplabcut.org](https://contrib.deeplabcut.org/).
- (3) a no-install DeepLabCut that you can use on ♾[Google Colab](https://colab.research.google.com/github/DeepLabCut/DeepLabCut/blob/master/examples/COLAB/COLAB_DLC_ModelZoo.ipynb),
Comment thread
MMathisLab marked this conversation as resolved.
Resolved
test our models in 🕸[the browser](https://contrib.deeplabcut.org/), or on our 🤗[HuggingFace](https://huggingface.co/spaces/DeepLabCut/MegaDetector_DeepLabCut) app!
- (4) new methods to make SuperAnimal Models that combine data across different labs/datasets, keypoints, animals/species, and use on your data!
- (4) new methods to make SuperAnimal Foundation Models that combine data across different labs/datasets, keypoints, animals/species, and use on your data!

## Quick Start:
```
pip install deeplabcut[tf,gui,modelzoo]
pip install deeplabcut[gui,modelzoo]
```

## About the SuperAnimal Models
Expand All @@ -25,26 +26,27 @@ To provide the community with easy access to such high performance models across

### We now introduce two SuperAnimal members, namely, `superanimal_quadruped` and `superanimal_topviewmouse`.

#### `superanimal_quadruped` model aim to work across a large range of quadruped animals, from horses, dogs, sheep, rodents, to elephants. The camera perspective is orthogonal to the animal ("side view"), and most of the data includes the animals face (thus the front and side of the animal). Here are example images of what the model is trained on:
- `superanimal_quadruped_x` models aim to work across a large range of quadruped animals, from horses, dogs, sheep, rodents, to elephants. The camera perspective is orthogonal to the animal ("side view"), and most of the data includes the animals face (thus the front and side of the animal). You will note we have several variants that differ in speed vs. performance, so please do test them out on your data to see which is best suited for your application. Also note we have a "video adaptation" feature, which lets you adapt your data to the model in a self-supervised way. No labeling needed!

- Here are example images of what the model is trained on:

![SA_Q](https://user-images.githubusercontent.com/28102185/209957688-954fb616-7750-4521-bb52-20a51c3a7718.png)

#### `superanimal_topviewmouse` aims to work across lab mice in different lab settings from a top-view perspective; this is very polar in many behavioral assays in freely moving mice. Here are example images of what the model is trained on:
- `superanimal_topviewmouse_x` aims to work across lab mice in different lab settings from a top-view perspective; this is very polar in many behavioral assays in freely moving mice. Here are example images of what the model is trained on:

![SA-TVM](https://user-images.githubusercontent.com/28102185/209957260-c0db72e0-4fdf-434c-8579-34bc5f27f907.png)


IMPORTANT: we currently only support single animal scenarios

### Our perspective.

Via DeepLabCut Model Zoo, we aim to provide plug and play models that do not need any labeling and will just work decently on novel videos. If the predictions are not great enough due to failure modes described below, please give us feedback! We are rapidly improving our models and adaptation methods.


### To use our models in DeepLabCut (versions 2.3+), please use the following API
### To use our models in DeepLabCut (versions 3+), please use the following API

```
pip install deeplabcut[tf,modelzoo]
pip install deeplabcut[modelzoo]
```

#### Practical example: Using SuperAnimal models for inference without training.
Expand Down Expand Up @@ -80,25 +82,23 @@ deeplabcut.train_network(config_path,



### To see the list of available models, check out the [Home page](http://modelzoo.deeplabcut.org/).

**Coming soon:** The DeepLabCut Project Manager GUI will allow you to use the SuperAnimal Models. You can run the model and do ``active learning" to improve performance on your data.
Specifically, we have *new* video adaptation methods to make your tracking extra smooth and robust!

### Potential failure modes for SuperAnimal Models and how to fix it.

Spatial domain shift: typical DNN models suffer from the spatial resolution shift between training datasets and test videos. To help find the proper resolution for our model, please try a range of `scale_list` in the API (details in the API docs). For `superanimal_quadruped`, we empirically observe that if your video is larger than 1500 pixels, it is better to pass `scale_list` in the range within 1000.

Pixel statistics domain shift: The brightness of your video might look very different from our training datasets. This might either result in jittering predictions in the video or fail modes for lab mice videos (if the brightness of the mice is unusual compared to our training dataset). You can use our "video adaptation" model (released soon) to counter this.
### To see our first preprint on the work, check out [our paper](https://arxiv.org/abs/2203.07436v1):
### To see our first preprint on the work, check out [our paper](https://arxiv.org/abs/2203.07436v1).

## Publication:

```{hint}
Here is the citation:
@article{Ye2022PanopticAP,
title={Panoptic animal pose estimators are zero-shot performers},
author={Shaokai Ye and Alexander Mathis and Mackenzie W. Mathis},
journal={ArXiv},
year={2022},
volume={abs/2203.07436}
@article{Ye2024,
title={SuperAnimal pretrained pose estimation models for behavioral analysis},
author={Shaokai Ye and Anastasiia Filippova and Jessy Lauer and Steffen Schneider and Maxime Vidal and Tian Qiu and Alexander Mathis and Mackenzie Weygandt Mathis},
journal={Nature Communications},
year={2024},
preprint={abs/2203.07436}
}
```