deeplabcut.create_new_project(
    "Name of the project",
    "Name of the experimenter",
    ["Full path of video 1", "Full path of video2", "Full path of video3"],
    working_directory="Full path of the working directory",
    copy_videos=True/False,
    multianimal=False
)

deeplabcut.add_new_videos(
    "Full path of the project configuration file*",
    ["full path of video 4", "full path of video 5"],
    copy_videos=True/False
)

Click the button to see API Docs

deeplabcut.create_project.new.create_new_project(project: str, experimenter: str, videos: list[str], working_directory: str | None = None, copy_videos: bool = False, videotype: str = '', multianimal: bool = False, individuals: list[str] | None = None)#

Create the necessary folders and files for a new project.

Creating a new project involves creating the project directory, sub-directories and a basic configuration file. The configuration file is loaded with the default values. Change its parameters to your projects need.

Parameters:

projectstring

The name of the project.

experimenterstring

The name of the experimenter.

videoslist[str]

A list of strings representing the full paths of the videos to include in the project. If the strings represent a directory instead of a file, all videos of videotype will be imported.

working_directorystring, optional

The directory where the project will be created. The default is the current working directory.

copy_videosbool, optional, Default: False.

If True, the videos are copied to the videos directory. If False, symlinks of the videos will be created in the project/videos directory; in the event of a failure to create symbolic links, videos will be moved instead.

multianimal: bool, optional. Default: False.

For creating a multi-animal project (introduced in DLC 2.2)

individuals: list[str]|None = None,

Relevant only if multianimal is True. list of individuals to be used in the project configuration. If None - defaults to [‘individual1’, ‘individual2’, ‘individual3’]

Returns:

str

Path to the new project configuration file.

Examples

Linux/MacOS:

>>> deeplabcut.create_new_project(
        project='reaching-task',
        experimenter='Linus',
        videos=[
            '/data/videos/mouse1.avi',
            '/data/videos/mouse2.avi',
            '/data/videos/mouse3.avi'
        ],
        working_directory='/analysis/project/',
    )
>>> deeplabcut.create_new_project(
        project='reaching-task',
        experimenter='Linus',
        videos=['/data/videos'],
        videotype='.mp4',
    )

Windows:

>>> deeplabcut.create_new_project(
        'reaching-task',
        'Bill',
        [r'C:\yourusername\rig-95\Videos\reachingvideo1.avi'],
        copy_videos=True,
    )

Users must format paths with either: r’C:OR ‘C:\ <- i.e. a double backslash )

deeplabcut.extract_frames(
    config_path,
    mode="automatic/manual",
    algo="uniform/kmeans",
    crop=True/False,
    userfeedback=False
)

deeplabcut.extract_frames(config_path, "manual")

Click the button to see API Docs

Extracts frames from the project videos.

Frames will be extracted from videos listed in the config.yaml file.

The frames are selected from the videos in a randomly and temporally uniformly distributed way (uniform), by clustering based on visual appearance (k-means), or by manual selection.

After frames have been extracted from all videos from one camera, matched frames from other cameras can be extracted using mode = "match". This is necessary if you plan to use epipolar lines to improve labeling across multiple camera angles. It will overwrite previously extracted images from the second camera angle if necessary.

Please refer to the user guide for more details on methods and parameters https://www.nature.com/articles/s41596-019-0176-0 or the preprint: https://www.biorxiv.org/content/biorxiv/early/2018/11/24/476531.full.pdf

Parameters:

configstring

Full path of the config.yaml file as a string.

modestring. Either "automatic", "manual" or "match".

String containing the mode of extraction. It must be either "automatic" or "manual" to extract the initial set of frames. It can also be "match" to match frames between the cameras in preparation for the use of epipolar line during labeling; namely, extract from camera_1 first, then run this to extract the matched frames in camera_2.

WARNING: if you use "match", and you previously extracted and labeled frames from the second camera, this will overwrite your data. This will require you to delete the collectdata(.h5/.csv) files before labeling. Use with caution!

algostring, Either "kmeans" or "uniform", Default: “kmeans”.

String specifying the algorithm to use for selecting the frames. Currently, deeplabcut supports either kmeans or uniform based selection. This flag is only required for automatic mode and the default is kmeans. For "uniform", frames are picked in temporally uniform way, "kmeans" performs clustering on downsampled frames (see user guide for details).

NOTE: Color information is discarded for "kmeans", thus e.g. for camouflaged octopus clustering one might want to change this.

cropbool or str, optional

If True, video frames are cropped according to the corresponding coordinates stored in the project configuration file. Alternatively, if cropping coordinates are not known yet, crop=``”GUI”`` triggers a user interface where the cropping area can be manually drawn and saved.

userfeedback: bool, optional

If this is set to False during "automatic" mode then frames for all videos are extracted. The user can set this to "True", which will result in a dialog, where the user is asked for each video if (additional/any) frames from this video should be extracted. Use this, e.g. if you have already labeled some folders and want to extract data for new videos.

cluster_resizewidth: int, default: 30

For "k-means" one can change the width to which the images are downsampled (aspect ratio is fixed).

cluster_step: int, default: 1

By default each frame is used for clustering, but for long videos one could only use every nth frame (set using this parameter). This saves memory before clustering can start, however, reading the individual frames takes longer due to the skipping.

cluster_color: bool, default: False

If "False" then each downsampled image is treated as a grayscale vector (discarding color information). If "True", then the color channels are considered. This increases the computational complexity.

opencv: bool, default: True

Uses openCV for loading & extractiong (otherwise moviepy (legacy)).

slider_width: int, default: 25

Width of the video frames slider, in percent of window.

config3d: string, optional

Path to the project configuration file in the 3D project. This will be used to match frames extracted from all cameras present in the field ‘camera_names’ to the frames extracted from the camera given by the parameter ‘extracted_cam’.

extracted_cam: int, default: 0

The index of the camera that already has extracted frames. This will match frame numbers to extract for all other cameras. This parameter is necessary if you wish to use epipolar lines in the labeling toolbox. Only use if mode='match' and config3d is provided.

videos_list: list[str], Default: None

A list of the string containing full paths to videos to extract frames for. If this is left as None all videos specified in the config file will have frames extracted. Otherwise one can select a subset by passing those paths.

Returns:

None

Notes

Use the function add_new_videos at any stage of the project to add new videos to the config file and extract their frames.

The following parameters for automatic extraction are used from the config file

• numframes2pick

• start and stop

While selecting the frames manually, you do not need to specify the crop parameter in the command. Rather, you will get a prompt in the graphic user interface to choose if you need to crop or not.

Examples

To extract frames automatically with ‘kmeans’ and then crop the frames

>>> deeplabcut.extract_frames(
        config='/analysis/project/reaching-task/config.yaml',
        mode='automatic',
        algo='kmeans',
        crop=True,
    )

To extract frames automatically with ‘kmeans’ and then defining the cropping area using a GUI

>>> deeplabcut.extract_frames(
        '/analysis/project/reaching-task/config.yaml',
        'automatic',
        'kmeans',
        'GUI',
    )

To consider the color information when extracting frames automatically with ‘kmeans’

>>> deeplabcut.extract_frames(
        '/analysis/project/reaching-task/config.yaml',
        'automatic',
        'kmeans',
        cluster_color=True,
    )

To extract frames automatically with ‘uniform’ and then crop the frames

>>> deeplabcut.extract_frames(
        '/analysis/project/reaching-task/config.yaml',
        'automatic',
        'uniform',
        crop=True,
    )

To extract frames manually

>>> deeplabcut.extract_frames(
        '/analysis/project/reaching-task/config.yaml', 'manual'
    )

To extract frames manually, with a 60% wide frames slider

>>> deeplabcut.extract_frames(
        '/analysis/project/reaching-task/config.yaml', 'manual', slider_width=60,
    )

To extract frames from a second camera that match the frames extracted from the first

>>> deeplabcut.extract_frames(
        '/analysis/project/reaching-task/config.yaml',
        mode='match',
        extracted_cam=0,
    )

deeplabcut.label_frames(config_path)

Ctrl + C: Copy labels from previous frame.
Keyboard arrows: advance frames.
Delete key: delete label.

deeplabcut.check_labels(config_path, visualizeindividuals=True/False)

Click the button to see API Docs

deeplabcut.generate_training_dataset.trainingsetmanipulation.check_labels(config, Labels=['+', '.', 'x'], scale=1, dpi=100, draw_skeleton=True, visualizeindividuals=True)#

Check the labeled frames.

Double check if the labels were at the correct locations and stored in the proper file format.

This creates a new subdirectory for each video under the ‘labeled-data’ and all the frames are plotted with the labels.

Make sure that these labels are fine.

Parameters:

configstring

Full path of the config.yaml file as a string.

Labels: list, default=’+’

List of at least 3 matplotlib markers. The first one will be used to indicate the human ground truth location (Default: +)

scalefloat, default=1

Change the relative size of the output images.

dpiint, optional, default=100

Output resolution in dpi.

draw_skeleton: bool, default=True

Plot skeleton overlaid over body parts.

visualizeindividuals: bool, default: True.

For a multianimal project, if True, the different individuals have different colors (and all bodyparts the same). If False, the colors change over bodyparts rather than individuals.

Returns:

None

Examples

>>> deeplabcut.check_labels('/analysis/project/reaching-task/config.yaml')

deeplabcut.create_training_dataset(config_path)

Hint

🚨 If they do not download (you will see this downloading in the terminal), then you may not have permission to do so - be sure to open your terminal “as an admin” (This is only something we have seen with some Windows users - see the docs for more help!).

Added in version 3.0.0: You can now create new shuffles using the same train/test split as existing shuffles with create_training_dataset_from_existing_split. This allows you to compare model performance (between different architectures or when using different training hyper-parameters) as the shuffles were trained on the same data, and evaluated on the same test data!

Example usage - creating 3 new shuffles (with indices 10, 11 and 12) for a ResNet 50 pose estimation model, using the same data split as was used for shuffle 0:

deeplabcut.create_training_dataset_from_existing_split(
    config_path,
    from_shuffle=0,
    shuffles=[10, 11, 12],
    net_type="resnet_50",
)

Click the button to see API Docs for deeplabcut.create_training_dataset

deeplabcut.generate_training_dataset.trainingsetmanipulation.create_training_dataset(config, num_shuffles=1, Shuffles=None, windows2linux=False, userfeedback=True, trainIndices=None, testIndices=None, net_type=None, detector_type=None, augmenter_type=None, posecfg_template=None, superanimal_name='', weight_init: WeightInitialization | None = None, engine: Engine | None = None, ctd_conditions: int | str | Path | tuple[int, str] | tuple[int, int] | None = None)#

Creates a training dataset.

Labels from all the extracted frames are merged into a single .h5 file. Only the videos included in the config file are used to create this dataset.

Parameters:

configstring

Full path of the config.yaml file as a string.

num_shufflesint, optional, default=1

Number of shuffles of training dataset to create, i.e. [1,2,3] for num_shuffles=3.

Shuffles: list[int], optional

Alternatively the user can also give a list of shuffles.

userfeedback: bool, optional, default=True

If False, all requested train/test splits are created (no matter if they already exist). If you want to assure that previous splits etc. are not overwritten, set this to True and you will be asked for each split.

trainIndices: list of lists, optional, default=None

List of one or multiple lists containing train indexes. A list containing two lists of training indexes will produce two splits.

testIndices: list of lists, optional, default=None

List of one or multiple lists containing test indexes.

net_type: list, optional, default=None

Type of networks. The options available depend on which engine is used. Currently supported options are:

TensorFlow

• resnet_50

• resnet_101

• resnet_152

• mobilenet_v2_1.0

• mobilenet_v2_0.75

• mobilenet_v2_0.5

• mobilenet_v2_0.35

• efficientnet-b0

• efficientnet-b1

• efficientnet-b2

• efficientnet-b3

• efficientnet-b4

• efficientnet-b5

• efficientnet-b6

PyTorch (call deeplabcut.pose_estimation_pytorch.available_models() for a complete list)

• animaltokenpose_base

• cspnext_m

• cspnext_s

• cspnext_x

• ctd_coam_w32

• ctd_coam_w48

• ctd_prenet_cspnext_m

• ctd_prenet_cspnext_x

• ctd_prenet_rtmpose_x_human

• ctd_prenet_hrnet_w32

• ctd_prenet_hrnet_w48

• ctd_prenet_rtmpose_m

• ctd_prenet_rtmpose_x

• ctd_prenet_rtmpose_x_human

• dekr_w18

• dekr_w32

• dekr_w48

• dlcrnet_stride16_ms5

• dlcrnet_stride32_ms5

• hrnet_w18

• hrnet_w32

• hrnet_w48

• resnet_101

• resnet_50

• rtmpose_m

• rtmpose_s

• rtmpose_x

• top_down_cspnext_m

• top_down_cspnext_s

• top_down_cspnext_x

• top_down_hrnet_w18

• top_down_hrnet_w32

• top_down_hrnet_w48

• top_down_resnet_101

• top_down_resnet_50

detector_type: string, optional, default=None

Only for the PyTorch engine. When passing creating shuffles for top-down models, you can specify which detector you want. If the detector_type is None, the `ssdlite` will be used. The list of all available detectors can be obtained by calling deeplabcut.pose_estimation_pytorch.available_detectors(). Supported options:

• ssdlite

• fasterrcnn_mobilenet_v3_large_fpn

• fasterrcnn_resnet50_fpn_v2

augmenter_type: string, optional, default=None

Type of augmenter. The options available depend on which engine is used. Currently supported options are:

TensorFlow

• default

• scalecrop

• imgaug

• tensorpack

• deterministic

PyTorch

• albumentations

posecfg_template: string, optional, default=None

Only for the TensorFlow engine. Path to a pose_cfg.yaml file to use as a template for generating the new one for the current iteration. Useful if you would like to start with the same parameters a previous training iteration. None uses the default pose_cfg.yaml.

superanimal_name: string, optional, default=””

Only for the TensorFlow engine. For the PyTorch engine, use the weight_init parameter. Specify the superanimal name is transfer learning with superanimal is desired. This makes sure the pose config template uses superanimal configs as template.

weight_init: WeightInitialisation, optional, default=None

PyTorch engine only. Specify how model weights should be initialized. The default mode uses transfer learning from ImageNet weights.

engine: Engine, optional

Whether to create a pose config for a Tensorflow or PyTorch model. Defaults to the value specified in the project configuration file. If no engine is specified for the project, defaults to deeplabcut.compat.DEFAULT_ENGINE.

ctd_conditions: int | str | Path | tuple[int, str] | tuple[int, int] | None, default = None,

If using a conditional-top-down (CTD) net_type, this argument should be specified. It defines the conditions that will be used with the CTD model. It can be either:

• A shuffle number (ctd_conditions: int), which must correspond to a

  bottom-up (BU) network type.

• A predictions file path (ctd_conditions: string | Path), which must

  correspond to a .json or .h5 predictions file.

• A shuffle number and a particular snapshot

  (ctd_conditions: tuple[int, str] | tuple[int, int]), which respectively correspond to a bottom-up (BU) network type and a particular snapshot name or index.

Returns:

list(tuple) or None

If training dataset was successfully created, a list of tuples is returned. The first two elements in each tuple represent the training fraction and the shuffle value. The last two elements in each tuple are arrays of integers representing the training and test indices.

Returns None if training dataset could not be created.

Notes

Use the function add_new_videos at any stage of the project to add more videos to the project.

Examples

Linux/MacOS: >>> deeplabcut.create_training_dataset(

‘/analysis/project/reaching-task/config.yaml’, num_shuffles=1,

)

>>> deeplabcut.create_training_dataset(
        '/analysis/project/reaching-task/config.yaml', Shuffles=[2], engine=deeplabcut.Engine.TF,
    )

Windows: >>> deeplabcut.create_training_dataset(

‘C:UsersUlflooming-taskconfig.yaml’, Shuffles=[3,17,5],

)

Click the button to see API Docs for deeplabcut.create_training_model_comparison

deeplabcut.generate_training_dataset.trainingsetmanipulation.create_training_model_comparison(config, trainindex=0, num_shuffles=1, net_types=['resnet_50'], augmenter_types=['imgaug'], userfeedback=False, windows2linux=False)#

Creates a training dataset to compare networks and augmentation types.

The datasets are created such that the shuffles have same training and testing indices. Therefore, this function is useful for benchmarking the performance of different network and augmentation types on the same training/testdata.

Parameters:

config: str

Full path of the config.yaml file.

trainindex: int, optional, default=0

Either (in case uniform = True) indexes which element of TrainingFraction in the config file should be used (note it is a list!). Alternatively (uniform = False) indexes which folder is dropped, i.e. the first if trainindex=0, the second if trainindex=1, etc.

num_shufflesint, optional, default=1

Number of shuffles of training dataset to create, i.e. [1,2,3] for num_shuffles=3.

net_types: list[str], optional, default=[“resnet_50”]

Currently supported networks are

• "resnet_50"

• "resnet_101"

• "resnet_152"

• "mobilenet_v2_1.0"

• "mobilenet_v2_0.75"

• "mobilenet_v2_0.5"

• "mobilenet_v2_0.35"

• "efficientnet-b0"

• "efficientnet-b1"

• "efficientnet-b2"

• "efficientnet-b3"

• "efficientnet-b4"

• "efficientnet-b5"

• "efficientnet-b6"

augmenter_types: list[str], optional, default=[“imgaug”]

Currently supported augmenters are

• "default"

• "imgaug"

• "tensorpack"

• "deterministic"

userfeedback: bool, optional, default=False

If False, then all requested train/test splits are created, no matter if they already exist. If you want to assure that previous splits etc. are not overwritten, then set this to True and you will be asked for each split.

windows2linux

..deprecated::

Has no effect since 2.2.0.4 and will be removed in 2.2.1.

Returns:

shuffle_list: list

List of indices corresponding to the trainingsplits/models that were created.

Examples

On Linux/MacOS

>>> shuffle_list = deeplabcut.create_training_model_comparison(
        '/analysis/project/reaching-task/config.yaml',
        num_shuffles=1,
        net_types=['resnet_50','resnet_152'],
        augmenter_types=['tensorpack','deterministic'],
    )

On Windows

>>> shuffle_list = deeplabcut.create_training_model_comparison(
        'C:\Users\Ulf\looming-task\config.yaml',
        num_shuffles=1,
        net_types=['resnet_50','resnet_152'],
        augmenter_types=['tensorpack','deterministic'],
    )

See examples/testscript_openfielddata_augmentationcomparison.py for an example of how to use shuffle_list.

Click the button to see API Docs for deeplabcut.create_training_dataset_from_existing_split

deeplabcut.generate_training_dataset.trainingsetmanipulation.create_training_dataset_from_existing_split(config: str, from_shuffle: int, from_trainsetindex: int = 0, num_shuffles: int = 1, shuffles: list[int] | None = None, userfeedback: bool = True, net_type: str | None = None, detector_type: str | None = None, augmenter_type: str | None = None, ctd_conditions: int | str | Path | tuple[int, str] | tuple[int, int] | None = None, posecfg_template: dict | None = None, superanimal_name: str = '', weight_init: WeightInitialization | None = None, engine: Engine | None = None) → None | list[int]#

Labels from all the extracted frames are merged into a single .h5 file. Only the videos included in the config file are used to create this dataset.

Args:

config: Full path of the config.yaml file as a string.

from_shuffle: The index of the shuffle from which to copy the train/test split.

from_trainsetindex: The trainset index of the shuffle from which to use the data

split. Default is 0.

num_shuffles: Number of shuffles of training dataset to create, used if

shuffles is None.

shuffles: If defined, num_shuffles is ignored and a shuffle is created for

each index given in the list.

userfeedback: If False, all requested train/test splits are created (no

matter if they already exist). If you want to assure that previous splits etc. are not overwritten, set this to True and you will be asked for each existing split if you want to overwrite it.

net_type: The type of network to create the shuffle for. Currently supported

options for engine=Engine.TF are:

• resnet_50

• resnet_101

• resnet_152

• mobilenet_v2_1.0

• mobilenet_v2_0.75

• mobilenet_v2_0.5

• mobilenet_v2_0.35

• efficientnet-b0

• efficientnet-b1

• efficientnet-b2

• efficientnet-b3

• efficientnet-b4

• efficientnet-b5

• efficientnet-b6

Currently supported options for engine=Engine.TF can be obtained by calling deeplabcut.pose_estimation_pytorch.available_models().

detector_type: string, optional, default=None

Only for the PyTorch engine. When passing creating shuffles for top-down models, you can specify which detector you want. If the detector_type is None, the `ssdlite` will be used. The list of all available detectors can be obtained by calling deeplabcut.pose_estimation_pytorch.available_detectors(). Supported options:

• ssdlite

• fasterrcnn_mobilenet_v3_large_fpn

• fasterrcnn_resnet50_fpn_v2

augmenter_type: Type of augmenter. Currently supported augmenters for

engine=Engine.TF are

• default

• scalecrop

• imgaug

• tensorpack

• deterministic

The only supported augmenter for Engine.PYTORCH is albumentations.

posecfg_template: Only for Engine.TF. Path to a pose_cfg.yaml file to use as

a template for generating the new one for the current iteration. Useful if you would like to start with the same parameters a previous training iteration. None uses the default pose_cfg.yaml.

superanimal_name: Specify the superanimal name is transfer learning with

superanimal is desired. This makes sure the pose config template uses superanimal configs as template.

weight_init: Only for Engine.PYTORCH. Specify how model weights should be

initialized. The default mode uses transfer learning from ImageNet weights.

engine: Whether to create a pose config for a Tensorflow or PyTorch model.

Defaults to the value specified in the project configuration file. If no engine is specified for the project, defaults to deeplabcut.compat.DEFAULT_ENGINE.

ctd_conditions: int | str | Path | tuple[int, str] | tuple[int, int] | None, default = None,

If using a conditional-top-down (CTD) net_type, this argument should be specified. It defines the conditions that will be used with the CTD model. It can be either:

• A shuffle number (ctd_conditions: int), which must correspond to a bottom-up (BU) network type.

• A predictions file path (ctd_conditions: string | Path), which must correspond to a .json or .h5 predictions file.

• A shuffle number and a particular snapshot (ctd_conditions: tuple[int, str] | tuple[int, int]), which respectively correspond to a bottom-up (BU) network type and a particular snapshot name or index.

Returns:

If training dataset was successfully created, a list of tuples is returned. The first two elements in each tuple represent the training fraction and the shuffle value. The last two elements in each tuple are arrays of integers representing the training and test indices.

Returns None if training dataset could not be created.

Raises:

ValueError: If the shuffle from which to copy the data split doesn’t exist.

deeplabcut.train_network(config_path)

Tips on training models with the PyTorch Engine

Example parameters that one can call:

deeplabcut.train_network(
    config_path,
    shuffle=1,
    trainingsetindex=0,
    device="cuda:0",
    max_snapshots_to_keep=5,
    displayiters=100,
    save_epochs=5,
    epochs=200,
)

Pytorch models in DeepLabCut 3.0 are trained for a set number of epochs, instead of a maximum number of iterations (which is what was used for TensorFlow models). An epoch is a single pass through the training dataset, which means your model has seen each training image exactly once. So if you have 64 training images for your network, an epoch is 64 iterations with batch size 1 (or 32 iterations with batch size 2, 16 with batch size 4, etc.).

By default, the pretrained networks are not in the DeepLabCut toolbox (as they can be more than 100MB), but they get downloaded automatically before you train.

If the user wishes to restart the training at a specific checkpoint they can specify the full path of the checkpoint to the variable resume_training_from in the pytorch_config.yaml file (checkout the “Restarting Training at a Specific Checkpoint” section of the docs) under the train subdirectory.

CRITICAL POINT: It is recommended to train the networks until the loss plateaus (depending on the dataset, model architecture and training hyper-parameters this happens after 100 to 250 epochs of training).

The variables display_iters and save_epochs in the pytorch_config.yaml file allows the user to alter how often the loss is displayed and how often the weights are stored. We suggest saving every 5 to 25 epochs.

Tips on training models with the TensorFlow Engine

Example parameters that one can call:

deeplabcut.train_network(
    config_path,
    shuffle=1,
    trainingsetindex=0,
    gputouse=None,
    max_snapshots_to_keep=5,
    autotune=False,
    displayiters=100,
    saveiters=25000,
    maxiters=300000,
    allow_growth=True,
)

By default, the pretrained networks are not in the DeepLabCut toolbox (as they are around 100MB each), but they get downloaded before you train. However, if not previously downloaded from the TensorFlow model weights, it will be downloaded and stored in a subdirectory pre-trained under the subdirectory models in Pose_Estimation_Tensorflow. At user specified iterations during training checkpoints are stored in the subdirectory train under the respective iteration directory.

If the user wishes to restart the training at a specific checkpoint they can specify the full path of the checkpoint to the variable init_weights in the pose_cfg.yaml file under the train subdirectory (see Box 2).

CRITICAL POINT: It is recommended to train the networks for thousands of iterations until the loss plateaus (typically around 500,000) if you use batch size 1. If you want to batch train, we recommend using Adam, see more here.

The variables display_iters and save_iters in the pose_cfg.yaml file allows the user to alter how often the loss is displayed and how often the weights are stored.

maDeepLabCut CRITICAL POINT: For multi-animal projects we are using not only different and new output layers, but also new data augmentation, optimization, learning rates, and batch training defaults. Thus, please use a lower save_iters and maxiters. I.e. we suggest saving every 10K-15K iterations, and only training until 50K-100K iterations. We recommend you look closely at the loss to not overfit on your data. The bonus, training time is much less!!!

Click the button to see API Docs for train_network

deeplabcut.compat.train_network(config: str | Path, shuffle: int = 1, trainingsetindex: int = 0, max_snapshots_to_keep: int | None = None, displayiters: int | None = None, saveiters: int | None = None, maxiters: int | None = None, epochs: int | None = None, save_epochs: int | None = None, allow_growth: bool = True, gputouse: str | None = None, autotune: bool = False, keepdeconvweights: bool = True, modelprefix: str = '', superanimal_name: str = '', superanimal_transfer_learning: bool = False, engine: Engine | None = None, device: str | None = None, snapshot_path: str | Path | None = None, detector_path: str | Path | None = None, batch_size: int | None = None, detector_batch_size: int | None = None, detector_epochs: int | None = None, detector_save_epochs: int | None = None, pose_threshold: float | None = 0.1, pytorch_cfg_updates: dict | None = None)#

Trains the network with the labels in the training dataset.

Parameters:

configstring

Full path of the config.yaml file as a string.

shuffle: int, optional, default=1

Integer value specifying the shuffle index to select for training.

trainingsetindex: int, optional, default=0

Integer specifying which TrainingsetFraction to use. Note that TrainingFraction is a list in config.yaml.

max_snapshots_to_keep: int or None

Sets how many snapshots are kept, i.e. states of the trained network. Every saving iteration many times a snapshot is stored, however only the last max_snapshots_to_keep many are kept! If you change this to None, then all are kept. See: DeepLabCut/DeepLabCut#8

displayiters: optional, default=None

This variable is actually set in pose_config.yaml. However, you can overwrite it with this hack. Don’t use this regularly, just if you are too lazy to dig out the pose_config.yaml file for the corresponding project. If None, the value from there is used, otherwise it is overwritten!

saveiters: optional, default=None

Only for the TensorFlow engine (for the PyTorch engine see the torch_kwargs: you can use save_epochs). This variable is actually set in pose_config.yaml. However, you can overwrite it with this hack. Don’t use this regularly, just if you are too lazy to dig out the pose_config.yaml file for the corresponding project. If None, the value from there is used, otherwise it is overwritten!

maxiters: optional, default=None

Only for the TensorFlow engine (for the PyTorch engine see the torch_kwargs: you can use epochs). This variable is actually set in pose_config.yaml. However, you can overwrite it with this hack. Don’t use this regularly, just if you are too lazy to dig out the pose_config.yaml file for the corresponding project. If None, the value from there is used, otherwise it is overwritten!

epochs: optional, default=None

Only for the PyTorch engine (equivalent to the maxiters parameter for the TensorFlow engine). The maximum number of epochs to train the model for. If None, the value will be read from the pytorch_config.yaml file. An epoch is a single pass through the training dataset, which means your model has seen each training image exactly once. So if you have 64 training images for your network, an epoch is 64 iterations with batch size 1 (or 32 iterations with batch size 2, 16 with batch size 4, etc.).

save_epochs: optional, default=None

Only for the PyTorch engine (equivalent to the saveiters parameter for the TensorFlow engine). The number of epochs between each snapshot save. If None, the value will be read from the pytorch_config.yaml file.

allow_growth: bool, optional, default=True.

Only for the TensorFlow engine. For some smaller GPUs the memory issues happen. If True, the memory allocator does not pre-allocate the entire specified GPU memory region, instead starting small and growing as needed. See issue: https://forum.image.sc/t/how-to-stop-running-out-of-vram/30551/2

gputouse: optional, default=None

Only for the TensorFlow engine (for the PyTorch engine see the torch_kwargs: you can use device). Natural number indicating the number of your GPU (see number in nvidia-smi). If you do not have a GPU put None. See: https://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries

autotune: bool, optional, default=False

Only for the TensorFlow engine. Property of TensorFlow, somehow faster if False (as Eldar found out, see tensorflow/tensorflow#13317).

keepdeconvweights: bool, optional, default=True

Also restores the weights of the deconvolution layers (and the backbone) when training from a snapshot. Note that if you change the number of bodyparts, you need to set this to false for re-training.

modelprefix: str, optional, default=””

Directory containing the deeplabcut models to use when evaluating the network. By default, the models are assumed to exist in the project folder.

superanimal_name: str, optional, default =””

Only for the TensorFlow engine. For the PyTorch engine, you need to specify this through the weight_init when creating the training dataset. Specified if transfer learning with superanimal is desired

superanimal_transfer_learning: bool, optional, default = False.

Only for the TensorFlow engine. For the PyTorch engine, you need to specify this through the weight_init when creating the training dataset. If set true, the training is transfer learning (new decoding layer). If set false, and superanimal_name is True, then the training is fine-tuning (reusing the decoding layer)

engine: Engine, optional, default = None.

The default behavior loads the engine for the shuffle from the metadata. You can overwrite this by passing the engine as an argument, but this should generally not be done.

device: str, optional, default = None.

Only for the PyTorch engine. The device to run the training on (e.g. “cuda:0”)

snapshot_path: str or Path, optional, default = None.

Only for the PyTorch engine. The path to the pose model snapshot to resume training from.

detector_path: str or Path, optional, default = None.

Only for the PyTorch engine. The path to the detector model snapshot to resume training from.

batch_size: int, optional, default = None.

Only for the PyTorch engine. The batch size to use while training.

detector_batch_size: int, optional, default = None.

Only for the PyTorch engine. The batch size to use while training the detector.

detector_epochs: int, optional, default = None.

Only for the PyTorch engine. The number of epochs to train the detector for.

detector_save_epochs: int, optional, default = None.

Only for the PyTorch engine. The number of epochs between each detector snapshot save.

pose_threshold: float, optional, default = 0.1.

Only for the PyTorch engine. Used for memory-replay. Pseudo-predictions with confidence lower

than this threshold are discarded for memory-replay

pytorch_cfg_updates: dict, optional, default = None.

A dictionary of updates to the pytorch config. The keys are the dot-separated paths to the values to update in the config. For example, to update the gpus to run the training on, you can use: ` pytorch_cfg_updates={"runner.gpus": [0,1,2,3]} `

Returns:

None

Examples

To train the network for first shuffle of the training dataset

>>> deeplabcut.train_network('/analysis/project/reaching-task/config.yaml')

To train the network for second shuffle of the training dataset

>>> deeplabcut.train_network(
        '/analysis/project/reaching-task/config.yaml',
        shuffle=2,
        keepdeconvweights=True,
    )

To train the network for shuffle created with a PyTorch engine, while overriding the number of epochs, batch size and other parameters.

>>> deeplabcut.train_network(
        '/analysis/project/reaching-task/config.yaml',
        shuffle=1,
        batch_size=8,
        epochs=100,
        save_epochs=10,
        displayiters=50,
    )

deeplabcut.evaluate_network(config_path, Shuffles=[1], plotting=True)

deeplabcut.extract_save_all_maps(config_path, shuffle=shuffle, Indices=[0, 5])

Click the button to see API Docs

deeplabcut.compat.evaluate_network(config: str | Path, Shuffles: Iterable[int] = (1,), trainingsetindex: int | str = 0, plotting: bool | str = False, show_errors: bool = True, comparisonbodyparts: str | list[str] = 'all', gputouse: str | None = None, rescale: bool = False, modelprefix: str = '', per_keypoint_evaluation: bool = False, snapshots_to_evaluate: list[str] | None = None, pcutoff: float | list[float] | dict[str, float] | None = None, engine: Engine | None = None, **torch_kwargs)#

Evaluates the network.

Evaluates the network based on the saved models at different stages of the training network. The evaluation results are stored in the .h5 and .csv file under the subdirectory ‘evaluation_results’. Change the snapshotindex parameter in the config file to ‘all’ in order to evaluate all the saved models.

Parameters:

configstring

Full path of the config.yaml file.

Shuffles: list, optional, default=[1]

List of integers specifying the shuffle indices of the training dataset.

trainingsetindex: int or str, optional, default=0

Integer specifying which “TrainingsetFraction” to use. Note that “TrainingFraction” is a list in config.yaml. This variable can also be set to “all”.

plotting: bool or str, optional, default=False

Plots the predictions on the train and test images. If provided it must be either True, False, "bodypart", or "individual". Setting to True defaults as "bodypart" for multi-animal projects. If a detector is used, the predicted bounding boxes will also be plotted.

show_errors: bool, optional, default=True

Display train and test errors.

comparisonbodyparts: str or list, optional, default=”all”

The average error will be computed for those body parts only. The provided list has to be a subset of the defined body parts.

gputouse: int or None, optional, default=None

Indicates the GPU to use (see number in nvidia-smi). If you do not have a GPU put None`. See: https://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries

rescale: bool, optional, default=False

Evaluate the model at the 'global_scale' variable (as set in the pose_config.yaml file for a particular project). I.e. every image will be resized according to that scale and prediction will be compared to the resized ground truth. The error will be reported in pixels at rescaled to the original size. I.e. For a [200,200] pixel image evaluated at global_scale=.5, the predictions are calculated on [100,100] pixel images, compared to 1/2*ground truth and this error is then multiplied by 2!. The evaluation images are also shown for the original size!

modelprefix: str, optional, default=””

Directory containing the deeplabcut models to use when evaluating the network. By default, the models are assumed to exist in the project folder.

per_keypoint_evaluation: bool, default=False

Compute the train and test RMSE for each keypoint, and save the results to a {model_name}-keypoint-results.csv in the evaluation-results folder

snapshots_to_evaluate: List[str], optional, default=None

List of snapshot names to evaluate (e.g. [“snapshot-5000”, “snapshot-7500”]).

pcutoff: float | list[float] | dict[str, float] | None, default=None

Only for the PyTorch engine. For the TensorFlow engine, please set the pcutoff in the config.yaml file. The cutoff to use for computing evaluation metrics. When None (default), the cutoff will be loaded from the project config. If a list is provided, there should be one value for each bodypart and one value for each unique bodypart (if there are any). If a dict is provided, the keys should be bodyparts mapping to pcutoff values for each bodypart. Bodyparts that are not defined in the dict will have pcutoff set to 0.6.

engine: Engine, optional, default = None.

The default behavior loads the engine for the shuffle from the metadata. You can overwrite this by passing the engine as an argument, but this should generally not be done.

torch_kwargs:

You can add any keyword arguments for the deeplabcut.pose_estimation_pytorch evaluate_network function here. These arguments are passed to the downstream function. Available parameters are snapshotindex, which overrides the snapshotindex parameter in the project configuration file. For top-down models the detector_snapshot_index parameter can override the index of the detector to use for evaluation in the project configuration file.

Returns:

None

Examples

If you do not want to plot and evaluate with shuffle set to 1.

>>> deeplabcut.evaluate_network(
        '/analysis/project/reaching-task/config.yaml', Shuffles=[1],
    )

If you want to plot and evaluate with shuffle set to 0 and 1.

>>> deeplabcut.evaluate_network(
        '/analysis/project/reaching-task/config.yaml',
        Shuffles=[0, 1],
        plotting=True,
    )

If you want to plot assemblies for a maDLC project

>>> deeplabcut.evaluate_network(
        '/analysis/project/reaching-task/config.yaml',
        Shuffles=[1],
        plotting="individual",
    )

If you have a PyTorch model for which you want to set a different p-cutoff for “left_ear” and “right_ear” bodyparts, and keep the one set in the project config for other bodyparts:

>>> deeplabcut.evaluate_network(
>>>     "/analysis/project/reaching-task/config.yaml",
>>>     Shuffles=[0, 1],
>>>     pcutoff={"left_ear": 0.8, "right_ear": 0.8},
>>> )

Note: This defaults to standard plotting for single-animal projects.

deeplabcut.analyze_videos(
    config_path, ["fullpath/analysis/project/videos/reachingvideo1.avi"],
    save_as_csv=True
)

deeplabcut.analyze_videos(
    config_path,
    videos,
    videotype="avi",
    shuffle=1,
    trainingsetindex=0,
    gputouse=None,
    save_as_csv=False,
    destfolder=None,
    dynamic=(True, .5, 10)
)

Click the button to see API Docs

deeplabcut.compat.analyze_videos(config: str, videos: list[str], videotype: str = '', shuffle: int = 1, trainingsetindex: int = 0, gputouse: str | None = None, save_as_csv: bool = False, in_random_order: bool = True, destfolder: str | None = None, batchsize: int | None = None, cropping: list[int] | None = None, TFGPUinference: bool = True, dynamic: tuple[bool, float, int] = (False, 0.5, 10), modelprefix: str = '', robust_nframes: bool = False, allow_growth: bool = False, use_shelve: bool = False, auto_track: bool = True, n_tracks: int | None = None, animal_names: list[str] | None = None, calibrate: bool = False, identity_only: bool = False, use_openvino: str | None = None, engine: Engine | None = None, **torch_kwargs)#

Makes prediction based on a trained network.

The index of the trained network is specified by parameters in the config file (in particular the variable ‘snapshotindex’).

The labels are stored as MultiIndex Pandas Array, which contains the name of the network, body part name, (x, y) label position in pixels, and the likelihood for each frame per body part. These arrays are stored in an efficient Hierarchical Data Format (HDF) in the same directory where the video is stored. However, if the flag save_as_csv is set to True, the data can also be exported in comma-separated values format (.csv), which in turn can be imported in many programs, such as MATLAB, R, Prism, etc.

Parameters:

config: str

Full path of the config.yaml file.

videos: list[str]

A list of strings containing the full paths to videos for analysis or a path to the directory, where all the videos with same extension are stored.

videotype: str, optional, default=””

Checks for the extension of the video in case the input to the video is a directory. Only videos with this extension are analyzed. If left unspecified, videos with common extensions (‘avi’, ‘mp4’, ‘mov’, ‘mpeg’, ‘mkv’) are kept.

shuffle: int, optional, default=1

An integer specifying the shuffle index of the training dataset used for training the network.

trainingsetindex: int, optional, default=0

Integer specifying which TrainingsetFraction to use. By default the first (note that TrainingFraction is a list in config.yaml).

gputouse: int or None, optional, default=None

Only for the TensorFlow engine (for the PyTorch engine see the torch_kwargs: you can use device). Indicates the GPU to use (see number in nvidia-smi). If you do not have a GPU put None. See: https://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries

save_as_csv: bool, optional, default=False

Saves the predictions in a .csv file.

in_random_order: bool, optional (default=True)

Whether or not to analyze videos in a random order. This is only relevant when specifying a video directory in videos.

destfolder: string or None, optional, default=None

Specifies the destination folder for analysis data. If None, the path of the video is used. Note that for subsequent analysis this folder also needs to be passed.

batchsize: int or None, optional, default=None

Currently not supported by the PyTorch engine. Change batch size for inference; if given overwrites value in pose_cfg.yaml.

cropping: list or None, optional, default=None

List of cropping coordinates as [x1, x2, y1, y2]. Note that the same cropping parameters will then be used for all videos. If different video crops are desired, run analyze_videos on individual videos with the corresponding cropping coordinates.

TFGPUinference: bool, optional, default=True

Only for the TensorFlow engine. Perform inference on GPU with TensorFlow code. Introduced in “Pretraining boosts out-of-domain robustness for pose estimation” by Alexander Mathis, Mert Yüksekgönül, Byron Rogers, Matthias Bethge, Mackenzie W. Mathis. Source: https://arxiv.org/abs/1909.11229

dynamic: tuple(bool, float, int) triple containing (state, det_threshold, margin)

If the state is true, then dynamic cropping will be performed. That means that if an object is detected (i.e. any body part > detectiontreshold), then object boundaries are computed according to the smallest/largest x position and smallest/largest y position of all body parts. This window is expanded by the margin and from then on only the posture within this crop is analyzed (until the object is lost, i.e. <detectiontreshold). The current position is utilized for updating the crop window for the next frame (this is why the margin is important and should be set large enough given the movement of the animal).

modelprefix: str, optional, default=””

Directory containing the deeplabcut models to use when evaluating the network. By default, the models are assumed to exist in the project folder.

robust_nframes: bool, optional, default=False

Evaluate a video’s number of frames in a robust manner. This option is slower (as the whole video is read frame-by-frame), but does not rely on metadata, hence its robustness against file corruption.

allow_growth: bool, optional, default=False.

Only for the TensorFlow engine. For some smaller GPUs the memory issues happen. If True, the memory allocator does not pre-allocate the entire specified GPU memory region, instead starting small and growing as needed. See issue: https://forum.image.sc/t/how-to-stop-running-out-of-vram/30551/2

use_shelve: bool, optional, default=False

By default, data are dumped in a pickle file at the end of the video analysis. Otherwise, data are written to disk on the fly using a “shelf”; i.e., a pickle-based, persistent, database-like object by default, resulting in constant memory footprint.

The following parameters are only relevant for multi-animal projects:

auto_track: bool, optional, default=True

By default, tracking and stitching are automatically performed, producing the final h5 data file. This is equivalent to the behavior for single-animal projects.

If False, one must run convert_detections2tracklets and stitch_tracklets afterwards, in order to obtain the h5 file.

This function has 3 related sub-calls:

identity_only: bool, optional, default=False

If True and animal identity was learned by the model, assembly and tracking rely exclusively on identity prediction.

calibrate: bool, optional, default=False

If True, use training data to calibrate the animal assembly procedure. This improves its robustness to wrong body part links, but requires very little missing data.

n_tracks: int or None, optional, default=None

Number of tracks to reconstruct. By default, taken as the number of individuals defined in the config.yaml. Another number can be passed if the number of animals in the video is different from the number of animals the model was trained on.

animal_names: list[str], optional

If you want the names given to individuals in the labeled data file, you can specify those names as a list here. If given and n_tracks is None, n_tracks will be set to len(animal_names). If n_tracks is not None, then it must be equal to len(animal_names). If it is not given, then animal_names will be loaded from the individuals in the project config.yaml file.

use_openvino: str, optional

Only for the TensorFlow engine. Use “CPU” for inference if OpenVINO is available in the Python environment.

engine: Engine, optional, default = None.

The default behavior loads the engine for the shuffle from the metadata. You can overwrite this by passing the engine as an argument, but this should generally not be done.

torch_kwargs:

Any extra parameters to pass to the PyTorch API, such as device which can be used to specify the CUDA device to use for training.

Returns:

DLCScorer: str

the scorer used to analyze the videos

Examples

Analyzing a single video on Windows

>>> deeplabcut.analyze_videos(
        'C:\myproject\reaching-task\config.yaml',
        ['C:\yourusername\rig-95\Videos\reachingvideo1.avi'],
    )

Analyzing a single video on Linux/MacOS

>>> deeplabcut.analyze_videos(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/videos/reachingvideo1.avi'],
    )

Analyze all videos of type avi in a folder

>>> deeplabcut.analyze_videos(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/videos'],
        videotype='.avi',
    )

Analyze multiple videos

>>> deeplabcut.analyze_videos(
        '/analysis/project/reaching-task/config.yaml',
        [
            '/analysis/project/videos/reachingvideo1.avi',
            '/analysis/project/videos/reachingvideo2.avi',
        ],
    )

Analyze multiple videos with shuffle=2

>>> deeplabcut.analyze_videos(
        '/analysis/project/reaching-task/config.yaml',
        [
            '/analysis/project/videos/reachingvideo1.avi',
            '/analysis/project/videos/reachingvideo2.avi',
        ],
        shuffle=2,
    )

Analyze multiple videos with shuffle=2, save results as an additional csv file

>>> deeplabcut.analyze_videos(
        '/analysis/project/reaching-task/config.yaml',
        [
            '/analysis/project/videos/reachingvideo1.avi',
            '/analysis/project/videos/reachingvideo2.avi',
        ],
        shuffle=2,
        save_as_csv=True,
    )

dynamic: triple containing (state, detectiontreshold, margin)

    If the state is true, then dynamic cropping will be performed.
    That means that if an object is detected (i.e., any body part > detectiontreshold),
    then object boundaries are computed according to the smallest/largest x position and
    smallest/largest y position of all body parts. This window is expanded by the margin
    and from then on only the posture within this crop is analyzed (until the object is lost;
    i.e., < detectiontreshold). The current position is utilized for updating the crop window
    for the next frame (this is why the margin is important and should be set large enough
    given the movement of the animal).

deeplabcut.filterpredictions(
    config_path,
    ["fullpath/analysis/project/videos/reachingvideo1.avi"]
)

deeplabcut.filterpredictions(
    config_path,
    ["fullpath/analysis/project/videos"],
    videotype=".mp4",
    filtertype="arima",
    ARdegree=5,
    MAdegree=2
)

deeplabcut.filterpredictions(
    config_path,
    ["fullpath/analysis/project/videos/reachingvideo1.avi"],
    shuffle=1,
    trainingsetindex=0,
    filtertype="arima",
    p_bound=0.01,
    ARdegree=3,
    MAdegree=1,
    alpha=0.01
)

Click the button to see API Docs

deeplabcut.post_processing.filtering.filterpredictions(config, video, videotype='', shuffle=1, trainingsetindex=0, filtertype='median', windowlength=5, p_bound=0.001, ARdegree=3, MAdegree=1, alpha=0.01, save_as_csv=True, destfolder=None, modelprefix='', track_method='', return_data=False, **kwargs)#

Fits frame-by-frame pose predictions.

The pose predictions are fitted with ARIMA model (filtertype=’arima’) or median filter (default).

Parameters:

configstring

Full path of the config.yaml file.

videostring

Full path of the video to extract the frame from. Make sure that this video is already analyzed.

shuffleint, optional, default=1

The shuffle index of training dataset. The extracted frames will be stored in the labeled-dataset for the corresponding shuffle of training dataset.

trainingsetindex: int, optional, default=0

Integer specifying which TrainingsetFraction to use. Note that TrainingFraction is a list in config.yaml.

filtertype: string, optional, default=”median”.

The filter type - ‘arima’, ‘median’ or ‘spline’.

windowlength: int, optional, default=5

For filtertype=’median’ filters the input array using a local window-size given by windowlength. The array will automatically be zero-padded. https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.medfilt.html. The windowlenght should be an odd number. If filtertype=’spline’, windowlength is the maximal gap size to fill.

p_bound: float between 0 and 1, optional, default=0.001

For filtertype ‘arima’ this parameter defines the likelihood below, below which a body part will be consided as missing data for filtering purposes.

ARdegree: int, optional, default=3

For filtertype ‘arima’ Autoregressive degree of Sarimax model degree. see https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.sarimax.SARIMAX.html

MAdegree: int, optional, default=1

For filtertype ‘arima’ Moving Average degree of Sarimax model degree. See https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.sarimax.SARIMAX.html

alpha: float, optional, default=0.01

Significance level for detecting outliers based on confidence interval of fitted SARIMAX model.

save_as_csv: bool, optional, default=True

Saves the predictions in a .csv file.

destfolder: string, optional, default=None

Specifies the destination folder for analysis data. If None, the path of the video is used by default. Note that for subsequent analysis this folder also needs to be passed.

modelprefix: str, optional, default=””

Directory containing the deeplabcut models to use when evaluating the network. By default, the models are assumed to exist in the project folder.

track_method: string, optional, default=””

Specifies the tracker used to generate the data. Empty by default (corresponding to a single animal project). For multiple animals, must be either ‘box’, ‘skeleton’, or ‘ellipse’ and will be taken from the config.yaml file if none is given.

return_data: bool, optional, default=False

If True, returns a dictionary of the filtered data keyed by video names.

kwargs: additional arguments.

For torch-based shuffles, can be used to specify:

• snapshot_index

• detector_snapshot_index

Returns:

video_to_filtered_df

Dictionary mapping video filepaths to filtered dataframes.

• If no videos exist, the dictionary will be empty.

• If a video is not analyzed, the corresponding value in the dictionary will be None.

Examples

Arima model:

>>> deeplabcut.filterpredictions(
        'C:\myproject\reaching-task\config.yaml',
        ['C:\myproject\trailtracking-task\test.mp4'],
        shuffle=3,
        filterype='arima',
        ARdegree=5,
        MAdegree=2,
    )

Use median filter over 10 bins:

>>> deeplabcut.filterpredictions(
        'C:\myproject\reaching-task\config.yaml',
        ['C:\myproject\trailtracking-task\test.mp4'],
        shuffle=3,
        windowlength=10,
    )

One can then use the filtered rather than the frame-by-frame predictions by calling:

>>> deeplabcut.plot_trajectories(
        'C:\myproject\reaching-task\config.yaml',
        ['C:\myproject\trailtracking-task\test.mp4'],
        shuffle=3,
        filtered=True,
    )

>>> deeplabcut.create_labeled_video(
        'C:\myproject\reaching-task\config.yaml',
        ['C:\myproject\trailtracking-task\test.mp4'],
        shuffle=3,
        filtered=True,
    )

deeplabcut.plot_trajectories(config_path, [‘fullpath/analysis/project/videos/reachingvideo1.avi’])

Click the button to see API Docs

deeplabcut.utils.plotting.plot_trajectories(config, videos, videotype='', shuffle=1, trainingsetindex=0, filtered=False, displayedbodyparts='all', displayedindividuals='all', showfigures=False, destfolder=None, modelprefix='', imagetype='.png', resolution=100, linewidth=1.0, track_method='', pcutoff: float | None = None, **kwargs)#

Plots the trajectories of various bodyparts across the video.

Parameters:

config: str

Full path of the config.yaml file.

videos: list[str]

Full paths to videos for analysis or a path to the directory, where all the videos with same extension are stored.

videotype: str, optional, default=””

Checks for the extension of the video in case the input to the video is a directory. Only videos with this extension are analyzed. If left unspecified, videos with common extensions (‘avi’, ‘mp4’, ‘mov’, ‘mpeg’, ‘mkv’) are kept.

shuffle: int, optional, default=1

Integer specifying the shuffle index of the training dataset.

trainingsetindex: int, optional, default=0

Integer specifying which TrainingsetFraction to use. Note that TrainingFraction is a list in config.yaml.

filtered: bool, optional, default=False

Boolean variable indicating if filtered output should be plotted rather than frame-by-frame predictions. Filtered version can be calculated with deeplabcut.filterpredictions.

displayedbodyparts: list[str] or str, optional, default=”all”

This select the body parts that are plotted in the video. Either all, then all body parts from config.yaml are used, or a list of strings that are a subset of the full list. E.g. [‘hand’,’Joystick’] for the demo Reaching-Mackenzie-2018-08-30/config.yaml to select only these two body parts.

showfigures: bool, optional, default=False

If True then plots are also displayed.

destfolder: string or None, optional, default=None

Specifies the destination folder that was used for storing analysis data. If None, the path of the video is used.

modelprefix: str, optional, default=””

Directory containing the deeplabcut models to use when evaluating the network. By default, the models are assumed to exist in the project folder.

imagetype: string, optional, default=”.png”

Specifies the output image format - ‘.tif’, ‘.jpg’, ‘.svg’ and “.png”.

resolution: int, optional, default=100

Specifies the resolution (in dpi) of saved figures. Note higher resolution figures take longer to generate.

linewidth: float, optional, default=1.0

Specifies width of line for line and histogram plots.

track_method: string, optional, default=””

Specifies the tracker used to generate the data. Empty by default (corresponding to a single animal project). For multiple animals, must be either ‘box’, ‘skeleton’, or ‘ellipse’ and will be taken from the config.yaml file if none is given.

pcutoff: string, optional, default=None

Overrides the pcutoff set in the project configuration to plot the trajectories.

kwargs: additional arguments.

For torch-based shuffles, can be used to specify:

• snapshot_index

• detector_snapshot_index

Returns:

None

Examples

To label the frames

>>> deeplabcut.plot_trajectories(
        'home/alex/analysis/project/reaching-task/config.yaml',
        ['/home/alex/analysis/project/videos/reachingvideo1.avi'],
    )

deeplabcut.create_labeled_video(
    config_path,
    ["fullpath/analysis/project/videos/reachingvideo1.avi",
     "fullpath/analysis/project/videos/reachingvideo2.avi"],
    save_frames = True/False
)

deeplabcut.create_labeled_video(
    config_path,
    ["fullpath/afolderofvideos"],
    videotype=".mp4",
    filtered=True
)

deeplabcut.create_labeled_video(
    config_path,
    ["fullpath/afolderofvideos"],
    videotype=".mp4",
    trailpoints=10
)

# Plotting configuration
skeleton:
  - ["snout", "leftear"]
  - ["snout", "rightear"]
  - ["leftear", "tailbase"]
  - ["leftear", "rightear"]
  - ["rightear", "tailbase"]
skeleton_color: white
pcutoff: 0.4
dotsize: 4
alphavalue: 0.5
colormap: jet

deeplabcut.create_labeled_video(
    config_path,
    ["fullpath/afolderofvideos"],
    videotype=".mp4",
    draw_skeleton=True
)

deeplabcut.create_labeled_video(
    config_path,["fullpath/afolderofvideos"],
    videotype=".mp4",
    keypoints_only=True
)

Click the button to see API Docs

deeplabcut.utils.make_labeled_video.create_labeled_video(config: str, videos: list[str], videotype: str = '', shuffle: int = 1, trainingsetindex: int = 0, filtered: bool = False, fastmode: bool = True, save_frames: bool = False, keypoints_only: bool = False, Frames2plot: list[int] | None = None, displayedbodyparts: list[str] | str = 'all', displayedindividuals: list[str] | str = 'all', codec: str = 'mp4v', outputframerate: int | None = None, destfolder: str | Path | None = None, draw_skeleton: bool = False, trailpoints: int = 0, displaycropped: bool = False, color_by: str = 'bodypart', modelprefix: str = '', init_weights: str = '', track_method: str = '', superanimal_name: str = '', pcutoff: float | None = None, skeleton: list = [], skeleton_color: str = 'white', dotsize: int = 8, colormap: str = 'rainbow', alphavalue: float = 0.5, overwrite: bool = False, confidence_to_alpha: bool | Callable[[float], float] = False, plot_bboxes: bool = True, bboxes_pcutoff: float | None = None, max_workers: int | None = None, **kwargs)#

Labels the bodyparts in a video.

Make sure the video is already analyzed by the function deeplabcut.analyze_videos.

Parameters:

configstring

Full path of the config.yaml file.

videoslist[str]

A list of strings containing the full paths to videos for analysis or a path to the directory, where all the videos with same extension are stored.

videotype: str, optional, default=””

Checks for the extension of the video in case the input to the video is a directory. Only videos with this extension are analyzed. If left unspecified, videos with common extensions (‘avi’, ‘mp4’, ‘mov’, ‘mpeg’, ‘mkv’) are kept.

shuffleint, optional, default=1

Number of shuffles of training dataset.

trainingsetindex: int, optional, default=0

Integer specifying which TrainingsetFraction to use. Note that TrainingFraction is a list in config.yaml.

filtered: bool, optional, default=False

Boolean variable indicating if filtered output should be plotted rather than frame-by-frame predictions. Filtered version can be calculated with deeplabcut.filterpredictions.

fastmode: bool, optional, default=True

If True, uses openCV (much faster but less customization of video) instead of matplotlib if False. You can also “save_frames” individually or not in the matplotlib mode (if you set the “save_frames” variable accordingly). However, using matplotlib to create the frames it therefore allows much more flexible (one can set transparency of markers, crop, and easily customize).

save_frames: bool, optional, default=False

If True, creates each frame individual and then combines into a video. Setting this to True is relatively slow as it stores all individual frames.

keypoints_only: bool, optional, default=False

By default, both video frames and keypoints are visible. If True, only the keypoints are shown. These clips are an hommage to Johansson movies, see https://www.youtube.com/watch?v=1F5ICP9SYLU and of course his seminal paper: “Visual perception of biological motion and a model for its analysis” by Gunnar Johansson in Perception & Psychophysics 1973.

Frames2plot: List[int] or None, optional, default=None

If not None and save_frames=True then the frames corresponding to the index will be plotted. For example, Frames2plot=[0,11] will plot the first and the 12th frame.

displayedbodyparts: list[str] or str, optional, default=”all”

This selects the body parts that are plotted in the video. If all, then all body parts from config.yaml are used. If a list of strings that are a subset of the full list. E.g. [‘hand’,’Joystick’] for the demo Reaching-Mackenzie-2018-08-30/config.yaml to select only these body parts.

displayedindividuals: list[str] or str, optional, default=”all”

Individuals plotted in the video. By default, all individuals present in the config will be shown.

codec: str, optional, default=”mp4v”

Codec for labeled video. For available options, see http://www.fourcc.org/codecs.php. Note that this depends on your ffmpeg installation.

outputframerate: int or None, optional, default=None

Positive number, output frame rate for labeled video (only available for the mode with saving frames.) If None, which results in the original video rate.

destfolder: Path, string or None, optional, default=None

Specifies the destination folder that was used for storing analysis data. If None, the path of the video file is used.

draw_skeleton: bool, optional, default=False

If True adds a line connecting the body parts making a skeleton on each frame. The body parts to be connected and the color of these connecting lines are specified in the config file.

trailpoints: int, optional, default=0

Number of previous frames whose body parts are plotted in a frame (for displaying history).

displaycropped: bool, optional, default=False

Specifies whether only cropped frame is displayed (with labels analyzed therein), or the original frame with the labels analyzed in the cropped subset.

color_bystring, optional, default=’bodypart’

Coloring rule. By default, each bodypart is colored differently. If set to ‘individual’, points belonging to a single individual are colored the same.

modelprefix: str, optional, default=””

Directory containing the deeplabcut models to use when evaluating the network. By default, the models are assumed to exist in the project folder.

init_weights: str,

Checkpoint path to the super model

track_method: string, optional, default=””

Specifies the tracker used to generate the data. Empty by default (corresponding to a single animal project). For multiple animals, must be either ‘box’, ‘skeleton’, or ‘ellipse’ and will be taken from the config.yaml file if none is given.

superanimal_name: str, optional, default=””

Name of the superanimal model.

pcutoff: float, optional, default=None

Overrides the pcutoff set in the project configuration to plot the trajectories.

skeleton: list, optional, default=[],

skeleton_color: string, optional, default=”white”,

Color for the skeleton

dotsize, int, optional, default=8,

Size of label dots tu use

colormap: str, optional, default=”rainbow”,

Colormap to use for the labels

alphavalue: float, optional, default=0.5,

overwrite: bool, optional, default=False

If True overwrites existing labeled videos.

confidence_to_alpha: Union[bool, Callable[[float], float], default=False

If False, all keypoints will be plot with alpha=1. Otherwise, this can be defined as a function f: [0, 1] -> [0, 1] such that the alpha value for a keypoint will be set as a function of its score: alpha = f(score). The default function used when True is f(x) = max(0, (x - pcutoff)/(1 - pcutoff)).

plot_bboxes: bool, optional, default=True

If using Pytorch and in Top-Down mode, setting this to true will also plot the bounding boxes

bboxes_pcutoff, float, optional, default=None:

If plotting bounding boxes, this overrides the bboxes_pcutoff set in the model configuration.

max_workers (int | None):

Maximum number of processes to use for multiprocessing. Set this parameter to limit the total RAM-usage of simultaneous processes. Default: no maximum (i.e. number of spawned processes is based on the number of cores and the number of input videos).

kwargs: additional arguments.

For torch-based shuffles, can be used to specify:

• snapshot_index

• detector_snapshot_index

Returns:

resultslist[bool]

True if the video is successfully created for each item in videos.

Examples

Create the labeled video for a single video

>>> deeplabcut.create_labeled_video(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/videos/reachingvideo1.avi'],
    )

Create the labeled video for a single video and store the individual frames

>>> deeplabcut.create_labeled_video(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/videos/reachingvideo1.avi'],
        fastmode=True,
        save_frames=True,
    )

Create the labeled video for multiple videos

>>> deeplabcut.create_labeled_video(
        '/analysis/project/reaching-task/config.yaml',
        [
            '/analysis/project/videos/reachingvideo1.avi',
            '/analysis/project/videos/reachingvideo2.avi',
        ],
    )

Create the labeled video for all the videos with an .avi extension in a directory.

>>> deeplabcut.create_labeled_video(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/videos/'],
    )

Create the labeled video for all the videos with an .mp4 extension in a directory.

>>> deeplabcut.create_labeled_video(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/videos/'],
        videotype='mp4',
    )

Click the button to see API Docs

deeplabcut.post_processing.analyze_skeleton.analyzeskeleton(config, videos, videotype='', shuffle=1, trainingsetindex=0, filtered=False, save_as_csv=False, destfolder=None, modelprefix='', track_method='', return_data=False, **kwargs)#

Extracts length and orientation of each “bone” of the skeleton.

The bone and skeleton information is defined in the config file.

Parameters:

config: str

Full path of the config.yaml file.

videos: list[str]

The full paths to videos for analysis or a path to the directory, where all the videos with same extension are stored.

videotype: str, optional, default=””

Checks for the extension of the video in case the input to the video is a directory. Only videos with this extension are analyzed. If left unspecified, videos with common extensions (‘avi’, ‘mp4’, ‘mov’, ‘mpeg’, ‘mkv’) are kept.

shuffleint, optional, default=1

The shuffle index of training dataset. The extracted frames will be stored in the labeled-dataset for the corresponding shuffle of training dataset.

trainingsetindex: int, optional, default=0

Integer specifying which TrainingsetFraction to use. Note that TrainingFraction is a list in config.yaml.

filtered: bool, optional, default=False

Boolean variable indicating if filtered output should be plotted rather than frame-by-frame predictions. Filtered version can be calculated with deeplabcut.filterpredictions.

save_as_csv: bool, optional, default=False

Saves the predictions in a .csv file.

destfolder: string or None, optional, default=None

Specifies the destination folder for analysis data. If None, the path of the video is used. Note that for subsequent analysis this folder also needs to be passed.

modelprefix: str, optional, default=””

Directory containing the deeplabcut models to use when evaluating the network. By default, the models are assumed to exist in the project folder.

track_method: string, optional, default=””

Specifies the tracker used to generate the data. Empty by default (corresponding to a single animal project). For multiple animals, must be either ‘box’, ‘skeleton’, or ‘ellipse’ and will be taken from the config.yaml file if none is given.

return_data: bool, optional, default=False

If True, returns a dictionary of the filtered data keyed by video names.

kwargs: additional arguments.

For torch-based shuffles, can be used to specify:

• snapshot_index

• detector_snapshot_index

Returns:

video_to_skeleton_df

Dictionary mapping video filepaths to skeleton dataframes.

• If no videos exist, the dictionary will be empty.

• If a video is not analyzed, the corresponding value in the dictionary will be None.

Click the button to see API Docs

Extracts the outlier frames.

Extracts the outlier frames if the predictions are not correct for a certain video from the cropped video running from start to stop as defined in config.yaml.

Another crucial parameter in config.yaml is how many frames to extract numframes2extract.

Parameters:

config: str

Full path of the config.yaml file.

videoslist[str]

The full paths to videos for analysis or a path to the directory, where all the videos with same extension are stored.

videotype: str, optional, default=””

Checks for the extension of the video in case the input to the video is a directory. Only videos with this extension are analyzed. If left unspecified, videos with common extensions (‘avi’, ‘mp4’, ‘mov’, ‘mpeg’, ‘mkv’) are kept.

shuffleint, optional, default=1

The shuffle index of training dataset. The extracted frames will be stored in the labeled-dataset for the corresponding shuffle of training dataset.

trainingsetindex: int, optional, default=0

Integer specifying which TrainingsetFraction to use. Note that TrainingFraction is a list in config.yaml.

outlieralgorithm: str, optional, default=”jump”.

String specifying the algorithm used to detect the outliers.

• 'fitting' fits an Auto Regressive Integrated Moving Average model to the data and computes the distance to the estimated data. Larger distances than epsilon are then potentially identified as outliers

• 'jump' identifies larger jumps than ‘epsilon’ in any body part

• 'uncertain' looks for frames with confidence below p_bound

• 'manual' launches a GUI from which the user can choose the frames

• 'list' looks for user to provide a list of frame numbers to use, ‘frames2use’. In this case, 'extractionalgorithm' is forced to be 'uniform.'

frames2use: list[str], optional, default=None

If 'outlieralgorithm' is 'list', provide the list of frames here.

comparisonbodyparts: list[str] or str, optional, default=”all”

This selects the body parts for which the comparisons with the outliers are carried out. If "all", then all body parts from config.yaml are used. If a list of strings that are a subset of the full list E.g. [‘hand’,’Joystick’] for the demo Reaching-Mackenzie-2018-08-30/config.yaml to select only these body parts.

p_bound: float between 0 and 1, optional, default=0.01

For outlieralgorithm 'uncertain' this parameter defines the likelihood below which a body part will be flagged as a putative outlier.

epsilon: float, optional, default=20

If 'outlieralgorithm' is 'fitting', this is the float bound according to which frames are picked when the (average) body part estimate deviates from model fit.

If 'outlieralgorithm' is 'jump', this is the float bound specifying the distance by which body points jump from one frame to next (Euclidean distance).

ARdegree: int, optional, default=3

For outlieralgorithm 'fitting': Autoregressive degree of ARIMA model degree. (Note we use SARIMAX without exogeneous and seasonal part) See https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.sarimax.SARIMAX.html

MAdegree: int, optional, default=1

For outlieralgorithm 'fitting': Moving Average degree of ARIMA model degree. (Note we use SARIMAX without exogeneous and seasonal part) See https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.sarimax.SARIMAX.html

alpha: float, optional, default=0.01

Significance level for detecting outliers based on confidence interval of fitted ARIMA model. Only the distance is used however.

extractionalgorithmstr, optional, default=”kmeans”

String specifying the algorithm to use for selecting the frames from the identified putatative outlier frames. Currently, deeplabcut supports either kmeans or uniform based selection (same logic as for extract_frames).

automaticbool, optional, default=False

If True, extract outliers without being asked for user feedback.

cluster_resizewidth: number, default=30

If "extractionalgorithm" is "kmeans", one can change the width to which the images are downsampled (aspect ratio is fixed).

cluster_color: bool, optional, default=False

If False, each downsampled image is treated as a grayscale vector (discarding color information). If True, then the color channels are considered. This increases the computational complexity.

opencv: bool, optional, default=True

Uses openCV for loading & extractiong (otherwise moviepy (legacy)).

savelabeled: bool, optional, default=False

If True, frame are saved with predicted labels in each folder.

copy_videos: bool, optional, default=False

If True, newly-added videos (from which outlier frames are extracted) are copied to the project folder. By default, symbolic links are created instead.

destfolder: str or None, optional, default=None

Specifies the destination folder that was used for storing analysis data. If None, the path of the video is used.

modelprefix: str, optional, default=””

Directory containing the deeplabcut models to use when evaluating the network. By default, the models are assumed to exist in the project folder.

track_method: str, optional, default=””

Specifies the tracker used to generate the data. Empty by default (corresponding to a single animal project). For multiple animals, must be either ‘box’, ‘skeleton’, or ‘ellipse’ and will be taken from the config.yaml file if none is given.

kwargs: additional arguments.

For torch-based shuffles, can be used to specify:

• snapshot_index

• detector_snapshot_index

Returns:

None

Examples

Extract the frames with default settings on Windows.

>>> deeplabcut.extract_outlier_frames(
        'C:\myproject\reaching-task\config.yaml',
        ['C:\yourusername\rig-95\Videos\reachingvideo1.avi'],
    )

Extract the frames with default settings on Linux/MacOS.

>>> deeplabcut.extract_outlier_frames(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/video/reachinvideo1.avi'],
    )

Extract the frames using the “kmeans” algorithm.

>>> deeplabcut.extract_outlier_frames(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/video/reachinvideo1.avi'],
        extractionalgorithm='kmeans',
    )

Extract the frames using the “kmeans” algorithm and "epsilon=5" pixels.

>>> deeplabcut.extract_outlier_frames(
        '/analysis/project/reaching-task/config.yaml',
        ['/analysis/project/video/reachinvideo1.avi'],
        epsilon=5,
        extractionalgorithm='kmeans',
    )

deeplabcut.refine_labels(config_path)

deeplabcut.merge_datasets(config_path)

Click the button to see API Docs

deeplabcut.gui.tabs.label_frames.refine_labels(config_path: str | Path | None = None, image_folder: str | None = None)#

Launches the napari-deeplabcut labelling GUI.

For more information on labelling data with napari-deeplabcut, see our docs:

DeepLabCut/napari-deeplabcut

If no parameters are given, the napari-deeplabcut labelling GUI is simply open, and the folder containing the images to label can be dropped into the GUI.

If the config_path and the image_folder are given as arguments, the given image_folder for the project is opened in the napari-deeplabcut GUI to be labeled. If only the config_path is given, the first image folder is opened.

Parameters:

config_path: str, Path, None

Full path of the project config.yaml file.

image_folder: str, None

Name of the image folder to open for labelling.

Examples

Opening the napari-deeplabcut annotation GUI without opening a specific folder of images to label. You then need to drag-and-drop your image folder into the GUI. See the napari-deeplabcut docs linked above for more information about labelling in napari-deeplabcut. >>> import deeplabcut >>> deeplabcut.label_frames()

Opening the images extracted from the “2025-01-01-experiment7” video in napari-deeplabcut on Windows. The project’s folder structure should look as follows: reaching-task/ # project root directory ├── config.yaml # project configuration file └── labeled-data/ # folder containing all extracted image folders

├── … ├── 2025-01-01-experiment7 # folder containing the images to label └── …

>>> deeplabcut.label_frames(
>>>     "C:\myproject\reaching-task\config.yaml",
>>>     "2025-01-01-experiment7",
>>> )

Opening the images extracted from the first video listed in the project configuration in napari-deeplabcut on a Unix system. >>> deeplabcut.label_frames(“/users/john/project/config.yaml”)

Click the button to see API Docs

deeplabcut.refine_training_dataset.outlier_frames.merge_datasets(config, forceiterate=None)#

Merge the original training dataset with the newly refined data.

Checks if the original training dataset can be merged with the newly refined training dataset. To do so it will check if the frames in all extracted video sets were relabeled.

If this is the case then the "iteration" variable is advanced by 1.

Parameters:

config: str

Full path of the config.yaml file.

forceiterate: int or None, optional, default=None

If an integer is given the iteration variable is set to this value This is only done if all datasets were labeled or refined.

Examples

>>> deeplabcut.merge_datasets('/analysis/project/reaching-task/config.yaml')