PS-VAE is a new semi-supervised framework that finds a low-dimensional, latent representation of animal behavior that is partitioned into two subspaces: a supervised subspace containing the output of supervised pose estimation algorithms (e.g. DeepLabCut) and an orthogonal unsupervised subspace to capture additional variability unaccounted for by explicit labels. The combination of the two provides a richer and more interpretable representation of behavior than either alone.
The NeuroCAAS implementation of BehaveNet works with the PS-VAE version of BehaveNet in addition to the original vanilla VAE model. Examples of the input files listed below may be found in the 'example/' folder of the repo linked under 'Bash Script Link' above.
Main configuration file (.JSON): this MUST be named 'config.json', contains the names of the files for each of the Input file types below, along with settings for the (optional) hyperparameter search
Inputs (more information can be found on https://behavenet.readthedocs.io/en/develop/index.html):
Preprocessed video (.HDF5 file): a previously preprocessed file (note: if this already exists, set raw video and labels to 'null')
Raw video (.MP4 or .AVI): raw video of the animal/experiment (if set, this file will be combined with the labels into a .HDF5 file)
Labels (.CSV): the labels outputted by a pose estimation toolbox such as DeepLabCut (DLC) or DeepGraphPose (DGP)
Autoencoder architecture (.JSON file): specifies settings of the auto encoder architecture (stride, layers, etc.); if null, the default architecture from the PS-VAE paper will be used
Compute settings (.JSON file): specifies computational resources to be used (no user changes needed here!)
BehaveNet model settings (.JSON file): specifies settings of the PS-VAE model, can be switched to a vanilla VAE model through the 'model_class' field
Test parameters (.JSON file): contains metadata about the experiment
Version folder: Each version folder corresponds to a setting of the alpha and beta hyperparameters (values can be seen in filenames)
Model (.PT): the best-performing autoencoder model generated
Latents (.PKL): the chosen latents saved in a pickle format
Label reconstructions (.PDF): graphs comparing predicted latents to originals
Latent traversal movie (.MP4): a latent traversal movie generated for each latent in a grid
Training/validation loss (.PNG): loss for training and validation sets
Metadata (.experiment, .CSV, .PKL): metadata about the experiment (information from JSON files in the Inputs)
Metrics (.CSV): different types of loss for each epoch, batch, trial, etc.
R2/MSE per label/trial (.CSV): measures of loss for labeled data
Session info (.CSV): short metadata (subset of test parameters file from Inputs)
Hyperparameter search results: correlations and MSE per pixel for different values of alpha and beta (see paper for decision process)
PS-VAE training curves: training curves for the best setting of alpha (in filename), across all beta values, and vice versa
You must login to use an analysis.