CVNets: A library for training computer vision networks

Figure: CVNets can be used to improve the performance of different deep neural networks on the ImageNet dataset significantly with simple training recipes (e.g., random resized cropping and horizontal flipping). The official MobileNetv1  and ResNet results are from TensorflowLite and Torchvision, respectively.

CVNets follows the Following Design Principles

1. Modularity: CVNets provide independent components; allowing users to plug-and-play different components across different visual recognition tasks for both research and production use cases. CVNets implement different components, including datasets and models for different tasks and input modalities, independently. For example, different classification backbones (e.g., ResNet-50) trained in CVNets can be seamlessly integrated with object detection (e.g., SSD) or semantic segmentation (e.g., DeepLabv3) pipelines for studying the generic nature of architecture.
2. Flexibility: With CVNets, we would like to enable new use cases in research as well as production. We designed CVNets such that new components (e.g., models, datasets, loss functions, data samplers, and optimizers) can be integrated easily. We achieve this by registering each component. As an example, the ADE20k dataset for the task of segmentation is registered in CVNets as @register_dataset(name=“ade20k”, task=“segmentation”)     To use this dataset for training, one can use dataset. name and dataset. category as command-line arguments.
3. Reproducibility: CVNets provide reproducible implementations of standard models for different computer vision tasks. Each model is benchmarked against the performance reported in original publications as well as the previous best reproduction studies. The pre-trained weights of each model are released online to enable future research.
4. Compatibility: CVNets are compatible with hardware-accelerated frameworks (e.g., CoreML) and domain-specific libraries (e.g., PyTorchVideo). The models from domain-specific libraries can be easily consumed in the CVNets, as shown in Listing 1; reducing researchers' overhead in implementing new components or sub-modules in CVNets

CVNets Library Components

1. Data Samplers:  CVNets offer data samplers with three sampling strategies:
1. Single-scale with fixed batch size (SSc-FBS): This method is the default sampling strategy in most deep learning frameworks (e.g., PyTorch, Tensorflow, and MixNet) and libraries built on top of them (e.g., the timm library [21]). At the 𝑡-th training iteration, this method samples a batch of 𝑏 images per GPU with a pre-defined spatial resolution of height 𝐻 and width 𝑊.
2. Multi-scale with fixed batch size (MSc-FBS): The SSc-FBS method allows a network to learn representations at a single scale (or resolution). However, objects in the real world are composed at different scales. To allow a network to learn representations at multiple scales, MSc-FBS extends SSc-FBS to multiple scales [16]. Unlike the SSc-FBS method that takes a pre-defined spatial resolution as an input, this method takes a sorted set of 𝑛 spatial resolutions S = {(𝐻1,𝑊1), (𝐻2,𝑊2), · · · , (𝐻𝑛,𝑊𝑛)} as an input. At the 𝑡-th iteration, this method randomly samples 𝑏 images per GPU of spatial resolution (𝐻𝑡 ,𝑊𝑡) ∈ S.
3. Multi-scale with variable batch size (MSc-VBS): Networks trained using the MSc-FBS methods are more robust to scale changes as compared to SSc-FBS [13]. However, depending on the maximum spatial resolution in S, MSc-FBS methods may have a higher peak GPU memory utilization (see Figure 2c) as compared to SSc-FBS; causing out-of-memory errors on GPUs with limited memory. For example, MSc-FBS with S = {(128, 128), (192, 192), (224, 224), (320, 320)} and 𝑏 = 256 would need about 2× more GPU memory (for images only) than SSc-FBS with a spatial resolution of (224, 224) and 𝑏 = 256. To address this memory issue, we extend MSc-FBS to variably-batch sizes in our previous work [13]. For a given sorted set of spatial resolutions S = {(𝐻1,𝑊1), (𝐻2,𝑊2), · · · , (𝐻𝑛,𝑊𝑛)} and a batch size 𝑏 for a maximum spatial resolution of (𝐻𝑛,𝑊𝑛), a spatial resolution (𝐻𝑡 ,𝑊𝑡) ∈ S with a batch size of 𝑏𝑡 = 𝐻𝑛𝑊𝑛𝑏/𝐻𝑡𝑊𝑡 is sampled randomly at 𝑡-th training iteration on each GPU.
2. Sample Efficient Training: Previous works remove and re-weight data samples to reduce optimization updates (or a number of forwarding passes) at the cost of performance degradation. Moreover, these methods are computed- and memory-intensive, and do not scale well to large models and datasets. This work aims to reduce optimization updates with minimal or no performance degradation. Models learn quickly during the initial phase of training. In other words, models can accurately classify many training data samples during earlier epochs and such samples do not contribute much to learning. Therefore, a natural question arises: Can we remove such samples to reduce total optimization updates? CVNets answer this question with a simple heuristic method, which we call Sample Efficient Training (SET). At each epoch, SET categorizes each sample as either hard or easy. To do so, SET uses a simple heuristic: if the model predicts the training data sample correctly with a confidence greater than a pre-defined threshold 𝜏, then it is an easy sample and we remove it from the training data. At each epoch, the model only trains using hard samples. Because of randomness in training due to data augmentation (e.g., random cropping), it is possible that the region of interest corresponding to the object category may be partially (or not) present in the model’s input and easily earlier may be classified harder during the later training stages. SET adds such samples back to the training data (see Figure 3c). Figure 3 shows results for ResNet-50 trained with and without SET using MSc-VBS. ResNet-50 without SET requires 22% more optimization updates while delivering similar performance; demonstrating the effectiveness of SET on top of MSc-VBS. Note that SET has an overhead. Therefore, the reduction in optimization updates does not translate to a reduction in training time. We believe SET can serve as a baseline in this direction and inspire future research to improve training speed while maintaining performance.  
   
   
   

1.     Classification of ImageNet dataset. CVNets implement popular light- and heavy-weight image classification models. The performance of some of these models on the ImageNet dataset is shown in Table 1. With CVNets, we are able to achieve better performance (e.g., MobileNetv1/v2) or similar performance (ResNet-50/101) with fewer optimization updates (faster training). 
2. Similar to image classification, CVNets can be used to train standard detection and segmentation models with better performance. For example, SSD with ResNet-101 backbone trained with CVNets at a resolution of 384 × 384 delivers a 1.6% better mAP than the same model trained at a resolution of 512 × 512. Similarly, on the task of semantic segmentation on the ADE20k dataset using DeepLabv3 with MobileNetv2 as the backbone, CVNets deliver 1.1% better performance than the MMSegmentation library with 2× fewer epochs and optimization updates. For more details, please see our benchmarking results at https://github.com/apple/ml-cvnets.

    git clone git@github.com:apple/ml-cvnets.git
    cd ml-cvnets
    pip install -r requirements.txt
    pip install --editable .

References: