Data Study Group Final Report: Keep Wales Tidy and Keep Scotland Beautiful

This data study involved the use of litter datasets to establish a focused proof of concept (PoC) for automatic litter detection that can potentially replace the manual component of the local environmental audit and management system (LEAMS) methodology. Specifically, the study has been carried out using several datasets comprising over 16,800 images of various types and having various levels of annotations. The images were broadly categorised into (1) Images with complete annotation, (2) Images with incomplete annotation, (3) Duplicate images and (4) Non-duplicate images. To better evaluate the datasets, an initial systematic data analysis was performed to identify any potential distinctions and/or similarities between the various subsets of the datasets. In particular, the distribution of the annotations and categories, as well as any potential biases were taken into consideration during the initial systematic data analysis. The initial analysis allowed for a more robust preparation of the datasets for the data study.

Following a review of the literature to identify past, current, and state-of-the-art approaches for the building of object detection models for the automatic detection of litter, several models were built and tested to allow for a comparison of their accuracies and efficiencies under various annotation techniques. The models built include mask R-CNN (region-based convolutional neural network), fast R-CNN, YOLO (you only look once), and DETR (detection transformer) models. The performance metric used for their evaluation in terms of object detection is mainly the mean average precision (mAP) with an IoU (Intersection over Union) threshold of 0.5. It should be noted that the higher the mAP, the more precise and the higher the recall of the object detection model. However, for classification, the F1 score was mostly used because it offers a good balance between precision and recall metrics.

One-class litter detection and six-class classification problems were formulated and addressed. For both problems, DETR with a backbone of ResNet-50 (a 50-layer convolutional neural network with 48 convolutional layers, one MaxPool layer, and one average pool layer) showed the highest mAP of 51.2 and 16.1 for the one-class litter and six-class classification problems, respectively, whereas mask R-CNN showed the least mAP of 27.6 and 6.1 for the one-class litter and six-class classification problems, respectively. It should be noted that due to the brevity of the data study group (DSG) event, hyperparameter tuning or optimisation was neither explored nor exploited by the participants. It should also be noted that aside from the experiments mentioned above, models were also built using YOLO (you only look once) algorithms. The results also indicate that these models can be employed satisfactorily for object detection (i.e., litter detection).