Real-Time Segmentation of Surgical Tools and Needle Using a Mobile-U-Net

This paper presents a CNN-based method for markerless segmentation of surgical tools and suture needles for surgical task automation. The proposed CNN, which we refer to as Mobile-U-Net, is based on the classic U-net encoder-decoder architecture and uses a lightweight MobileNet encoder backbone. This enables the network to perform real-time segmentation at ∼36 and ∼90 frames per second on 672×1120 and 448×448 pixel images. The network is equipped with a multi-kernel output segmentation layer with three k×k×N softmax kernels with k = 1, 3, and 5. On a proprietary dataset from a surgical robot laboratory setup, the multi-kernel Mobile-U-Net achieves intersection over union scores of 0.9495 for surgical tool shafts, 0.8631 for tool end-effectors, 0.9350 for phantom tissue suture pad, 0.8531 for marked needle insertion points, and 0.7524 for suture needles. The method is validated on a second set of images achieving intersection over union scores of 0.9515, 0.8225, and 0.5638 for tool shafts, end-effectors and suture needles. Using multiple kernels improve needle segmentation by 3.11% and 7.00% on the two datasets compared to the baseline of using a single 1×1×N filter and 10.97% and 1.72% for overall mean intersection over union.