What is semantic segmentation

Semantic segmentation classifies every pixel in an image. Instead of returning one label for the whole image, the model returns a grid of class scores with a prediction at every image location.

Compare four common computer vision tasks:

Task	Typical output	What it tells the application
Image classification	One list of class scores	Which objects or scenes are likely to appear somewhere in the image
Object detection	Class labels and bounding boxes	Which objects were found and their approximate rectangular locations
Semantic segmentation	One class for every pixel	Which image regions belong to each class
Instance segmentation	A separate mask for each detected object	Which pixels belong to each individual object instance

For example, consider a photograph containing a cat sitting on a chair:

a classifier might report cat and chair, but it does not identify their pixels
a detector might draw one box around the cat and another around the chair
a semantic segmentation model labels cat pixels as cat, chair pixels as chair, and the remaining pixels as background
an instance segmentation model could also distinguish two separate cats, even though both use the same class label

This module uses semantic segmentation. If an image contains two cats, both cats receive the same cat class. The model does not return separate cat objects or object IDs.

Optional information on semantic segmentation

Pascal VOC classes

The pretrained LR-ASPP model uses semantic segmentation categories from the Pascal Visual Object Classes (VOC) dataset. Every output pixel is assigned one of 21 labels: background plus 20 supported object classes.

    

        
        
background   aeroplane    bicycle      bird          boat
bottle       bus          car          cat           chair
cow          diningtable  dog          horse         motorbike
person       pottedplant  sheep        sofa          train
tvmonitor

These are the exact label names used by the model. There is no separate label for other objects such as a phone, laptop, or tree. An unsupported object may be classified as background or confused with one of the supported classes.

From logits to a class mask

A semantic segmentation model returns a tensor with one class-score plane for each output class. NCHW describes the order of the four tensor dimensions:

N — batch: the number of images processed together
C — channels: RGB channels in the input or class-score planes in the output
H — height: the number of pixel rows
W — width: the number of pixel columns

For this model, an input shape of [1, 3, 520, 520] means one image, three RGB channels, 520 rows, and 520 columns. Its output shape of [1, 21, 520, 520] means one result containing 21 class-score planes at the same height and width. Task 1 records these shapes in model_spec.json.

The output values are logits. A logit is a raw model score, not a class name and not the finished cutout. At each output pixel, the application compares 21 logits and selects the class with the largest value:

    

        
        
   output[0, :, y, x]
        |
        +-- batch item 0, all 21 class scores for pixel (x, y)
                    |
                    +-- argmax selects one class ID

If the scores at one pixel are highest for class ID 8, that pixel becomes cat. If the highest score is class ID 0, the pixel becomes background. Repeating this operation for every output location creates a two-dimensional class-ID mask.

From a class mask to a transparent cutout

The segmentation model does not directly return a transparent image. The cutout is an application postprocessing choice:

    

        
        
RGB image
   -> model logits for 21 classes
   -> class ID at every pixel
   -> choose one foreground class
   -> convert that class to an alpha mask
   -> make every other pixel transparent

An alpha mask stores transparency rather than color. In this app, a mask value of 255 keeps a pixel fully opaque and 0 makes it fully transparent.

This application counts the pixels assigned to every non-background class and selects the class with the largest count. Suppose the class mask contains:

Class	Predicted pixels
Background	140,000
Cat	95,000
Chair	30,000
Other classes combined	5,400

The app selects cat and sets the alpha value of all other pixels to zero. The chair becomes transparent even though the model detected it correctly. This behavior keeps the example small and predictable, but it is not the only possible postprocessing strategy.

Another application could let the user choose a class, keep several classes, preserve all foreground pixels, smooth the mask boundary, or use instance segmentation to select one specific object.

Practical limits

Semantic segmentation has practical limits:

unsupported objects may become background or be confused with a supported class
small or thin objects may disappear after resizing
several objects with the same class merge into one semantic region
the largest predicted class is not necessarily the object the user wanted
hard argmax masks can produce rough edges without additional refinement

What is semantic segmentation

Optional information on semantic segmentation

Nice work