COCO-Stuff 10k - Dataset Ninja

Introduction #

Released 2017-04-06 ·Holger Caesar, Jasper Uijlings, Vittorio Ferrari

The authors of the COCO-Stuff 10k dataset address the distinction between semantic classes, categorizing them as either thing (an object with well-defined shapes such as cars and people) or stuff (amorphous background regions like grass and sky). They noted that while a significant amount of research has focused on “thing” classes, relatively less attention has been devoted to “stuff” classes. They emphasized the importance of “stuff” classes in image understanding, as they play a crucial role in defining scene types, contextual reasoning, and describing physical attributes and geometric properties of scenes.

To promote the understanding of “stuff” and “things” within context, the authors introduced COCO-Stuff, an extension of the COCO 2015 dataset. COCO-Stuff augmented 10K images from COCO 2014. To be compatible with COCO, version 1.1 of COCO-Stuff has 91 thing classes (1-91), 91 stuff classes (92-182) and 1 class “unlabeled” (0). Note that 11 of the thing classes from COCO 2015 do not have any segmentation annotations. The classes desk, door and mirror could be either stuff or things and therefore occur in both COCO and COCO-Stuff. To avoid confusion we add the suffix “-stuff” to those classes in COCO-Stuff.

Furthermore, the authors used COCO-Stuff to analyze various aspects, including the importance of “stuff” and “thing” classes in terms of surface coverage and frequency in image captions, the spatial relationships between “stuff” and “things,” and the performance of modern semantic segmentation methods on these classes.

They underscored the significance of “stuff” classes, emphasizing that they constitute the majority of visual surroundings and provide critical context for recognizing and understanding “things.” “Stuff” classes influence the type of scene and constrain the possible locations of “things.” Additionally, they help determine depth ordering and relative positions of “things” and support the interpretation of relationships between them. The context provided by “stuff” is instrumental in recognizing smaller or less common “things” in images.

The hierarchy of labels:

COCO-Stuff was introduced as a valuable addition to COCO, enabling the exploration of rich relationships between “stuff” and “things” in complex scenes. It was noted that COCO-Stuff offered a significant contribution to complete scene understanding.

Expand

Homepage

Research Paper 1 (main)

Research Paper 2

Summary #

COCO-Stuff 10K Dataset: Common Objects in Context Stuff 10k v1.1 is a dataset for instance segmentation, semantic segmentation, and object detection tasks. It is applicable or relevant across various domains.

The dataset consists of 10000 images with 228313 labeled objects belonging to 183 different classes including unlabeled, person, tree, and other: wall-other, sky-other, grass, building-other, clouds, road, pavement, chair, car, structural-other, dining table, fence, window-other, ground-other, cup, plant-other, bottle, bush, ceiling-other, furniture-other, light, bowl, table, dirt, door-stuff, and 155 more.

Images in the COCO-Stuff 10k dataset have pixel-level instance segmentation annotations. Due to the nature of the instance segmentation task, it can be automatically transformed into a semantic segmentation (only one mask for every class) or object detection (bounding boxes for every object) tasks. All images are labeled (i.e. with annotations). There are 2 splits in the dataset: train (9000 images) and test (1000 images). Additionally, images have caption tags, while objects contain category tags with information about labels hierarchy. Explore them in supervisely. The dataset was released in 2017 by the University of Edinburgh, United Kingdom and Google AI Perception.

Here is a visualized example for randomly selected sample classes:

Explore #

COCO-Stuff 10k dataset has 10000 images. Click on one of the examples below or open "Explore" tool anytime you need to view dataset images with annotations. This tool has extended visualization capabilities like zoom, translation, objects table, custom filters and more. Hover the mouse over the images to hide or show annotations.

Sample annotation mask from COCO-Stuff 10k

👀

Have a look at 10000 images

View images along with annotations and tags, search and filter by various parameters

Class balance #

There are 183 annotation classes in the dataset. Find the general statistics and balances for every class in the table below. Click any row to preview images that have labels of the selected class. Sort by column to find the most rare or prevalent classes.

Rows 1-10 of 183

Class ㅤ	Images ㅤ	Objects ㅤ	Count on image average	Area on image average
unlabeled➔ mask	8759	8759	1	13.7%
person➔ mask	5492	15805	2.88	16.34%
tree➔ mask	3008	8101	2.69	17.43%
wall-other➔ mask	2450	7170	2.93	19.88%
sky-other➔ mask	2230	5770	2.59	22.36%
grass➔ mask	1761	5616	3.19	22.26%
building-other➔ mask	1610	4381	2.72	17.03%
clouds➔ mask	1524	4115	2.7	22.07%
road➔ mask	1313	3315	2.52	18.18%
pavement➔ mask	1054	2876	2.73	13.51%

Images #

Explore every single image in the dataset with respect to the number of annotations of each class it has. Click a row to preview selected image. Sort by any column to find anomalies and edge cases. Use horizontal scroll if the table has many columns for a large number of classes in the dataset.

Class sizes #

The table below gives various size properties of objects for every class. Click a row to see the image with annotations of the selected class. Sort columns to find classes with the smallest or largest objects or understand the size differences between classes.

Rows 1-10 of 172

Class	Object count	Avg area	Max area	Min area	Min height	Min height	Max height	Max height	Avg height	Avg height	Min width	Min width	Max width	Max width
unlabeled mask	76367	1.57%	100%	0.01%	1px	0.16%	640px	100%	65px	13.62%	1px	0.16%	640px	100%
person mask	15805	5.68%	94.64%	0.01%	3px	0.75%	640px	100%	139px	29.13%	3px	0.47%	640px	100%
tree mask	8101	6.47%	98.7%	0.02%	1px	0.16%	640px	100%	111px	23.43%	1px	0.16%	640px	100%
wall-other mask	7170	6.79%	100%	0.02%	1px	0.21%	640px	100%	144px	29.82%	1px	0.16%	640px	100%
sky-other mask	5770	8.64%	99.79%	0.01%	1px	0.21%	640px	100%	99px	20.88%	1px	0.16%	640px	100%
grass mask	5616	6.98%	98.64%	0.01%	2px	0.42%	640px	100%	91px	19.45%	1px	0.16%	640px	100%
building-other mask	4381	6.26%	99.39%	0.01%	3px	0.47%	640px	100%	115px	23.98%	1px	0.16%	640px	100%
clouds mask	4115	8.17%	99.81%	0.02%	2px	0.55%	640px	100%	96px	20.28%	1px	0.16%	640px	100%
road mask	3315	7.2%	78.55%	0.01%	1px	0.2%	612px	100%	98px	21.01%	1px	0.16%	640px	100%
chair mask	2912	2.2%	87.91%	0.02%	2px	0.53%	569px	100%	87px	18.65%	2px	0.31%	640px	100%

Spatial Heatmap #

The heatmaps below give the spatial distributions of all objects for every class. These visualizations provide insights into the most probable and rare object locations on the image. It helps analyze objects' placements in a dataset.

Objects #

Table contains all 98735 objects. Click a row to preview an image with annotations, and use search or pagination to navigate. Sort columns to find outliers in the dataset.

Rows 1-10 of 98735

Object ID ㅤ	Class ㅤ	Image name click row to open	Image size height x width	Height ㅤ	Height ㅤ	Width ㅤ	Width ㅤ	Area ㅤ
1➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	11px	2.29%	54px	8.44%	0.03%
2➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	11px	2.29%	28px	4.38%	0.03%
3➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	62px	12.92%	26px	4.06%	0.05%
4➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	46px	9.58%	14px	2.19%	0.08%
5➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	20px	4.17%	15px	2.34%	0.03%
6➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	56px	11.67%	41px	6.41%	0.38%
7➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	51px	10.62%	76px	11.88%	0.27%
8➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	24px	5%	26px	4.06%	0.13%
9➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	21px	4.38%	23px	3.59%	0.08%
10➔	unlabeled mask	COCO_train2014_000000412281.jpg	480 x 640	180px	37.5%	155px	24.22%	0.5%

License #

COCO-Stuff is a derivative work of the COCO dataset. The authors of COCO do not in any form endorse this work. Different licenses apply:

COCO images: Flickr Terms of use
COCO annotations: Creative Commons Attribution 4.0 License
COCO-Stuff annotations & code: Creative Commons Attribution 4.0 License

Source

Citation #

If you make use of the COCO-Stuff 10k data, please cite the following reference:

@misc{caesar2018cocostuff,
  title={COCO-Stuff: Thing and Stuff Classes in Context}, 
  author={Holger Caesar and Jasper Uijlings and Vittorio Ferrari},
  year={2018},
  eprint={1612.03716},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

Source

If you are happy with Dataset Ninja and use provided visualizations and tools in your work, please cite us:

@misc{ visualization-tools-for-cocostuff10k-dataset,
  title = { Visualization Tools for COCO-Stuff 10k Dataset },
  type = { Computer Vision Tools },
  author = { Dataset Ninja },
  howpublished = { \url{ https://datasetninja.com/cocostuff10k } },
  url = { https://datasetninja.com/cocostuff10k },
  journal = { Dataset Ninja },
  publisher = { Dataset Ninja },
  year = { 2024 },
  month = { nov },
  note = { visited on 2024-11-21 },
}

Download #

Dataset COCO-Stuff 10k can be downloaded in Supervisely format:

As an alternative, it can be downloaded with dataset-tools package:

pip install --upgrade dataset-tools

… using following python code:

import dataset_tools as dtools

dtools.download(dataset='COCO-Stuff 10k', dst_dir='~/dataset-ninja/')

Make sure not to overlook the python code example available on the Supervisely Developer Portal. It will give you a clear idea of how to effortlessly work with the downloaded dataset.

The data in original format can be downloaded here:

. . .

Disclaimer #

Our gal from the legal dep told us we need to post this:

Dataset Ninja provides visualizations and statistics for some datasets that can be found online and can be downloaded by general audience. Dataset Ninja is not a dataset hosting platform and can only be used for informational purposes. The platform does not claim any rights for the original content, including images, videos, annotations and descriptions. Joint publishing is prohibited.

You take full responsibility when you use datasets presented at Dataset Ninja, as well as other information, including visualizations and statistics we provide. You are in charge of compliance with any dataset license and all other permissions. You are required to navigate datasets homepage and make sure that you can use it. In case of any questions, get in touch with us at hello@datasetninja.com.