Skip to content

Commit 3f4fdb5

Browse files
benITo47msluszniak
andauthored
chore!: Rename image segmentation to semanatic segmentation (#837)
## Description This PR renames image segmentation, to more relevant and precise - semantic segmentation ### Introduces a breaking change? - [x] Yes - [ ] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [ ] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [x] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions Run tests, check Semantic Segmentation app in computer-vision/ Co-authored-by: Mateusz Słuszniak <mateusz.sluszniak@swmansion.com>
1 parent d12ebc3 commit 3f4fdb5

46 files changed

Lines changed: 441 additions & 431 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

apps/computer-vision/app/_layout.tsx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,10 +68,10 @@ export default function _layout() {
6868
}}
6969
/>
7070
<Drawer.Screen
71-
name="image_segmentation/index"
71+
name="semantic_segmentation/index"
7272
options={{
73-
drawerLabel: 'Image Segmentation',
74-
title: 'Image Segmentation',
73+
drawerLabel: 'Semantic Segmentation',
74+
title: 'Semantic Segmentation',
7575
headerTitleStyle: { color: ColorPalette.primary },
7676
}}
7777
/>

apps/computer-vision/app/index.tsx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,9 @@ export default function Home() {
1919
</TouchableOpacity>
2020
<TouchableOpacity
2121
style={styles.button}
22-
onPress={() => router.navigate('image_segmentation/')}
22+
onPress={() => router.navigate('semantic_segmentation/')}
2323
>
24-
<Text style={styles.buttonText}>Image Segmentation</Text>
24+
<Text style={styles.buttonText}>Semantic Segmentation</Text>
2525
</TouchableOpacity>
2626
<TouchableOpacity
2727
style={styles.button}

apps/computer-vision/app/image_segmentation/index.tsx renamed to apps/computer-vision/app/semantic_segmentation/index.tsx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ import { BottomBar } from '../../components/BottomBar';
33
import { getImage } from '../../utils';
44
import {
55
DEEPLAB_V3_RESNET50,
6-
useImageSegmentation,
6+
useSemanticSegmentation,
77
} from 'react-native-executorch';
88
import {
99
Canvas,
@@ -42,10 +42,10 @@ const numberToColor: number[][] = [
4242
[162, 51, 255], // 20 Amethyst
4343
];
4444

45-
export default function ImageSegmentationScreen() {
45+
export default function SemanticSegmentationScreen() {
4646
const { setGlobalGenerating } = useContext(GeneratingContext);
4747
const { isReady, isGenerating, downloadProgress, forward } =
48-
useImageSegmentation({
48+
useSemanticSegmentation({
4949
model: DEEPLAB_V3_RESNET50,
5050
});
5151
const [imageUri, setImageUri] = useState('');

docs/docs/02-benchmarks/inference-time.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ Benchmark times for text embeddings are highly dependent on the sentence length.
117117
Image embedding benchmark times are measured using 224×224 pixel images, as required by the model. All input images, whether larger or smaller, are resized to 224×224 before processing. Resizing is typically fast for small images but may be noticeably slower for very large images, which can increase total inference time.
118118
:::
119119

120-
## Image Segmentation
120+
## Semantic Segmentation
121121

122122
:::warning
123123
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.

docs/docs/02-benchmarks/memory-usage.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ The reported memory usage values include the memory footprint of the Phonemis pa
8383
| --------------------------- | :--------------------: | :----------------: |
8484
| CLIP_VIT_BASE_PATCH32_IMAGE | 345 | 340 |
8585

86-
## Image Segmentation
86+
## Semantic Segmentation
8787

8888
:::warning
8989
Data presented in the following sections is based on inference with non-resized output. When resize is enabled, expect higher memory usage and inference time with higher resolutions.

docs/docs/02-benchmarks/model-size.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ title: Model Size
8686
| --------------------------- | :----------: |
8787
| CLIP_VIT_BASE_PATCH32_IMAGE | 352 |
8888

89-
## Image Segmentation
89+
## Semantic Segmentation
9090

9191
| Model | XNNPACK [MB] |
9292
| ----------------- | ------------ |

docs/docs/03-hooks/02-computer-vision/useImageSegmentation.md renamed to docs/docs/03-hooks/02-computer-vision/useSemanticSegmentation.md

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,27 @@
11
---
2-
title: useImageSegmentation
2+
title: useSemanticSegmentation
33
---
44

5-
Semantic image segmentation, akin to image classification, tries to assign the content of the image to one of the predefined classes. However, in case of segmentation this classification is done on a per-pixel basis, so as the result the model provides an image-sized array of scores for each of the classes. You can then use this information to detect objects on a per-pixel basis. React Native ExecuTorch offers a dedicated hook `useImageSegmentation` for this task.
5+
Semantic semantic segmentation, akin to image classification, tries to assign the content of the image to one of the predefined classes. However, in case of segmentation this classification is done on a per-pixel basis, so as the result the model provides an image-sized array of scores for each of the classes. You can then use this information to detect objects on a per-pixel basis. React Native ExecuTorch offers a dedicated hook `useSemanticSegmentation` for this task.
66

77
:::warning
8-
It is recommended to use models provided by us which are available at our [Hugging Face repository](https://huggingface.co/collections/software-mansion/image-segmentation-68d5291bdf4a30bee0220f4f), you can also use [constants](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts) shipped with our library.
8+
It is recommended to use models provided by us which are available at our [Hugging Face repository](https://huggingface.co/collections/software-mansion/semantic-segmentation-68d5291bdf4a30bee0220f4f), you can also use [constants](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/constants/modelUrls.ts) shipped with our library.
99
:::
1010

1111
## API Reference
1212

13-
- For detailed API Reference for `useImageSegmentation` see: [`useImageSegmentation` API Reference](../../06-api-reference/functions/useImageSegmentation.md).
14-
- For all image segmentation models available out-of-the-box in React Native ExecuTorch see: [Image Segmentation Models](../../06-api-reference/index.md#models---image-segmentation).
13+
- For detailed API Reference for `useSemanticSegmentation` see: [`useSemanticSegmentation` API Reference](../../06-api-reference/functions/useSemanticSegmentation.md).
14+
- For all semantic segmentation models available out-of-the-box in React Native ExecuTorch see: [Semantic Segmentation Models](../../06-api-reference/index.md#models---semantic-segmentation).
1515

1616
## High Level Overview
1717

1818
```typescript
1919
import {
20-
useImageSegmentation,
20+
useSemanticSegmentation,
2121
DEEPLAB_V3_RESNET50,
2222
} from 'react-native-executorch';
2323

24-
const model = useImageSegmentation({
24+
const model = useSemanticSegmentation({
2525
model: DEEPLAB_V3_RESNET50,
2626
});
2727

@@ -37,24 +37,24 @@ try {
3737

3838
### Arguments
3939

40-
`useImageSegmentation` takes [`ImageSegmentationProps`](../../06-api-reference/interfaces/ImageSegmentationProps.md) that consists of:
40+
`useSemanticSegmentation` takes [`SemanticSegmentationProps`](../../06-api-reference/interfaces/SemanticSegmentationProps.md) that consists of:
4141

4242
- `model` - An object containing:
43-
- `modelName` - The name of a built-in model. See [`ModelSources`](../../06-api-reference/type-aliases/ModelSources.md) for the list of supported models.
43+
- `modelName` - The name of a built-in model. See [`SemanticSegmentationModelSources`](../../06-api-reference/type-aliases/SemanticSegmentationModelSources.md) for the list of supported models.
4444
- `modelSource` - The location of the model binary (a URL or a bundled resource).
45-
- An optional flag [`preventLoad`](../../06-api-reference/interfaces/ImageSegmentationProps.md#preventload) which prevents auto-loading of the model.
45+
- An optional flag [`preventLoad`](../../06-api-reference/interfaces/SemanticSegmentationProps.md#preventload) which prevents auto-loading of the model.
4646

4747
The hook is generic over the model config — TypeScript automatically infers the correct label type based on the `modelName` you provide. No explicit generic parameter is needed.
4848

4949
You need more details? Check the following resources:
5050

51-
- For detailed information about `useImageSegmentation` arguments check this section: [`useImageSegmentation` arguments](../../06-api-reference/functions/useImageSegmentation.md#parameters).
52-
- For all image segmentation models available out-of-the-box in React Native ExecuTorch see: [Image Segmentation Models](../../06-api-reference/index.md#models---image-segmentation).
51+
- For detailed information about `useSemanticSegmentation` arguments check this section: [`useSemanticSegmentation` arguments](../../06-api-reference/functions/useSemanticSegmentation.md#parameters).
52+
- For all semantic segmentation models available out-of-the-box in React Native ExecuTorch see: [Semantic Segmentation Models](../../06-api-reference/index.md#models---semantic-segmentation).
5353
- For more information on loading resources, take a look at [loading models](../../01-fundamentals/02-loading-models.md) page.
5454

5555
### Returns
5656

57-
`useImageSegmentation` returns an [`ImageSegmentationType`](../../06-api-reference/interfaces/ImageSegmentationType.md) object containing:
57+
`useSemanticSegmentation` returns an [`SemanticSegmentationType`](../../06-api-reference/interfaces/SemanticSegmentationType.md) object containing:
5858

5959
- `isReady` - Whether the model is loaded and ready to process images.
6060
- `isGenerating` - Whether the model is currently processing an image.
@@ -64,11 +64,11 @@ You need more details? Check the following resources:
6464

6565
## Running the model
6666

67-
To run the model, use the [`forward`](../../06-api-reference/interfaces/ImageSegmentationType.md#forward) method. It accepts three arguments:
67+
To run the model, use the [`forward`](../../06-api-reference/interfaces/SemanticSegmentationType.md#forward) method. It accepts three arguments:
6868

69-
- [`imageSource`](../../06-api-reference/interfaces/ImageSegmentationType.md#forward) (required) - The image to segment. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
70-
- [`classesOfInterest`](../../06-api-reference/interfaces/ImageSegmentationType.md#forward) (optional) - An array of label keys indicating which per-class probability masks to include in the output. Defaults to `[]` (no class masks). The `ARGMAX` map is always returned regardless of this parameter.
71-
- [`resizeToInput`](../../06-api-reference/interfaces/ImageSegmentationType.md#forward) (optional) - Whether to resize the output masks to the original input image dimensions. Defaults to `true`. If `false`, returns the raw model output dimensions (e.g. 224x224 for `DEEPLAB_V3_RESNET50`).
69+
- [`imageSource`](../../06-api-reference/interfaces/SemanticSegmentationType.md#forward) (required) - The image to segment. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
70+
- [`classesOfInterest`](../../06-api-reference/interfaces/SemanticSegmentationType.md#forward) (optional) - An array of label keys indicating which per-class probability masks to include in the output. Defaults to `[]` (no class masks). The `ARGMAX` map is always returned regardless of this parameter.
71+
- [`resizeToInput`](../../06-api-reference/interfaces/SemanticSegmentationType.md#forward) (optional) - Whether to resize the output masks to the original input image dimensions. Defaults to `true`. If `false`, returns the raw model output dimensions (e.g. 224x224 for `DEEPLAB_V3_RESNET50`).
7272

7373
:::warning
7474
Setting `resizeToInput` to `false` will make `forward` faster.
@@ -85,13 +85,13 @@ The return type is fully typed — TypeScript narrows it based on the labels you
8585

8686
```typescript
8787
import {
88-
useImageSegmentation,
88+
useSemanticSegmentation,
8989
DEEPLAB_V3_RESNET50,
9090
DeeplabLabel,
9191
} from 'react-native-executorch';
9292

9393
function App() {
94-
const model = useImageSegmentation({
94+
const model = useSemanticSegmentation({
9595
model: DEEPLAB_V3_RESNET50,
9696
});
9797

docs/docs/04-typescript-api/02-computer-vision/ImageSegmentationModule.md renamed to docs/docs/04-typescript-api/02-computer-vision/SemanticSegmentationModule.md

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,26 @@
11
---
2-
title: ImageSegmentationModule
2+
title: SemanticSegmentationModule
33
---
44

5-
TypeScript API implementation of the [useImageSegmentation](../../03-hooks/02-computer-vision/useImageSegmentation.md) hook.
5+
TypeScript API implementation of the [useSemanticSegmentation](../../03-hooks/02-computer-vision/useSemanticSegmentation.md) hook.
66

77
## API Reference
88

9-
- For detailed API Reference for `ImageSegmentationModule` see: [`ImageSegmentationModule` API Reference](../../06-api-reference/classes/ImageSegmentationModule.md).
10-
- For all image segmentation models available out-of-the-box in React Native ExecuTorch see: [Image Segmentation Models](../../06-api-reference/index.md#models---image-segmentation).
9+
- For detailed API Reference for `SemanticSegmentationModule` see: [`SemanticSegmentationModule` API Reference](../../06-api-reference/classes/SemanticSegmentationModule.md).
10+
- For all semantic segmentation models available out-of-the-box in React Native ExecuTorch see: [Semantic Segmentation Models](../../06-api-reference/index.md#models---semantic-segmentation).
1111

1212
## High Level Overview
1313

1414
```typescript
1515
import {
16-
ImageSegmentationModule,
16+
SemanticSegmentationModule,
1717
DEEPLAB_V3_RESNET50,
1818
} from 'react-native-executorch';
1919

2020
const imageUri = 'path/to/image.png';
2121

2222
// Creating an instance from a built-in model
23-
const segmentation = await ImageSegmentationModule.fromModelName({
23+
const segmentation = await SemanticSegmentationModule.fromModelName({
2424
modelName: 'deeplab-v3',
2525
modelSource: DEEPLAB_V3_RESNET50,
2626
});
@@ -32,18 +32,18 @@ const result = await segmentation.forward(imageUri);
3232

3333
### Methods
3434

35-
All methods of `ImageSegmentationModule` are explained in details here: [`ImageSegmentationModule` API Reference](../../06-api-reference/classes/ImageSegmentationModule.md)
35+
All methods of `SemanticSegmentationModule` are explained in details here: [`SemanticSegmentationModule` API Reference](../../06-api-reference/classes/SemanticSegmentationModule.md)
3636

3737
## Loading the model
3838

39-
`ImageSegmentationModule` uses static factory methods instead of `new()` + `load()`. There are two ways to create an instance:
39+
`SemanticSegmentationModule` uses static factory methods instead of `new()` + `load()`. There are two ways to create an instance:
4040

4141
### Built-in models — `fromModelName`
4242

43-
Use [`fromModelName`](../../06-api-reference/classes/ImageSegmentationModule.md#frommodelname) for models that ship with built-in label maps and preprocessing configs:
43+
Use [`fromModelName`](../../06-api-reference/classes/SemanticSegmentationModule.md#frommodelname) for models that ship with built-in label maps and preprocessing configs:
4444

4545
```typescript
46-
const segmentation = await ImageSegmentationModule.fromModelName(
46+
const segmentation = await SemanticSegmentationModule.fromModelName(
4747
DEEPLAB_V3_RESNET50,
4848
(progress) => console.log(`Download: ${Math.round(progress * 100)}%`)
4949
);
@@ -53,12 +53,12 @@ The `config` parameter is a discriminated union — TypeScript ensures you provi
5353

5454
### Custom models — `fromCustomConfig`
5555

56-
Use [`fromCustomConfig`](../../06-api-reference/classes/ImageSegmentationModule.md#fromcustomconfig) for custom-exported segmentation models with your own label map:
56+
Use [`fromCustomConfig`](../../06-api-reference/classes/SemanticSegmentationModule.md#fromcustomconfig) for custom-exported segmentation models with your own label map:
5757

5858
```typescript
5959
const MyLabels = { BACKGROUND: 0, FOREGROUND: 1 } as const;
6060

61-
const segmentation = await ImageSegmentationModule.fromCustomConfig(
61+
const segmentation = await SemanticSegmentationModule.fromCustomConfig(
6262
'https://example.com/custom_model.pte',
6363
{
6464
labelMap: MyLabels,
@@ -76,11 +76,11 @@ For more information on loading resources, take a look at [loading models](../..
7676

7777
## Running the model
7878

79-
To run the model, use the [`forward`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) method. It accepts three arguments:
79+
To run the model, use the [`forward`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) method. It accepts three arguments:
8080

81-
- [`imageSource`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) (required) - The image to segment. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
82-
- [`classesOfInterest`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) (optional) - An array of label keys indicating which per-class probability masks to include in the output. Defaults to `[]`. The `ARGMAX` map is always returned regardless.
83-
- [`resizeToInput`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) (optional) - Whether to resize the output masks to the original input image dimensions. Defaults to `true`. If `false`, returns the raw model output dimensions.
81+
- [`imageSource`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) (required) - The image to segment. Can be a remote URL, a local file URI, or a base64-encoded image (whole URI or only raw base64).
82+
- [`classesOfInterest`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) (optional) - An array of label keys indicating which per-class probability masks to include in the output. Defaults to `[]`. The `ARGMAX` map is always returned regardless.
83+
- [`resizeToInput`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) (optional) - Whether to resize the output masks to the original input image dimensions. Defaults to `true`. If `false`, returns the raw model output dimensions.
8484

8585
:::warning
8686
Setting `resizeToInput` to `false` will make `forward` faster.
@@ -107,4 +107,4 @@ result.DOG; // Float32Array
107107

108108
## Managing memory
109109

110-
The module is a regular JavaScript object, and as such its lifespan will be managed by the garbage collector. In most cases this should be enough, and you should not worry about freeing the memory of the module yourself, but in some cases you may want to release the memory occupied by the module before the garbage collector steps in. In this case use the method [`delete`](../../06-api-reference/classes/ImageSegmentationModule.md#delete) on the module object you will no longer use, and want to remove from the memory. Note that you cannot use [`forward`](../../06-api-reference/classes/ImageSegmentationModule.md#forward) after [`delete`](../../06-api-reference/classes/ImageSegmentationModule.md#delete) unless you create a new instance.
110+
The module is a regular JavaScript object, and as such its lifespan will be managed by the garbage collector. In most cases this should be enough, and you should not worry about freeing the memory of the module yourself, but in some cases you may want to release the memory occupied by the module before the garbage collector steps in. In this case use the method [`delete`](../../06-api-reference/classes/SemanticSegmentationModule.md#delete) on the module object you will no longer use, and want to remove from the memory. Note that you cannot use [`forward`](../../06-api-reference/classes/SemanticSegmentationModule.md#forward) after [`delete`](../../06-api-reference/classes/SemanticSegmentationModule.md#delete) unless you create a new instance.

0 commit comments

Comments
 (0)