Skip to content

Commit fb02c51

Browse files
msluszniakmsluszniak
authored
[RNE Rewrite] feat: add voice activity detection pipeline (#1298)
## Description Adds a Voice Activity Detection (VAD) task pipeline and a corresponding `speech` example app. Chunked inference, segment postprocessing and streaming run in TypeScript on top of the core `model.execute` primitive. The per-frame feature extraction (framing, mean-removal, pre-emphasis, Hann window) is a native `speech.frameWaveform` C++ op: on device it dominated a `detect()` call (~86%, ~40&nbsp;ms of Hermes vs ~6&nbsp;ms for the model forward pass), so per the extension guidelines it lives in C++. It writes straight into the pre-allocated model-input tensor, fusing mean-removal + pre-emphasis + Hann into one dependency-free (vectorizable) pass; framing drops to ~3.4&nbsp;ms (~12×), below the model's own inference cost. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [ ] iOS - [x] Android ### Testing instructions - [ ] Build the `speech` app on iOS and Android - [ ] Test the Voice Activity Detection screen on a physical device (mic + xnnpack; simulator can't record) - [ ] Verify speech toggles SPEAKING/SILENT and logs begin/end events - [ ] Check the HF repo: https://huggingface.co/software-mansion/react-native-executorch-fsmn-vad ### Screenshots <!-- Add screenshots here, if applicable --> ### Related issues Closes #1249 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes - Depends on the per-method `get_dynamic_dims_forward` input validation from the embeddings PR (#1292): VAD feeds a variable-length `[frames, 512]` input tensor per chunk. Outputs are still validated exactly, so the output tensor is pre-allocated at the model-declared shape. Requires #1292 to land. The `fsmn-vad` model is re-exported (tag `v0.10.0`) with a `get_dynamic_dims_forward` method returning int32 `[rank, 3]` bounds per input. - Segments are returned in seconds (the old native path returned raw sample indices). - The FSMN output contract is assumed to be `[1, frames, classes]` with class 0 = non-speech (`speech = 1 - p0`), matching the current native implementation.
1 parent e48ac5f commit fb02c51

34 files changed

Lines changed: 1361 additions & 1 deletion

‎.agents/skills/add-native-extension/SKILL.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,12 @@ namespace rnexecutorch::extensions::<domain>
151151

152152
### Step 3: TypeScript Bridge & Wrappers
153153

154-
Under `src/extensions/<domain>.ts` or `src/extensions/<domain>/index.ts`:
154+
**Where the wrapper file goes — general vs. model-specific ops:**
155+
156+
- **Domain-general ops** (reusable across every task in the domain, e.g. `image_ops`, `box_ops`) go in the domain's shared op files (`src/extensions/<domain>/ops.ts`), re-exported from `src/extensions/<domain>/index.ts`.
157+
- **Model- or task-specific ops** (only meaningful to one model/pipeline, e.g. FSMN-VAD framing) **must** go under `src/extensions/<domain>/utils/<name>.ts` — one file per model/task, named for it (e.g. `vadUtils.ts`, `supertonicUtils.ts`). Keep them out of the shared `ops.ts` so it stays a home for genuinely reusable ops, and group each model's helpers together under `utils/`. Re-export them from the domain `index.ts` too so power users can reach them.
158+
159+
Then, in that wrapper file:
155160

156161
- **Use the `rnexecutorchJsi` Symbol**: You must import and interact with native bindings using the `rnexecutorchJsi` symbol exported from [src/native/bridge.ts](../../../packages/react-native-executorch/src/native/bridge.ts). **Do not** reference the global `__rnexecutorch_jsi__` directly throughout your wrapper files.
157162
- Expose the TypeScript wrapper.
@@ -188,5 +193,6 @@ When adding a native extension, verify that:
188193
- [ ] Input and output tensors are locked using `tensor::tryLockShared` and `tensor::tryLockUnique` respectively.
189194
- [ ] No default parameter values are defined in the C++ header/source files.
190195
- [ ] The custom operation install function is registered in both the domain `install` function and core [cpp/RnExecutorch.cpp](../../../packages/react-native-executorch/cpp/RnExecutorch.cpp).
196+
- [ ] The TypeScript wrapper lives in the right place: domain-general ops in the shared `ops.ts`, model-/task-specific ops under `src/extensions/<domain>/utils/<name>.ts`.
191197
- [ ] The TypeScript wrapper imports and uses `rnexecutorchJsi` instead of the global `__rnexecutorch_jsi__`.
192198
- [ ] The TypeScript wrapper is marked with the `"worklet";` directive and defines all default parameter values.

‎.cspell-wordlist.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,3 +278,8 @@ binarization
278278
bugprone
279279
NOLINTNEXTLINE
280280
nullopt
281+
282+
hann
283+
preemphasis
284+
coeff
285+
Silero

‎apps/speech/app.json‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
{
2+
"expo": {
3+
"name": "speech",
4+
"slug": "speech",
5+
"version": "1.0.0",
6+
"orientation": "portrait",
7+
"icon": "./assets/icons/icon.png",
8+
"userInterfaceStyle": "light",
9+
"newArchEnabled": true,
10+
"scheme": "rne-speech",
11+
"splash": {
12+
"image": "./assets/icons/splash.png",
13+
"resizeMode": "contain",
14+
"backgroundColor": "#ffffff"
15+
},
16+
"ios": {
17+
"supportsTablet": true,
18+
"bundleIdentifier": "com.anonymous.speech",
19+
"infoPlist": {
20+
"NSMicrophoneUsageDescription": "This app uses the microphone to detect voice activity."
21+
}
22+
},
23+
"android": {
24+
"adaptiveIcon": {
25+
"foregroundImage": "./assets/icons/adaptive-icon.png",
26+
"backgroundColor": "#ffffff"
27+
},
28+
"package": "com.anonymous.speech",
29+
"permissions": [
30+
"android.permission.RECORD_AUDIO"
31+
]
32+
},
33+
"web": {
34+
"favicon": "./assets/icons/favicon.png"
35+
},
36+
"plugins": [
37+
"expo-router",
38+
[
39+
"expo-build-properties",
40+
{
41+
"android": {
42+
"minSdkVersion": 26
43+
},
44+
"ios": {
45+
"deploymentTarget": "17.0"
46+
}
47+
}
48+
]
49+
]
50+
}
51+
}

‎apps/speech/app/_layout.tsx‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
import { Drawer } from 'expo-router/drawer';
2+
import { ColorPalette } from '../theme';
3+
import React from 'react';
4+
5+
export default function Layout() {
6+
return (
7+
<Drawer
8+
screenOptions={{
9+
drawerActiveTintColor: ColorPalette.primary,
10+
drawerInactiveTintColor: '#888',
11+
headerTintColor: ColorPalette.primary,
12+
headerTitleStyle: { color: ColorPalette.primary },
13+
}}
14+
>
15+
<Drawer.Screen
16+
name="index"
17+
options={{
18+
drawerLabel: () => null,
19+
title: 'Main Menu',
20+
drawerItemStyle: { display: 'none' },
21+
}}
22+
/>
23+
<Drawer.Screen
24+
name="vad/index"
25+
options={{
26+
drawerLabel: 'Voice Activity Detection',
27+
title: 'Voice Activity Detection',
28+
}}
29+
/>
30+
</Drawer>
31+
);
32+
}

‎apps/speech/app/index.tsx‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
import { useRouter } from 'expo-router';
2+
import { View, Text, StyleSheet, TouchableOpacity } from 'react-native';
3+
import { ColorPalette } from '../theme';
4+
import ExecutorchLogo from '../assets/icons/executorch.svg';
5+
6+
export default function Home() {
7+
const router = useRouter();
8+
9+
return (
10+
<View style={styles.container}>
11+
<ExecutorchLogo width={64} height={64} />
12+
<Text style={styles.headerText}>Select a demo</Text>
13+
<View style={styles.buttonContainer}>
14+
<TouchableOpacity style={styles.button} onPress={() => router.navigate('vad/')}>
15+
<Text style={styles.buttonText}>Voice Activity Detection</Text>
16+
</TouchableOpacity>
17+
</View>
18+
</View>
19+
);
20+
}
21+
22+
const styles = StyleSheet.create({
23+
container: {
24+
flex: 1,
25+
justifyContent: 'center',
26+
alignItems: 'center',
27+
backgroundColor: '#fff',
28+
},
29+
headerText: {
30+
fontSize: 18,
31+
color: ColorPalette.strongPrimary,
32+
margin: 20,
33+
},
34+
buttonContainer: {
35+
width: '80%',
36+
justifyContent: 'space-evenly',
37+
marginBottom: 20,
38+
},
39+
button: {
40+
backgroundColor: ColorPalette.strongPrimary,
41+
borderRadius: 8,
42+
padding: 14,
43+
alignItems: 'center',
44+
marginBottom: 12,
45+
},
46+
buttonText: {
47+
color: 'white',
48+
fontSize: 16,
49+
fontWeight: '600',
50+
},
51+
});

‎apps/speech/app/vad/index.tsx‎

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
import React, { useEffect, useRef, useState } from 'react';
2+
import { View, Text, StyleSheet, ScrollView, Platform } from 'react-native';
3+
import { useVoiceActivityDetector, models, FSMN_VAD_SAMPLE_RATE_HZ } from 'react-native-executorch';
4+
import { AudioManager, AudioRecorder } from 'react-native-audio-api';
5+
import DeviceInfo from 'react-native-device-info';
6+
7+
import ScreenWrapper from '../../components/ScreenWrapper';
8+
import { ModelStatus } from '../../components/ModelStatus';
9+
import { Button } from '../../components/Button';
10+
import { theme } from '../../theme';
11+
12+
// Record at the model's expected sample rate rather than hardcoding it.
13+
const SAMPLE_RATE = FSMN_VAD_SAMPLE_RATE_HZ;
14+
const isSimulator = DeviceInfo.isEmulatorSync();
15+
16+
function VADContent() {
17+
const { isReady, downloadProgress, error, detectVoiceOnStream, resetStream } =
18+
useVoiceActivityDetector(models.voiceActivityDetection.FSMN_VAD);
19+
20+
const [isStreaming, setIsStreaming] = useState(false);
21+
const [isSpeaking, setIsSpeaking] = useState(false);
22+
const [hasMicPermission, setHasMicPermission] = useState(false);
23+
const [runError, setRunError] = useState<string | null>(null);
24+
const [logs, setLogs] = useState<string[]>([]);
25+
26+
const recorder = useRef(new AudioRecorder());
27+
const logScrollRef = useRef<ScrollView>(null);
28+
29+
const addLog = (message: string) => {
30+
setLogs((prev) => [...prev, `${new Date().toLocaleTimeString()}: ${message}`]);
31+
};
32+
33+
useEffect(() => {
34+
AudioManager.setAudioSessionOptions({
35+
iosCategory: 'playAndRecord',
36+
iosMode: 'spokenAudio',
37+
iosOptions: ['allowBluetoothHFP', 'defaultToSpeaker'],
38+
});
39+
AudioManager.requestRecordingPermissions().then((status) =>
40+
setHasMicPermission(status === 'Granted')
41+
);
42+
}, []);
43+
44+
const handleStart = async () => {
45+
if (isStreaming || !isReady || !detectVoiceOnStream || !resetStream) return;
46+
47+
if (!hasMicPermission) {
48+
setRunError('Microphone permission denied. Please enable it in Settings.');
49+
return;
50+
}
51+
52+
setRunError(null);
53+
setLogs([]);
54+
setIsStreaming(true);
55+
addLog('Starting VAD stream…');
56+
57+
resetStream();
58+
recorder.current.onAudioReady(
59+
{ sampleRate: SAMPLE_RATE, bufferLength: 1600, channelCount: 1 },
60+
({ buffer }) => {
61+
const event = detectVoiceOnStream(buffer.getChannelData(0), { detectionMargin: 300 });
62+
if (event === 'speechStart') {
63+
setIsSpeaking(true);
64+
addLog('Speech detected (begin)');
65+
} else if (event === 'speechEnd') {
66+
setIsSpeaking(false);
67+
addLog('Silence detected (end)');
68+
}
69+
}
70+
);
71+
72+
try {
73+
await AudioManager.setAudioSessionActivity(true);
74+
const started = await recorder.current.start();
75+
if (started.status === 'error') {
76+
throw new Error(started.message);
77+
}
78+
} catch (e) {
79+
setRunError(e instanceof Error ? e.message : String(e));
80+
setIsStreaming(false);
81+
}
82+
};
83+
84+
const handleStop = async () => {
85+
await recorder.current.stop();
86+
resetStream?.();
87+
setIsStreaming(false);
88+
setIsSpeaking(false);
89+
addLog('VAD stream stopped');
90+
};
91+
92+
const streamDisabled = isSimulator || !isReady;
93+
94+
return (
95+
<ScrollView style={styles.container} contentContainerStyle={styles.content}>
96+
<View style={styles.card}>
97+
<Text style={styles.cardTitle}>Voice Activity Detection</Text>
98+
<Text style={styles.cardDescription}>
99+
Streams microphone audio through the FSMN-VAD model and reports when speech begins and
100+
ends in real time.
101+
</Text>
102+
<ModelStatus
103+
isReady={isReady}
104+
downloadProgress={downloadProgress}
105+
error={error ? error.message : null}
106+
modelTypeLabel="VAD model"
107+
/>
108+
</View>
109+
110+
{runError && (
111+
<View style={styles.errorContainer}>
112+
<Text style={styles.errorText}>{runError}</Text>
113+
</View>
114+
)}
115+
116+
<View style={styles.card}>
117+
<View style={styles.visualizer}>
118+
<View style={[styles.indicator, isSpeaking ? styles.speaking : styles.silent]} />
119+
<Text
120+
style={[styles.visualizerText, isSpeaking ? styles.speakingText : styles.silentText]}
121+
>
122+
{isSpeaking ? 'SPEAKING' : 'SILENT'}
123+
</Text>
124+
</View>
125+
126+
{isStreaming ? (
127+
<Button title="Stop VAD stream" variant="accent" onPress={handleStop} />
128+
) : (
129+
<Button
130+
title={isSimulator ? 'Recording not available on simulator' : 'Start VAD stream'}
131+
onPress={handleStart}
132+
disabled={streamDisabled}
133+
/>
134+
)}
135+
</View>
136+
137+
<View style={styles.card}>
138+
<Text style={styles.sectionTitle}>VAD events</Text>
139+
<ScrollView
140+
ref={logScrollRef}
141+
style={styles.logScroll}
142+
onContentSizeChange={() => logScrollRef.current?.scrollToEnd({ animated: true })}
143+
>
144+
{logs.length > 0 ? (
145+
logs.map((log, i) => (
146+
<Text key={i} style={styles.logText}>
147+
{log}
148+
</Text>
149+
))
150+
) : (
151+
<Text style={styles.emptyText}>No events logged yet…</Text>
152+
)}
153+
</ScrollView>
154+
</View>
155+
</ScrollView>
156+
);
157+
}
158+
159+
export default function VADScreen() {
160+
return (
161+
<ScreenWrapper>
162+
<VADContent />
163+
</ScreenWrapper>
164+
);
165+
}
166+
167+
const styles = StyleSheet.create({
168+
container: { flex: 1, backgroundColor: theme.colors.background },
169+
content: { padding: theme.spacing.large, paddingBottom: 40 },
170+
card: {
171+
backgroundColor: theme.colors.cardBackground,
172+
borderRadius: theme.radius.large,
173+
padding: 20,
174+
marginBottom: 20,
175+
borderWidth: 1,
176+
borderColor: theme.colors.lightBorder,
177+
},
178+
cardTitle: {
179+
fontSize: theme.typography.title.fontSize,
180+
fontWeight: theme.typography.title.fontWeight,
181+
color: theme.colors.strongPrimary,
182+
marginBottom: 8,
183+
},
184+
cardDescription: {
185+
fontSize: 14,
186+
color: theme.colors.textMuted,
187+
lineHeight: 20,
188+
marginBottom: 16,
189+
},
190+
visualizer: { alignItems: 'center', marginBottom: 20 },
191+
indicator: {
192+
width: 96,
193+
height: 96,
194+
borderRadius: 48,
195+
marginBottom: 16,
196+
},
197+
speaking: { backgroundColor: '#22c55e' },
198+
silent: { backgroundColor: '#e9ecef' },
199+
visualizerText: { fontSize: 22, fontWeight: '800', letterSpacing: 2 },
200+
speakingText: { color: '#22c55e' },
201+
silentText: { color: theme.colors.textPlaceholder },
202+
sectionTitle: { fontSize: 16, fontWeight: '700', color: '#212529', marginBottom: 10 },
203+
logScroll: {
204+
maxHeight: 180,
205+
backgroundColor: '#f8fafc',
206+
borderRadius: theme.radius.small,
207+
borderWidth: 1,
208+
borderColor: theme.colors.lightBorder,
209+
padding: 12,
210+
},
211+
logText: {
212+
fontSize: 12,
213+
fontFamily: Platform.OS === 'ios' ? 'Menlo' : 'monospace',
214+
color: '#334155',
215+
marginBottom: 2,
216+
},
217+
emptyText: { color: theme.colors.textPlaceholder, fontStyle: 'italic' },
218+
errorContainer: {
219+
backgroundColor: theme.colors.errorBackground,
220+
padding: 12,
221+
borderRadius: theme.radius.small,
222+
marginBottom: 20,
223+
},
224+
errorText: { color: theme.colors.errorText, fontSize: 14, textAlign: 'center' },
225+
});

‎apps/speech/assets/icons/adaptive-icon.png‎

17.1 KB
Loading

‎apps/speech/assets/icons/executorch.svg‎

Lines changed: 9 additions & 0 deletions
Loading

‎apps/speech/assets/icons/favicon.png‎

1.43 KB
Loading

‎apps/speech/assets/icons/icon.png‎

21.9 KB
Loading

0 commit comments

Comments
 (0)