Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

classifications are not working properly in Audio stream #5786

Open
VijayalakshmiNagaraj97 opened this issue Dec 18, 2024 · 0 comments
Open
Assignees
Labels
os:windows MediaPipe issues on Windows platform:javascript MediaPipe Javascript issues task:audio classification Issues related to Audio Classification: Classify sounds into relevant tags type:support General questions

Comments

@VijayalakshmiNagaraj97
Copy link

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

windows

MediaPipe Tasks SDK version

No response

Task name (e.g. Image classification, Gesture recognition etc.)

audio classification

Programming Language and version (e.g. C++, Python, Java)

javascript

Describe the actual behavior

i want to do audioclassification in audio stream ,when i use the option scoreThersold ,I am not getting the proper result but in tryit demo i am getting proper result even i increased the score thersold with the same background setup

Describe the expected behaviour

i am expecting audioclassification like tryit with using real time audiostream

Standalone code/steps you may have used to try to get what you need

when i give scoreThersold ,classification results not working properly

Other info / Complete Logs

import { AudioClassifier, FilesetResolver } from "@mediapipe/tasks-audio";

let audioClassifier = null;

// Load the Audio Classifier once the worker starts
onmessage = async (event) => {
  const { type, data } = event.data;

  if (type === "initialize") {
    const audioFileset = await FilesetResolver.forAudioTasks(
      "https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/wasm"
    );

    audioClassifier = await AudioClassifier.createFromOptions(audioFileset, {
      baseOptions: {
        modelAssetPath:
          "https://storage.googleapis.com/mediapipe-models/audio_classifier/yamnet/float32/1/yamnet.tflite",
      },
     
      running_mode:"AUDIO_STREAM",
    
    });

    postMessage({ type: "initialized" });
  } else if (type === "classify" && audioClassifier) {
    
    const result =await audioClassifier.classify(data);
    console.log("Raw classification result:", result);
    postMessage({ type: "classificationResult", data: result });
  }
};
 useEffect(() => {
    workerRef.current = new Worker(new URL("../AudioSample/audioClassifierWorker.js", import.meta.url));

    workerRef.current.onmessage = (event) => {
      const { type, data } = event.data;
      if (type === "initialized") {
        console.log("Audio Classifier Initialized");
      } else if (type === "classificationResult") {
        handleClassificationResult(data);
      }
    };

    workerRef.current.postMessage({ type: "initialize" });

    return () => {
      if (workerRef.current) {
        workerRef.current.terminate();
        workerRef.current = null;
      }
    };
  }, []);
const handleClassificationResult = (result) => {
    if (result.length > 0 && result[0].classifications.length > 0) {
      const categories = result[0].classifications[0].categories;
      setClassificationResult(
        categories
          .slice(0, 3)
          .map(
            (category) =>
              `${category.categoryName} (${category.score.toFixed(3)})`
          )
          .join("\n")
      );
    } else {
      setClassificationResult("No classification results available.");
    }
  };

  const startClassification = async () => {
    try {
      const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
     
      if (!audioCtx) {
        const context = new (window.AudioContext || window.webkitAudioContext)({
          sampleRate: 16000,
        });

        setAudioCtx(context);
      } else if (audioCtx.state === "suspended") {
        await audioCtx.resume();
      }

      const source = audioCtx.createMediaStreamSource(stream);
      const bufferSize = 16384;
      await audioCtx.audioWorklet.addModule("/audio-processor.js")
  
      const worklet = new AudioWorkletNode(audioCtx, "worklet-processor", { processorOptions: { bufferSize }});
      

      worklet.port.onmessage = async (event) => {
        const inputData = event.data;
        if (workerRef.current) {
              workerRef.current.postMessage({ type: "classify", data: inputData });
            }
      }
      // processor.onaudioprocess = (event) => {
      //   const inputBuffer = event.inputBuffer.getChannelData(0);
      //   if (workerRef.current) {
      //     workerRef.current.postMessage({ type: "classify", data: inputBuffer });
      //   }
      // };

      source.connect(worklet);
      worklet.connect(audioCtx.destination);
      streamingRef.current = { stream, worklet, source };


      setIsClassifying(true);
    } catch (error) {
      console.error("Error accessing microphone: ", error);
    }
  };

package.json
"dependencies": {
"@mediapipe/tasks-audio": "^0.10.20",
"assert": "^2.1.0",
"cra-template": "1.2.0",
"difflib": "^0.2.4",
"microsoft-cognitiveservices-speech-sdk": "^1.41.0",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"react-scripts": "5.0.1",
"web-vitals": "^4.2.4"
},
@kuaashish kuaashish added platform:javascript MediaPipe Javascript issues task:audio classification Issues related to Audio Classification: Classify sounds into relevant tags os:windows MediaPipe issues on Windows type:support General questions labels Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
os:windows MediaPipe issues on Windows platform:javascript MediaPipe Javascript issues task:audio classification Issues related to Audio Classification: Classify sounds into relevant tags type:support General questions
Projects
None yet
Development

No branches or pull requests

2 participants