A Chrome extension that captures audio from the current tab, transcribes it using a self-hosted Whisper API, and optionally formats the text using a self-hosted Starcoder2 model.

• High-Quality Audio Capture: Records tab audio at 48kHz using AudioWorklet.
• Dual Audio Routing: Allows you to hear the audio while it's being recorded.
• Self-Hosted Transcription: Uses OpenAI's Whisper model running on your own server for privacy and control.
• AI Text Formatting:
  • Raw: Verbatim transcription.
  • General: Fixes punctuation, capitalization, and removes filler words.
  • Appen: Strict formatting rules for professional transcription.
• Dedicated Transcript View: Opens a separate tab to display results, play back audio, and download transcripts.

• Manifest V3: Modern extension architecture.
• Offscreen Document: Handles audio capture and processing to keep the extension responsive.
• AudioWorklet: Efficiently processes raw audio samples (PCM) and encodes them to WAV.
• Popup UI: Simple interface to control recording and settings.

The extension communicates with a backend hosted on https://xnet-ai.xnet.ngo (or your own domain).

• Caddy: Reverse proxy handling HTTPS and routing.
• Whisper: FastAPI service running OpenAI's Whisper model (Port 8001).
• Starcoder2: FastAPI service running BigCode's Starcoder2 model for text formatting (Port 8003).
• Coqui TTS: (Optional) Text-to-Speech service (Port 8000).

1. Clone this repository.
2. Open Chrome and navigate to chrome://extensions/.
3. Enable Developer mode in the top right corner.
4. Click Load unpacked.
5. Select the extension folder from this repository.

You need a server with a GPU (NVIDIA T4 or better recommended) and Docker installed.

1. Network Setup:

   docker network create xnet-bridge

2. Run Whisper Container:

   docker run -d --name whisper --network xnet-bridge -p 8001:8001 --gpus all whisper-image

   Note: Ensure ffmpeg is installed in the container.

3. Run Starcoder2 Container (For Formatting):

   # Ensure main_code.py is configured for FP16 and low_cpu_mem_usage if on 16GB RAM
   docker run -d --name starcoder2 --network xnet-bridge -p 8003:8003 --gpus all starcoder2-image

4. Configure Caddy: Update your Caddyfile to route traffic:

   xnet-ai.xnet.ngo {
       route /transcribe* {
           reverse_proxy http://whisper:8001
       }
       route /generate-code* {
           reverse_proxy http://starcoder2:8003
       }
   }

1. Open the Extension: Click the Whisper icon in your toolbar.
2. Initialize: Click "Initialize (Open Tab)" to open the dedicated transcript view.
3. Select Formatting: Choose "General", "Appen", or "None".
4. Start Recording: Click "Start Recording". The audio from the active tab will be captured.
5. Stop Recording: Click "Stop".
6. View Results: The transcript will appear in the dedicated tab. You can play back the recorded audio or download the text.

• 502 Bad Gateway: Check if the Docker containers are running and connected to the xnet-bridge network.
• Formatting Failed (500 Error): The Starcoder model might be crashing due to memory issues. Ensure it is loading in FP16 mode (torch_dtype=torch.float16) and low_cpu_mem_usage=True.
• No Audio: Ensure the tab is playing audio before you start recording.

MIT