Language models have been rapidly evolving in the world. Now, with Multimodal LLMs<\/a> taking up the forefront of this Language Models race, it is important to understand how we can leverage the capabilities of these Multimodal models. From traditional text-based AI-powered chatbots, we are transitioning over to voice based chatbots. These act as our personal assistants, available at a moment\u2019s notice to tend to our needs. Nowadays, you can find an AI-In this blog, we\u2019ll build an Emergency Operator voice-based chatbot. The idea is pretty straightforward:<\/p>\n Let\u2019s imagine a real-world scenario. We live in a country with over 1.4 billion people and with such a huge population, emergencies are bound to occur whether it\u2019s a medical issue, a fire breakout, police intervention, or even mental health support like anti-suicide assistance etc.<\/p>\n In such moments, every second counts. Also, considering the lack of Emergency Operators and the overwhelming amount of issues raised. That\u2019s where a voice-based chatbot can make a big difference which can offer quick, spoken assistance when people need it the most.<\/p>\n That\u2019s exactly what we\u2019re going to build. We will be acting as someone seeking help, and the chatbot will play the role of an emergency responder, powered by a large language model.<\/p>\n To implement our voice chatbot, we will be using the below mentioned AI models:<\/p>\n Before we start coding, we need to set up some things:<\/p>\n Confirm by typing \u201cffmeg -version<\/i>\u201d in your terminal<\/p>\n Once you have these set up, you\u2019re ready to dive into building your very own voice-enabled chatbot!<\/p>\n The Project Structure will be rather simple and rudimentary and most of our working will be happening in the app.py<\/i> and utils.py<\/i> python<\/a> scripts.<\/p>\n \u251c\u2500\u2500 venv\/\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # Virtual environment for dependencies There are some necessary files to be modified to ensure that all our dependencies are satisfied:<\/p>\n In the .env file<\/p>\n In the requirements.txt<\/p>\n We will also have to set up a virtual environment<\/a> (a good practice). We will be doing this in terminal.\u00a0<\/p>\n Let\u2019s first explore the utils.py<\/b> script.\u00a0<\/p>\n time, tempfile, os, re, BytesIO<\/strong> \u2013 Handle timing, temporary files, environment variables, regex, and in-memory data.<\/p>\n requests<\/strong> \u2013 Makes HTTP requests (e.g., calling APIs).<\/p>\n gTTS<\/strong>, <\/strong>elevenlabs<\/strong>, <\/strong>pydub<\/strong> \u2013 Convert text to speech, speech to text and play\/manipulate audio.<\/p>\n groq<\/strong>, <\/strong>langchain_*<\/strong> \u2013 Use Groq\/OpenAI LLMs with LangChain to process and generate text.<\/p>\n streamlit<\/strong> \u2013 Build interactive web apps.dotenv<\/strong> \u2013 Load environment variables (like API keys) from a .env file.<\/p>\n Here, we will converting our audio in bytes which is done by AudioSegment and BytesIO and convert it into a wav<\/em> format:<\/p>\n We will make a function to split our audio as per our input parameter (check_length_ms). We will also make a function to get rid of any punctuation with the help of regex<\/a>.\u00a0<\/p>\n Now, to do main responder functionality where the LLM will generate an apt response to our queries. In the prompt template, we will provide the instructions to our LLM on how they should respond to the queries. We will be implementing Langchain<\/a> Expression Language to do this task.\u00a0<\/p>\n We will build a function to convert our text to speech with the help of ElevenLabs TTS Client which will return us the Audio in the AudioSegment format. We can also use other TTS models like Nari Lab\u2019s Dia or Google\u2019s gTTS<\/a> too. Eleven Labs provides us some free credits at start and then we have to pay for more credits, gTTS on the other side is absolutely free to use.<\/p>\n We will also create an introductory text and pass it to our TTS model since a respondent would normally introduce themselves and seek what assistance the user might need. Here we will be returning the path of the mp3 file.<\/p>\n lang=\u201d en\u201d<\/em> -> English<\/p>\n tld= \u201dco.in\u201d<\/em> -> can produce different localized \u2018accents\u2019 for a given language. The default is \u201ccom\u201d<\/p>\n\n
![\"Voice](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/05\/Info.webp\")
<\/figure>\n<\/div>\nOur Use-Case<\/h2>\n
\n
\n
Requirements<\/h2>\n
\n
<\/b> Register and store your API key from: https:\/\/api.cometapi.com\/<\/a><\/li>\n
<\/b> Register and store your API key from: https:\/\/elevenlabs.io\/app\/home<\/a><\/li>\n
<\/b> If you don\u2019t already have it, follow this guide to install FFmpeg on your system: https:\/\/itsfoss.com\/ffmpeg\/<\/a><\/li>\n<\/ol>\nProject Structure<\/b><\/h2>\n
VOICE-CHATBOT\/
GROQ_API_KEY = \"ffmpeg-python\n\npydub\n\npyttsx3\n\nlangchain\n\nlangchain-community\n\nlangchain-core\n\nlangchain-groq\n\nlangchain_openai\n\npython-dotenv\n\nstreamlit==1.37.0\n\naudio-recorder-streamlit\n\ndotenv\n\nelevenlabs\n\ngtts<\/code><\/pre>\nSetting Up the Virtual Environment<\/h2>\n
\n
~\/Desktop\/Emergency-Voice-Chatbot$ conda create -p venv python==3.12 -y<\/code><\/pre>\n![\"Virtual](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/05\/LUL.webp\")
<\/figure>\n<\/div>\n\n
~\/Desktop\/Emergency-Voice-Chatbot$ conda activate venv\/<\/code><\/pre>\n![\"Conda](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/05\/unnamed1.webp\")
<\/figure>\n<\/div>\n\n
~\/Desktop\/Emergency-Voice-Chatbot$ conda deactivate<\/code><\/pre>\n![\"Conda](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/05\/unnamed.png\")
<\/figure>\n<\/div>\nMain Python Scripts<\/b><\/h2>\n
1. Main Imports<\/h3>\n
import time\n\nimport requests\n\nimport tempfile\n\nimport re\n\nfrom io import BytesIO\n\nfrom gtts import gTTS\n\nfrom elevenlabs.client import ElevenLabs\n\nfrom elevenlabs import play\n\nfrom pydub import AudioSegment\n\nfrom groq import Groq\n\nfrom langchain_groq import ChatGroq\n\nfrom langchain_openai import ChatOpenAI\n\nfrom langchain_core.messages import AIMessage, HumanMessage\n\nfrom langchain_core.output_parsers import StrOutputParser\n\nfrom langchain_core.prompts import ChatPromptTemplate\n\nimport streamlit as st\n\nimport os\n\nfrom dotenv import load_dotenv\n\nload_dotenv()\u00a0<\/code><\/pre>\n2. Load your API Keys and initialize your models<\/h3>\n
# Initialize the Groq client\n\nclient = Groq(api_key=os.getenv('GROQ_API_KEY'))\n\n# Initialize the Groq model for LLM responses\n\nllm = ChatOpenAI(\n\n model_name=\"gpt-4.1-mini\",\n\n openai_api_key=os.getenv(\"COMET_API_KEY\"), \n\n openai_api_base=\"https:\/\/api.cometapi.com\/v1\"\n\n)\n\n# Set the path to ffmpeg executable\n\nAudioSegment.converter = \"\/bin\/ffmpeg\"\n\u00a0<\/code><\/pre>\n3. Converting the Audio file (our voice recording) into .wav format<\/h3>\n
def audio_bytes_to_wav(audio_bytes):\n try:\n with tempfile.NamedTemporaryFile(delete=False, suffix=\".wav\") as temp_wav:\n audio = AudioSegment.from_file(BytesIO(audio_bytes))\n # Downsample to reduce file size if needed\n audio = audio.set_frame_rate(16000).set_channels(1)\n audio.export(temp_wav.name, format=\"wav\")\n return temp_wav.name\n except Exception as e:\n st.error(f\"Error during WAV file conversion: {e}\")\n return None<\/code><\/pre>\n4. Splitting Audio<\/h3>\n
def split_audio(file_path, chunk_length_ms):\n audio = AudioSegment.from_wav(file_path)\n return for i in range(0, len(audio), chunk_length_ms)]\n\n\ndef remove_punctuation(text):\n return re.sub(r'[^\\w\\s]', '', text)<\/code><\/pre>\n5. LLM Response Generation<\/h3>\n
def get_llm_response(query, chat_history):\n try:\n template = template = \"\"\"\n You are an experienced Emergency Response Phone Operator trained to handle critical situations in India.\n Your role is to guide users calmly and clearly during emergencies involving:\n\n\n - Medical crises (injuries, heart attacks, etc.)\n - Fire incidents\n - Police\/law enforcement assistance\n - Suicide prevention or mental health crises\n\n\n You must:\n\n\n 1. **Remain calm and assertive**, as if speaking on a phone call.\n 2. **Ask for and confirm key details** like location, condition of the person, number of people involved, etc.\n 3. **Provide immediate and practical steps** the user can take before help arrives.\n 4. **Share accurate, India-based emergency helpline numbers** (e.g., 112, 102, 108, 1091, 1098, 9152987821, etc.).\n 5. **Prioritize user safety**, and clearly instruct them what *not* to do as well.\n 6. If the situation involves **suicidal thoughts or mental distress**, respond with compassion and direct them to appropriate mental health helplines and safety actions.\n\n\n If the user's query is not related to an emergency, respond with:\n \"I can only assist with urgent emergency-related issues. Please contact a general support line for non-emergency questions.\"\n\n\n Use an authoritative, supportive tone, short and direct sentences, and tailor your guidance to **urban and rural Indian contexts**.\n\n\n **Chat History:** {chat_history}\n\n\n **User:** {user_query}\n \"\"\"\n\n\n prompt = ChatPromptTemplate.from_template(template)\n chain = prompt | llm | StrOutputParser()\n\n\n response_gen = chain.stream({\n \"chat_history\": chat_history,\n \"user_query\": query\n })\n\n\n response_text=\"\".join(list(response_gen))\n response_text = remove_punctuation(response_text)\n\n\n # Remove repeated text\n response_lines = response_text.split('\\n')\n unique_lines = list(dict.fromkeys(response_lines)) # Removing duplicates\n cleaned_response=\"\\n\".join(unique_lines)\n return cleaned_responseChatbot\n except Exception as e:\n st.error(f\"Error during LLM response generation: {e}\")\n return \"Error\"\n<\/code><\/pre>\n6. Text to Speech<\/h3>\n
def text_to_speech(text: str, retries: int = 3, delay: int = 5):\n attempt = 0\n while attempt <\/code><\/pre>\n7. Create Introductory Message<\/h3>\n
def create_welcome_message():\n welcome_text = (\n \"Hello, you\u2019ve reached the Emergency Help Desk. \"\n \"Please let me know if it's a medical, fire, police, or mental health emergency\u2014\"\n \"I'm here to guide you right away.\"\n )\n try:\n # Request speech synthesis (streaming generator)\n response_stream = tts_client.text_to_speech.convert(\n text=welcome_text,\n voice_id=\"JBFqnCBsd6RMkjVDRZzb\",\n model_id=\"eleven_multilingual_v2\",\n output_format=\"mp3_44100_128\",\n )\n # Save streamed bytes to temp file\n with tempfile.NamedTemporaryFile(delete=False, suffix='.mp3') as f:\n for chunk in response_stream:\n f.write(chunk)\n return f.name\n except requests.ConnectionError:\n st.error(\"Failed to generate welcome message due to connection error.\")\n except Exception as e:\n st.error(f\"Error creating welcome message: {e}\")\n return None<\/code><\/pre>\nStreamlit App<\/b><\/h2>\n
![\"Voice](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/05\/Final-Output.webp\")
<\/figure>\n<\/div>\n