Neiro Assistant

Neiro Assistant is a lightweight local-first desktop AI assistant MVP with a web UI, FastAPI backend, mock Live2D avatar, local LLM provider interface, STT/TTS provider interfaces, personality profiles, and SQLite memory.

The default mode is mock mode. It runs without downloaded AI models and does not require a GPU.

Project Layout

apps/desktop      React + Vite TypeScript UI
backend           FastAPI backend and tests
assets/live2d     Local Live2D model assets
data/personalities JSON personality profiles
models            Local GGUF/STT/TTS models, ignored by git
scripts           Placeholder model setup scripts

The older root-level main.py, gui.py, and model.py prototype files are preserved.

Backend Setup

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r backend\requirements.txt
cd backend
python -m uvicorn main:app --host 127.0.0.1 --port 8000 --reload

Check the backend:

Invoke-RestMethod http://127.0.0.1:8000/health

Quick Launch For Testing

For the recommended low-latency interactive mode, double-click this file from the project root:

start_neiro_fast.bat

It starts the local fast profile with Piper TTS. This is the stable mode for chat responsiveness and avoids XTTS voice-clone word hallucinations.

For experimental XTTS voice-clone testing, double-click this file from the project root:

start_neiro_xtts.bat

It checks XTTS dependencies, prepares emotion voice files, optionally runs the first XTTS warmup/sample, and opens backend/frontend windows.

Manual launch still works if you want separate control. Open two PowerShell windows from the project root.

Window 1, safe mock backend:

.\scripts\start_backend_mock.ps1

Window 1, real fast local backend instead:

.\scripts\start_backend_local_fast.ps1

Window 1, real fast local backend with XTTS-v2 instead:

.\scripts\start_backend_local_xtts.ps1

Window 2, frontend:

.\scripts\start_frontend.ps1

Then open:

http://127.0.0.1:5174

OBS overlay URL:

http://127.0.0.1:5174/overlay.html?model=/assets/live2d/akari_vts/akari.model3.json&api=http://127.0.0.1:8000

Runtime check:

.\scripts\check_runtime.ps1

Warm the loaded LLM after backend startup:

Invoke-RestMethod -Method Post http://127.0.0.1:8000/runtime/warmup

If port 8000 is already occupied, start backend and frontend on a different backend port:

$env:APP_PORT="8010"; .\scripts\start_backend_mock.ps1
$env:NEIRO_BACKEND_PORT="8010"; .\scripts\start_frontend.ps1

Frontend Setup

cd apps\desktop
npm install
npm run dev

Open the Vite URL, usually http://127.0.0.1:5173.

Mock Mode

Mock mode is configured by default in .env.example:

LLM_PROVIDER=mock
STT_PROVIDER=mock
TTS_PROVIDER=mock

This mode supports chat, push-to-talk simulation, mock TTS events, avatar emotion changes, and basic lip-sync values without model downloads.

Streaming Chat And Early Speech

The UI uses /chat/stream for live responses. The backend broadcasts partial assistant_token events over /ws/events, then sends assistant_final when generation is complete. The frontend updates the assistant message as tokens arrive and sends completed sentence chunks to TTS immediately, so the avatar can start speaking before the full answer is generated.

The blocking /chat endpoint remains available as a compatibility fallback.

Live2D Model

Place a Live2D Cubism model under assets/live2d and set:

LIVE2D_MODEL_PATH=./assets/live2d/model.model3.json

Vite serves the root assets/ directory as public assets. The frontend attempts to load /live2d/model.model3.json by default. If loading fails, it displays the mock avatar.

For Cubism 3/4 models, the browser also needs live2dcubismcore.min.js. You can place it at one of these paths:

• assets/live2dcubismcore.min.js
• assets/live2d/live2dcubismcore.min.js

If you have the file in another location, set:

VITE_LIVE2DCUBISMCORE_URL=<full_url_to_live2dcubismcore.min.js>

The frontend will also try the direct Cubism CDN fallback.

OBS Avatar Overlay

The desktop app also exposes a visual-only overlay page for OBS browser sources:

http://127.0.0.1:5174/overlay.html?model=/assets/live2d/akari_vts/akari.model3.json&api=http://127.0.0.1:8000

Use this URL as an OBS Browser Source and enable transparent background support in OBS. The overlay renders only the Live2D avatar, connects to the backend WebSocket events, and reacts to emotion and lip-sync updates. The Settings panel shows a copyable overlay URL for the currently selected model.

A later Tauri shell can host the same overlay route in a transparent always-on-top desktop window with click-through and position controls.

Local LLM

Put GGUF models in models/ and configure:

LLM_PROVIDER=llama_cpp
LLM_MODEL_PATH=./models/your-model.gguf

Then install the optional provider:

pip install llama-cpp-python

The mock provider remains available for development.

Runtime Profiles

The backend exposes runtime profiles at:

http://127.0.0.1:8000/runtime/profiles

Runtime status:

http://127.0.0.1:8000/runtime/status

The llama.cpp model is loaded once when the backend process starts and stays resident until that process exits. Do not use auto-reload for normal testing if you want to avoid reloading the model. Changing model path, GPU layers, or context size requires restarting the backend.

Current profiles:

• mock: no downloaded models, safest for UI and Live2D testing.
• local_fast: recommended interactive mode, Qwen3-4B-Q8_0.gguf, shorter context and response budget, faster-whisper CUDA, Piper TTS.
• local_quality: heavier 8B profile for better replies when latency is acceptable.

For responsive character interaction, prefer local_fast first. The current 8B Q8 model can feel slow when GPU memory is busy.

Speech-To-Text

The STT interface supports push-to-talk first. To try faster-whisper:

STT_PROVIDER=faster_whisper
STT_MODEL_SIZE=tiny
STT_MODEL_PATH=./models/stt/faster-whisper-tiny
STT_DEVICE=cpu
STT_COMPUTE_TYPE=int8

Install the optional dependency:

pip install faster-whisper

Always-listening mode is intentionally left as a later extension.

Text-To-Speech

TTS providers are selected with:

TTS_PROVIDER=mock
TTS_PROVIDER=piper
TTS_PROVIDER=silero
TTS_PROVIDER=xtts
TTS_PROVIDER=bark

mock works without downloads. piper is the fastest local option, silero is a better Russian CPU-friendly option, and xtts is the higher quality reference-voice option. Heavy dependencies stay optional.

For a more natural Russian assistant voice, try Silero:

cd backend
..\.venv\Scripts\python.exe -m pip install torch
cd ..
.\scripts\sample_silero_voices.ps1

Recommended speakers for the current anime assistant are xenia first, then baya, then kseniya.

For a higher quality voice closer to AI VTuber style, use XTTS-v2 with your own legally usable reference voice:

.\scripts\setup_xtts.ps1
mkdir data\voices
# Put a clean 6-30 second voice sample here:
# data\voices\neiro_reference.wav
.\scripts\sample_xtts_voice.ps1

setup_xtts.ps1 installs CUDA PyTorch for RTX GPUs and Coqui XTTS dependencies into backend\.venv312. This can download several GB on the first run.

Then set:

TTS_PROVIDER=xtts
TTS_DEVICE=cuda
XTTS_REFERENCE_WAV=./data/voices/neiro_reference.wav
XTTS_REFERENCE_NEUTRAL=./data/voices/neiro_neutral.wav
XTTS_REFERENCE_HAPPY=./data/voices/neiro_happy.wav
XTTS_REFERENCE_SAD=./data/voices/neiro_sad.wav
XTTS_REFERENCE_ANGRY=./data/voices/neiro_angry.wav
XTTS_REFERENCE_SHY=./data/voices/neiro_shy.wav
XTTS_LANGUAGE=ru

Emotion-aware XTTS uses these mappings:

neutral -> neiro_neutral.wav
happy -> neiro_happy.wav
sad -> neiro_sad.wav
annoyed -> neiro_angry.wav
embarrassed -> neiro_shy.wav
surprised -> neiro_happy.wav
thinking -> neiro_neutral.wav

If you exported files as Take1...Take5, prepare stable local copies:

.\scripts\prepare_xtts_emotion_voices.ps1

First XTTS run downloads the model and may ask you to confirm Coqui CPML/non-commercial terms in the terminal. After the model is downloaded, launch the app with:

.\scripts\start_backend_local_xtts.ps1
.\scripts\start_frontend.ps1

XTTS-v2 can clone from a short sample and supports Russian, but the official model license is non-commercial. Use a voice you created yourself, bought/licensed, or have explicit permission to use. Do not clone streamers, VTubers, actors, or celebrity voices without rights.

Good places to find or create a reference voice:

• ElevenLabs Voice Library - easiest way to browse/generate polished voices, cloud-based.
• Fish Audio - has many AI voice models, check each voice license before use.
• Your own recorded voice processed into an anime style, then exported as WAV.

Public sources do not show an official downloadable Neuro-sama voice. Neuro-sama is known to use a synthetic TTS/AI voice, and fan communities mention services like ElevenLabs, but those are not a clean license to copy her exact voice.

Personality And Memory

Personality profiles are JSON files in data/personalities/*.json. The backend creates three presets on startup:

• default_friendly
• calm_analytical
• playful_teasing

Memory is stored in SQLite at data/assistant.db with tables for messages, memories, user preferences, and conversation summaries. MVP search uses simple keyword scoring, with an embeddings interface left for later.

Tests

cd backend
pytest

Frontend helper tests:

cd apps\desktop
npm run test:chunker

Current tests cover prompt building, personality schema validation, memory repository behavior, and mock chat.

TODO

• Add a Tauri wrapper once the web UI flow settles.
• Add real microphone recording in the frontend.
• Add tool-use actions for programs and external local services.
• Add a voice picker UI for Piper/Silero/XTTS samples.
• Add real Live2D expression and motion mapping per model.