Vehicle Autonomous Network & General Utility Assistant for Research and Diagnostics
VANGUARD is a production-grade, highly modular, and futuristic desktop AI assistant inspired by Tony Stark's J.A.R.V.I.S. and classic sci-fi autonomous vehicle computers. It is engineered to run locally (using Ollama) or via cloud providers (like OpenAI), coordinating real-time sensory telemetry, high-fidelity neural voice synthesis, computer vision with live hand gesture tracking, semantic vector memory, network security audits, and dynamic scripting plugins.
[ V.A.N.G.U.A.R.D. CORE ]
│
┌───────────────────┬─────────────────────┼─────────────────────┬───────────────────┐
▼ ▼ ▼ ▼ ▼
🎙️ Neural Voice 🗣️ Duplex Audio 👁️ Live Vision HUD 🚨 Sentry Watchdog 🧠 Vector Memory
(Edge-TTS Butler) (Hands-Free Loop) (Face & Gestures) (Photo Surveillance) (ChromaDB Recall)
- 🎙️ Ultra-Realistic Neural Speech Synthesis: Studio-quality Microsoft Edge-TTS with natural British Butler tone (
en-GB-RyanNeural), persona-specific voice mapping, and localpyttsx3offline fallback. - 🗣️ Continuous Hands-Free Full-Duplex Dialogue: Automated continuous conversation loop with instant Barge-In speech interruption (
Ctrl+Spaceor speech trigger). - 👁️ Multi-Modal Neural Vision Intelligence HUD (
F2): Real-time 30 FPS HUD with 90-Class Object Detection (MobileNet SSD), Eye Gaze & Drowsiness EAR Sentinel, 3D Head Pose Nod/Shake detection, Facial Emotion classification, 4-Point Document Auto-Cropping & Live OCR, QR Code decoding, and YCrCb Hand Gesture Radar. - 🎵 Autonomous Background Media & YouTube Audio Streamer: Streams high-fidelity music, tracks, and playlists directly in background via
yt-dlp+ headlessmpvIPC with voice controls ("play english song", "pause", "resume", "stop music"). - 🚨 Autonomous Sentry Watchdog & Photo Surveillance (
F4): Background security daemon that detects suspicious processes, high-CPU anomalies, and captures timestamped optical photos of unauthorized faces/intruders while the pilot is away. - 👤 Pilot Profile & Dossier Vault (
F3): Interactive SQLite credentials and personal dossier manager (Full Name, NIC, DOB, Contact, Blood Group, Emergency Contact, and Bio). - 🔒 Multi-Factor Biometric Security Login Gate: Startup security gate requiring optical face match, voiceprint passphrase, or SHA-256 SQLite NIC/Password fallback before unlocking dashboard HUD.
- 🧠 Long-Term Semantic Vector Memory: Local neural vector embeddings powered by ChromaDB for associative memory storage and semantic recall across months of sessions.
- 🛡️ Network Security, Threat Intel & Digital Forensics: Integrated Nmap fast port auditing, VirusTotal hash integrity, HashID algorithm recognition, Whois / GeoIP OSINT, Checksec binary mitigations, Linux Hardening Score, CyberChef multi-layer decoder, ExifTool GPS metadata inspection, and Strings reverse engineering.
- ⚛️ 3D Holographic Arc Reactor Core: Animated multi-ring energy core visualizer pulsating dynamically with speech amplitude and AI thinking states.
- 🔌 Extensible Plugin Subsystem: 37 dynamically discovered reflection-based plugins executed prior to querying the LLM.
| Subsystem | Technology | Description |
|---|---|---|
| Threat Intelligence & OSINT | VirusTotal & GeoIP API | File hash reputation, HashID identification, Whois, and web security header audits. |
| System Hardening & Checksec |
readelf & /proc
|
Binary exploit mitigations (PIE, NX, Canary, RELRO), SUID scan, and Linux hardening score. |
| CyberChef Crypto Decoder | Multi-Layer Deobfuscator | Base64, Hex, URL, Rot13/Caesar, Binary 8-bit, XOR transformations and magic decoding. |
| Digital Forensics & EXIF |
strings & exifread
|
GPS location extraction, metadata inspection/stripping, and binary strings analysis. |
| Media & Music Streamer |
yt-dlp & mpv IPC |
Headless background YouTube audio search, streaming, and voice controls. |
| Neural Vision Intelligence Suite | TensorFlow & OpenCV | 90-class object detection, live document OCR, QR decoding, and eye fatigue EAR monitor (F2). |
| Autonomous Sentry Watchdog |
OpenCV & psutil
|
Records unauthorized faces, suspicious processes, and presents photo evidence gallery (F4). |
| Biometric Security Login Gate | Dual-Tier Cascades | Face match, voiceprint passphrase, or SHA-256 NIC authentication gate at launch. |
| Pilot Profile & Dossier Vault | SQLite user_profile
|
Full profile management (Name, NIC: 871240612v, DOB, Phone, Blood Group) via F3. |
| Neural JARVIS Speech Engine |
edge-tts & pyttsx3
|
Natural British butler voice output with automatic local offline fallback. |
| Hands-Free Duplex Dialogue | Full-Duplex Loop | Continuous conversational back-and-forth speech without repeating wake-words. |
| Semantic Vector Memory | ChromaDB |
Local embedding database for persistent knowledge storage and associative recall. |
| Network Security & Nmap Audit |
nmap & tshark
|
Deep host auditing, open service evaluation, and packet traffic monitoring. |
| WiFi Device Radar | scapy |
Discovers all connected hardware, IP addresses, and MAC bindings on the local WiFi network. |
| Internet Speedtest Benchmark | speedtest-cli |
Real-time Ping latency, Download Mbps, and Upload Mbps measurement. |
| 3D Arc Reactor Hologram Core | Canvas Visualizer | Multi-ring rotating energy core visualizer pulsating dynamically with speech amplitude. |
| Screen Context Vision |
PIL & Vision API |
Captures high-res display buffers to explain errors, open windows, and code ("explain screen"). |
| Autonomous Dev Controller |
git & docker
|
Inspects Git status, active Docker containers, and launches IDE workspaces. |
| GUI Dashboard HUD | CustomTkinter |
Cyberpunk dark theme (#080808) with vertical scrollbar, real-time clock, and terminal console. |
| Modular Visualizer Bay | Native TK Canvas | 5 switchable HUD visualizer styles (Classic KITT, JARVIS Waveform, Neon Equalizer, Cybernetic ECG, Hex Data Stream) with F5 hotkey and voice control. |
| Neural Voice Gender Switcher | CTkOptionMenu |
Live HUD switcher toggling between Male and Female neural TTS voices (JARVIS, FRIDAY, Aria, Sonia, Thilini). |
| Bilingual Language Switcher | CTkSegmentedButton |
Live HUD switcher toggling between English (en-US) and Sinhala (si-LK) speech models. |
| Autonomous Security Alerts | Background Poller | Voice alerts on high CPU/RAM load or critical battery level ( |
| Spoken Startup Briefing | BriefingPlugin |
Executive vocal briefing summarizing system load, disk status, and weather on boot or demand. |
| Interactive Help Center | VanguardHelpModal |
F1 shortcut and header button opening interactive command manual. |
| System Backup & Restore | SystemBackupPlugin |
Archives configuration settings, SQLite database, and vector memory into zip packages. |
| Standalone Installer & Builder |
build.sh & install.sh
|
Automated Linux installer and PyInstaller standalone binary compiler. |
| Shortcut | Action | Description |
|---|---|---|
Ctrl + Space |
Global Push-To-Talk | Activates voice input from anywhere across the OS (even when minimized). |
F1 |
User Manual & Help Center | Opens the interactive modal with all voice directives and keybindings. |
F2 |
Neural Vision Intelligence HUD | Opens live optical radar with 90-class object detection, eye fatigue EAR, OCR, and gestures. |
F3 |
Pilot Dossier & Profile Vault | Opens credentials and personal profile manager (Name, NIC: 871240612v, DOB, Blood Group). |
F4 |
Sentry Surveillance Gallery | Opens autonomous security incidents log and captured intruder photo evidence gallery. |
F5 |
Cycle HUD Visualizer Bay | Cycles top header visualizer between Classic KITT, JARVIS Waveform, Neon Equalizer, Cybernetic ECG, and Hex Data Stream. |
F11 / Esc |
Toggle Fullscreen | Expands dashboard into full-screen cockpit HUD or returns to windowed mode. |
Ctrl + M |
Tray Badge Mode | Minimizes main dashboard into a floating desktop mini-scanner badge and arms Sentry. |
KITT/
├── config/
│ └── settings.json # Main configuration file (JSON, hot-reloading)
├── database/
│ ├── memory.db # SQLite relational database (chat history, profiles, security events)
│ └── chroma_db/ # ChromaDB local vector embeddings database
├── logs/
│ ├── system.log # System diagnostics, warnings, and tracebacks
│ ├── conversations.log # Transcripts of chat history
│ └── commands.log # Command execution logs
├── assets/
│ ├── images/ # Concept mockups and UI graphics
│ ├── models/ # OpenCV Haar cascades, TensorFlow SSD MobileNet V2 neural weights
│ ├── surveillance/ # Timestamped intruder and security incident snapshots (.jpg)
│ ├── sounds/ # Sci-Fi audio SFX waveforms (.wav)
│ ├── screenshots/ # Captured desktop and webcam vision frames
│ └── backups/ # Archived system backup packages (.zip)
├── plugins/ # Extensible script plugins (37 dynamic plugins)
│ ├── alarm_clock.py # Specific clock-time alarm scheduler
│ ├── backup.py # System configuration and SQLite backup archive
│ ├── benchmark.py # Hardware CPU floating-point benchmark test
│ ├── briefing.py # Spoken startup and status report plugin
│ ├── browser.py # Web browser navigation and search plugin
│ ├── calculator.py # Safe math formula evaluation plugin
│ ├── cleaner.py # Garbage collection and memory cache cleaner
│ ├── clipboard_ai.py # Desktop clipboard reader and AI summarizer
│ ├── crypto_decoder.py # Multi-layer CyberChef decoding (Base64, Hex, URL, Rot13, XOR)
│ ├── dev_controller.py # Git repository monitor, Docker inspector, and IDE launcher
│ ├── exporter.py # Diagnostic Markdown report exporter
│ ├── face_presence.py # Optical user presence and facial sensor sentinel
│ ├── file_finder.py # Workspace file search and locator
│ ├── forensics_tool.py # ExifTool metadata inspection, GPS mapper, and Strings binary analysis
│ ├── lan_discovery.py # Scapy ARP WiFi network device discovery
│ ├── log_search.py # Database and system log search utility
│ ├── macros.py # Automated multi-action voice macro routines
│ ├── music_player.py # Autonomous YouTube music streamer via yt-dlp & MPV IPC
│ ├── net_audit.py # Nmap security port audit and packet inspection
│ ├── network_radar.py # Local IP detector and common port scanner
│ ├── process_killer.py # Safe background process and app killer
│ ├── profile_plugin.py # Pilot dossier viewer and profile editor launcher
│ ├── reminders.py # Voice-activated countdown timers and reminders
│ ├── screen_vision.py # Desktop display screen context vision analysis
│ ├── security_vault.py # Base64 cipher text encryption and decryption
│ ├── sentry_plugin.py # Sentry incident logs and intruder evidence viewer launcher
│ ├── soundboard.py # Manual SFX soundboard audio trigger
│ ├── speedtest_plugin.py # Internet bandwidth speedtest benchmark
│ ├── sys_specs.py # Hardware specs and boot uptime telemetry
│ ├── system_control.py # Master volume, screenshot, and screen lock
│ ├── system_hardening.py # Checksec binary exploit mitigations, SUID scan, and hardening score
│ ├── threat_intel.py # VirusTotal hash analysis, HashID, Whois/GeoIP, and security headers
│ ├── vector_memory.py # ChromaDB semantic memory storage and recall
│ ├── vision.py # Webcam snapshot and live vision HUD launcher
│ ├── vision_advanced.py # Multi-modal neural vision commands (Objects, OCR, QR, Drowsiness)
│ ├── weather.py # Environmental weather survey reports
│ └── web_search.py # Live DuckDuckGo web search and news scraper
├── main.py # Core orchestrator and application entry point
├── ui.py # CustomTkinter dashboard, modals, and visualizers
├── vision_engine.py # Master multi-modal neural vision suite (90 objects, EAR, OCR, QR)
├── media_engine.py # Headless MPV IPC & yt-dlp YouTube audio streaming daemon
├── sentry.py # Autonomous security watchdog & intruder photo surveillance daemon
├── ai.py # Conversational AI connector (OpenAI / Ollama)
├── voice.py # Neural Edge-TTS synthesis & speech recognition
├── database.py # SQLite DatabaseManager & ChromaDB VectorMemoryStore
├── diagnostics.py # Telemetry diagnostics monitor scraper
├── commands.py # Plugin manager and reflection loader
├── scanner.py # 16-segment horizontal LED scanner widget
├── utils.py # Logging helpers, soundboard, and autostart setup
├── build.sh # Standalone PyInstaller executable build script
├── install.sh # Automated Linux system installer script
└── requirements.txt # Python package dependencies
Run the automated installer to configure system audio, speech, computer vision, network security tools, and autostart entries:
chmod +x install.sh build.sh
./install.sh# 1. Install System Dependencies
sudo apt-get update
sudo apt-get install -y python3-pyaudio espeak-ng scrot xclip nmap tshark wireshark mpv ffmpeg binutils
# 2. Configure Non-Root Packet Capture
sudo usermod -aG wireshark $USER
# 3. Install Python Dependencies
pip install -r requirements.txt# 1. Start Ollama service
ollama serve
# 2. Pull your preferred model
ollama pull llama3.2python3 main.py- 🤖 Custom Local AI Model Engine & Offline Autonomy:
- VANGUARD creates its own dedicated local AI model (
vanguard:latest) via Ollama with embedded persona, system instructions, and etiquette. - "Switch to local model" / "Use vanguard model" / "Offline model" (runs 100% locally on your machine)
- "Switch to cloud model" / "Use tokenrouter" (routes through high-throughput cloud gateway)
- "Model status" / "Check model" (inspects active model, provider, and offline state)
- "Rebuild model" / "Build vanguard model" (recompiles
Modelfileon the fly) - "List models" (shows all available local Ollama models)
- VANGUARD creates its own dedicated local AI model (
- 🛡️ Continuous Network Activity Sentinel & Intrusion Defense:
- Automatically monitors all inbound & outbound TCP/UDP traffic in real-time.
- Pops up an interactive
VanguardIntrusionAlertModalwith voice warnings when unauthorized connections or remote access attempts (SSH, RDP, VNC, Telnet, C2 ports) are intercepted. - One-click confirmation to block target IPs via host firewall (
sudo ufw insert 1 deny from <IP> to any) and SQLite database. - Commands: "Network activity", "Block ip ", "Unblock ip ", "Whitelist ip ", "Blocked ips".
- 🎵 Autonomous Music & Audio Streaming:
- "Play a English song" or "Play Believer Imagine Dragons"
- "Pause music" / "Resume music" / "Stop music" / "Now playing"
- 🛡️ Digital Forensics & Metadata Analysis:
- "Exif assets/images/dashboard.jpg" or "Inspect metadata <photo.jpg>"
- "Strings /bin/ls" or "Scan binary "
- "Strip metadata <photo.jpg>"
- 👁️ Neural Vision Intelligence HUD (
F2):- Press
F2or say "Scan objects", "Read text from camera", "Scan QR code", "Check drowsiness"
- Press
- 🚨 Autonomous Sentry Surveillance (
F4):- Press
F4or say "Sentry status", "Who was at my PC", "Show captured images"
- Press
- 👤 Pilot Profile & Dossier Vault (
F3):- Press
F3or say "Open profile", "My profile", "Who am I"
- Press
- 🧠 Semantic Vector Memory:
- "Remember this server port is 8080 and dev environment is docker"
- "What do you remember about server port?"
- 💻 Screen Context Vision:
- "Explain screen" or "What is on my screen?"
- 🛠️ Developer Workspace & Container Status:
- "Prepare dev environment" or "Check git status" / "Docker status"
- 🌐 Network Security & WiFi Radar:
- "Nmap scan 127.0.0.1"
- "Who is on my wifi" / "Discover devices"
- "Test speed" / "Check bandwidth"
- ⚡ System Controls & Diagnostics:
- "Status report" / "Briefing"
- "Clean memory" / "Free RAM"
- "Set volume 60%" / "Take screenshot" / "Lock PC"
- 🌐 Web & News:
- "Search web AI breakthroughs" / "News updates"
To bundle VANGUARD into a single, standalone Linux binary executable (no Python installation required on target systems):
./build.shThe compiled binary executable will be saved at:
dist/VANGUARD/VANGUARD
Adding new capabilities to VANGUARD is completely modular. Create a new .py file in plugins/ that inherits from BasePlugin:
# plugins/my_custom_plugin.py
from typing import List, Dict, Any
from commands import BasePlugin
class CustomSentinelPlugin(BasePlugin):
@property
def name(self) -> str:
return "CustomSentinel"
@property
def description(self) -> str:
return "Executes custom automated routines."
@property
def commands(self) -> List[str]:
return ["custom routine", "execute sentinel"]
def execute(self, trigger: str, args: str, context: Dict[str, Any]) -> str:
return "CUSTOM SENTINEL EXECUTED: All parameters nominal."VANGUARD automatically discovers, imports, and registers your plugin upon startup!