fren is a minimal ambient AI companion that lives on your desktop — macOS, Windows or Linux. It is a small floating sphere with a face lit from within. When the light is on and the eyes are open, fren is observing what you do — which app is active and what the window title says. When the light goes out and the eyes close, it observes nothing at all. That signal and the capture pipeline share one source of truth in the Electron main process, so what you see is what is happening.
fren's long-term loop is:
OBSERVE → UNDERSTAND → REMEMBER → DETECT PATTERNS → SUGGEST → AUTOMATE
Version 0.1 implements the loop through SUGGEST. fren observes, remembers, summarises, and looks across hours of those summaries for a workflow you repeat. When it finds one it either says so or lights up and waits — which of those depends on what you told it during setup. Version 0.2 opens the AUTOMATE stage behind a runtime seam: with a secure execution environment available, fren can run an agent for a chat request or on a schedule ("every morning at 9, check Hacker News…") and show you what it did. The environment is optional and replaceable; see docs/runtime-architecture.md. Without it fren still observes, remembers, suggests and answers from memory.
Restraint is the hard part, not detection. A companion that volunteers something every ten minutes gets muted on day one, and a muted companion has failed completely — so a pattern must recur at least three separate times, clear a confidence floor, and never have been raised before.
This is a proof of concept, deliberately small: plain CommonJS JavaScript and no build step. The one exception is the face — it renders with WebGL via a vendored copy of three.js (~740KB, checked in, unmodified), because measurement showed the 3D renderer costs about a third of the CPU the SVG one did while looking considerably better. The SVG renderer is still in the tree and is used automatically if WebGL is unavailable.
That an ambient companion which quietly builds a semantic memory of your work can produce one specific moment of value: fren notices a repeated workflow you never told it about — "every morning you open the same three apps and copy data from A to B" — and offers to help. Everything in this codebase is in service of getting to that moment cheaply. If the moment never lands, the rest does not matter.
v0.1 tests the necessary precondition: can a timeline of app names and window titles, summarized by a small model every two minutes, produce memories that are accurate and useful enough to chat with?
┌──────────────────────────── Electron desktop app ───────────────────────────┐
│ │
│ mascot UI observer memory summarizer │
│ (sphere, ←→ (samples every 5s: → (SQLite, ←→ (every 2 min: │
│ chat panel) app + window title, local only) timeline → │
│ stored locally) gateway, │
│ stores a │
│ memory) │
└──────────────────────────────────────┬──────────────────────────────────────┘
│ HTTP, 127.0.0.1:4519, bearer token
│ (app names, titles, timestamps,
│ chat text — NEVER screenshots)
▼
┌──────────────────┐ ┌───────────────┐
│ local gateway │ ───→ │ DeepSeek or │
│ (holds API key, │ │ Anthropic │
│ prompts) │ │ (or mock) │
└──────────────────┘ └───────────────┘
- The desktop app samples your active app and window title every 5 seconds. It takes no screenshots on its own. Raw observations go into a local SQLite database.
- Every 2 minutes, the summarizer sends the recent app/window timeline — text only, never screenshots — to the local gateway, which asks the model for a compact summary ("debugging the auth flow in VS Code and Chrome"). That summary is stored as a memory. Old raw data is pruned.
- The gateway is a separate local process that holds the API key and all prompts. It picks a provider from whichever key is present — DeepSeek or Anthropic — and falls back to an offline mock when there is none. The desktop app never touches provider credentials and never learns which model answered, so providers can be swapped without changing the client.
- The chat panel lets you ask questions; fren answers using its stored memories.
See docs/architecture.md for the full picture.
Requirements: Node.js >= 23 (for built-in node:sqlite), on macOS, Windows or
Linux.
Reading which window is in front is the one genuinely platform-specific thing fren does, and each platform pays a different price for it:
| Platform | App name | Window title |
|---|---|---|
| macOS | lsappinfo, no permission needed |
osascript, needs Accessibility |
| Windows | PowerShell, no permission needed | same call, no permission needed |
| Linux (X11) | xdotool + xprop |
xdotool |
| Linux (Wayland) | xdotool under XWayland, else unavailable |
not available — the protocol does not permit it |
Where a title cannot be read, fren records the app name alone and says so in the log rather than pretending. On Wayland that is the ceiling, not a bug.
npm install
cp .env.example .env # set DEEPSEEK_API_KEY or ANTHROPIC_API_KEY (empty = mock mode)
npm start # starts the gateway and the desktop app togetherWithout an API key the gateway runs a mock provider — everything works offline, with canned summaries and chat replies. Useful for development.
Other commands:
npm run gateway # gateway only
npm run desktop # desktop app only (expects a gateway on 127.0.0.1:4519)
npm test # node --test suites for memory, intelligence, gateway, desktopThe first time you wake fren up, macOS will prompt for Accessibility and register the app under Screen Recording in System Settings (newer macOS versions prompt for it too; on older ones, enable it manually and restart). In dev the app shows up as "Electron", because it runs under the stock Electron binary:
- Screen Recording — only for the one screenshot fren takes when you ask it to look at your screen. Without it, that look is refused; nothing else changes.
- Accessibility — needed for window titles. Without it, fren degrades to app names only.
- Microphone — only for push-to-talk. Declining disables the mic button; typing still works.
Permission changes require an app restart.
Both halves are optional and independent.
Listening is local. Install whisper.cpp and a model:
brew install whisper-cpp
mkdir -p ~/.cache/whisper.cpp
curl -L -o ~/.cache/whisper.cpp/ggml-base.en.bin \
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.binBoth are auto-detected. Then click the orb to talk, or hold the mic button in the panel, and release to send. Nothing is recorded unless the button is down, and the audio is transcribed on this machine — it never leaves it.
Speaking uses ElevenLabs. Put ELEVENLABS_API_KEY in .env. Without it
fren simply replies in text. When it is set, the reply text (only) is sent to
ElevenLabs and the mouth is driven by the returned audio's own amplitude.
Everything lives locally in Electron's userData folder:
~/Library/Application Support/fren/
├── fren.db # SQLite: observations, memories, suggestions
└── (screenshots) # only from earlier versions, which kept them; they age out within 7 days
Raw observations are kept for 7 days. To delete everything fren knows: quit the app and delete that folder. There is no cloud copy to chase down.
Only this, and only via the local gateway:
- app names, window titles, and timestamps (the activity timeline)
- your typed chat questions
- derived activity summaries and the recent raw timeline (up to the last 50 observed app/title entries), sent as context with each chat message
- the transcript of anything you say with push-to-talk — never the audio
- the text of fren's reply, if you configured a voice
Looking at your screen is opt-in per use: press the eye button next to the message box and fren sends one screenshot to answer that one question. There is no setting to leave on, it announces itself before sending, it refuses while paused, and the image is never written to disk.
It needs a model that can see, and DeepSeek's chat models cannot. Vision is
picked separately from text, so this adds a capability without changing
which model answers your chat. Any OpenAI-compatible vision endpoint works —
set FREN_VISION_API_KEY (and optionally FREN_VISION_BASE_URL and
FREN_VISION_MODEL) and point it wherever you like. It defaults to Alibaba's
qwen-vl-plus, which is roughly a fifth the price of the cheapest Claude for
this kind of work. An ANTHROPIC_API_KEY is used as a fallback if you already
have one.
What never leaves your machine:
- a picture of your screen you did not ask for: fren takes none on its own (a test enforces it; earlier versions kept local ones, which age out)
- microphone audio (recorded only between the click that starts it and the one that stops it, transcribed locally by whisper.cpp, then deleted)
- the SQLite database
- keystrokes (never captured at all, by anything)
Full details in docs/privacy.md.
Click the orb and speak. Click again to send. You do not need the chat panel open — the reply is spoken aloud, and the panel is there for reading back what was said rather than as the way in.
While the microphone is open a ring pulses around the orb. It is drawn outside the character rather than as part of its face, so it reads the same whether fren is lit and watching or dark and paused — that is the moment you most need to know the microphone is open, not the least.
| Gesture | What happens |
|---|---|
| Left click | Starts recording; click again to stop and send |
| Right click | Opens the menu — the chat panel |
| Press and move | Carries fren anywhere on screen |
| Scroll | Makes fren bigger or smaller, and it stays that size |
A click and a carry are told apart by what your hand does, not by a timer: move past a few pixels and you are carrying it, and putting it down never starts or stops a recording.
A recording stops itself after two minutes. Holding a button had your hand as its natural bound; a click does not, so it needs one.
You can talk over it. Clicking the orb while fren is replying cuts the reply short and starts listening, and anything you say while it is still thinking is answered next rather than discarded.
While it is listening you can tell. A short tone marks the microphone opening and another marks it closing, and the orb turns brighter and glossier and pulses with your voice — the level comes from the live input, so it is visibly hearing you rather than just showing a state.
If your machine is muted, fren opens the chat panel instead of talking to an empty room. It asks the system before it speaks; where that cannot be asked, it assumes you can hear it rather than popping a panel you did not need.
The first time you run fren it wakes up, says hello, and asks five short questions — your name, what you are working on, how it should talk to you, whether it should speak up or stay quiet, and what would actually be useful.
It asks out loud, and the chat panel stays shut: click the orb to answer. The panel only opens if you open it, or if fren has no voice configured and therefore cannot be heard at all. Saying "skip" ends the interview, and so does pausing fren.
The last two answers are not about you — they define fren. They are written to
SOUL.md in fren's data folder as instructions it is told to follow, and you
can open that file, rewrite it, and have it take effect on your next message.
With the secure execution environment ready (the dot in the chat header says so, and its tooltip says what is missing when it is not), say it, spoken or typed, the way you would to a person. Three shapes are understood: a schedule ("every morning at 9, check Hacker News and give me the five most interesting AI stories"), a single later moment ("tomorrow at 3, remind me to call Ana", "in twenty minutes, tell me to stretch"), and a whenever ("whenever I open Figma, remind me to check the design tokens", "when I'm on github.com, list my open pull requests"). fren says the proposal back, the time or the trigger and the task in words, with a Keep it chip; a spoken or typed "yes" keeps it and a "no" lets it go. Nothing is created until you keep it. To see what is kept, ask: "what are you running?" brings up a card for each automation and routine, with when it next runs, what it said last time, and chips to pause, resume, run now or delete. "Pause the Hacker News one" works too. What an automation finds arrives in the chat, spoken like any other reply. A moment comes once: after it, the automation is done and off. A whenever runs once per sighting, with half an hour between sightings, and is fed by what fren notices on your desktop; see docs/privacy.md.
An agent automation runs in isolation: its own workspace, no access to your
files or accounts unless you grant it, and the model reached through fren's
own proxy so no key ever enters the environment. It does not receive what fren
observed about you. It reaches the web only through fren's proxy, which lets it
out to the exact domains its automation declares and nothing else; the reach is
stated on each automation and in docs/privacy.md. The
environment is the vendored runtime host in vendor/nanoclaw. On a Mac nothing
else is needed: npm run runtime:build -- --runner installs the runner, and
fren runs each agent as a process confined by macOS itself (the system
sandbox), using the Bun and Claude Code it finds on the machine. That is
lighter isolation than a container. With Docker Desktop
running, npm run runtime:build -- --image builds the agent image and fren
uses a container instead; FREN_RUNTIME_TIER=process|container|auto picks.
Without either (or with FREN_RUNTIME=mock) a mock environment stands in, so
the whole loop can be tried with nothing installed.
fren used to be able to run a shell script it had drafted, on a schedule, behind three gates: you read and approved the exact text, it ran by hand once, and you turned the schedule on. Every one of those gates was a button in a second window, and that window is gone. A script nobody can review, stop or delete is not something fren should be running, so the whole lane went with it: fren does not run host scripts, scheduled or otherwise. Work that has to do something is an agent automation, above, which runs sandboxed and asks before it reaches anything it was not given.
Routines: tell fren when. Say "every weekday at nine, tell me what I did yesterday" and it sets one up. At that time fren asks itself the question, works it out from what it observed, and reads the answer back.
A routine asks a question — it does not run commands. That is a deliberate limit: scheduling a generated script is a much larger decision than scheduling a question, and it is not one this quietly makes for you. A missed routine expires rather than arriving hours late, and none of them fire while fren is paused.
There is one window: the orb and its chat. Right-click the orb to open the chat (Escape, or another right-click, closes it). The header has three controls and no more: a red light that quits fren (it asks first), a settings button that opens the models pane — which model answers, which voice speaks, which ear listens — and the watching switch. Everything else is done by asking: "what are you running?", "pause the stretch one", "don't read this site", "change your colour to moss", "stop interrupting me", "what do you know about me?", "what patterns have you noticed?", "forget this conversation". fren does it, says what it did, and keeps the rest to itself.
To see everything fren holds about you, ask "what do you know about
me?". It lists what it has kept, and offers Open my notes folder: SOUL.md,
USER.md, MEMORY.md and the daily logs are plain files there, exactly as fren
reads them, to open and edit in your own editor. Nothing is summarised on the
way out — a companion whose notes about you cannot be inspected is not a
companion.
On the first launch fren wakes by itself to introduce itself, and says so: its light is on, that means it is watching, and the menu on a right-click stops that. Capture is real for that conversation. Lighting up without watching would make the light a lie, which is worse than starting awake and saying so.
Every launch after that starts asleep — dark, eyes closed, capturing nothing. Click it to wake it up; that one click lights it, starts observing, and opens the panel. To stop, click "pause watching". The light goes out, the eyes close, and observation stops — no sampling, no screenshots, no summarizing. This is a single boolean in the main process, and every face change in the renderer is funnelled through a check on it: fren cannot look awake while capture is off, and capture cannot run while it looks asleep.
These are designed to exist later. They are not built now, and the code does not pretend otherwise:
- always-on listening (voice is push-to-talk only, by design)
- fren acting on its own initiative. It runs the automations and routines you asked for, and it does not run scripts on your machine at all.
- a browser extension
- WhatsApp or mobile clients
- multi-device sync
- plugins or a marketplace
- remote control of any kind
- complex agent frameworks
- a vector database (SQLite is enough at this scale)
If you find code for any of the above in this repo, it is a bug in the docs or in the code — file an issue.