Zikang Leng*, Megha Thukral*, Yaqi Liu*, Hrudhai Rajasekhar, Shruthi K. Hiremath, Jiaman He, Thomas Plötz
Georgia Institute of Technology · *equal contribution
AAAI 2026
A generated day from the paper: persona "Aaron" in home 0 on Monday morning. The agent follows its LLM-written routine, motion sensors light up when it is in range, and the resulting sensor event stream is what activity-recognition models are trained on.
Smart-home activity recognition is starved for labelled data. Every home has a different layout, different sensors and a different resident. AgentSense generates that data instead of collecting it:
- An LLM writes diverse personas and their daily routines.
- Embodied agents live out those routines in an extended VirtualHome simulator instrumented with ambient sensors.
- The result is labelled, privacy-preserving smart-home sensor data at scale.
| 18 LLM-generated personas |
22 simulated homes |
250 simulated days |
5 real smart-home datasets improved |
- Personas. An LLM generates residents with distinct ages, occupations, health conditions and habits.
- Schedules and routines. Each persona gets a weekly schedule, which is expanded into step-by-step daily routines.
- Grounding. Every step is mapped to the simulator's action set and to objects that actually exist in the chosen home, using embedding search with similarity thresholds.
- Simulation. Agents execute the routines in an extended VirtualHome. Every room has motion sensors, and door and device sensors are derived from the environment graph.
- Pretraining. The virtual sensor data pretrains activity-recognition models, which are then fine-tuned on real homes.
Agents carry out their generated routines in the extended VirtualHome. The ambient sensors read exactly these movements.
Persona Sam's Monday morning, 06:50: walking to the counter and making toast (home 18).
 Watching TV on the sofa (home 29) |
 Opening the fridge (home 18) |
 At the stove (home 0) |
 A whole simulated home, ceilings removed (home 18) |
More renders and clips are on the project page. TV screens show a frame from Big Buck Bunny (© Blender Foundation, CC BY 3.0).
Pretraining on AgentSense data and then fine-tuning on real data improves macro F1 on all five real datasets (TDOST-Basic + Bi-LSTM, mean of 3 folds):
| Aruba | Milan | Cairo | Kyoto7 | Orange4Home | |
|---|---|---|---|---|---|
| Real only | 63.98 | 70.81 | 51.51 | 52.48 | 21.56 |
| Real + AgentSense | 72.20 | 74.44 | 62.47 | 56.07 | 41.83 |
The gains are largest when real data is scarce.
- The paper also reports accuracy, weighted F1, the temporal model, and the low-data and diversity ablations.
- The project page has interactive visualisations.
| Directory | Description |
|---|---|
AgentSense_pipeline/ |
LLM-based generation of personas, schedules and activity scripts (steps 1–9 and 11). |
VirtualHome_API/ |
Modified VirtualHome simulator for script execution and sensor data collection (step 10). |
- LLM pipeline
- Go to
AgentSense_pipeline/and run the notebooks in order (step 1 to step 9). - Detailed guide
- Go to
- Simulator
- Go to
VirtualHome_API/. - Download the Unity simulator from Google Drive and extract it into
VirtualHome_API/exe/. - Set up the environment and run the simulation (step 10).
- Detailed guide
- Go to
- [Jan 2026] Initial release of the AgentSense pipeline and simulation environment.
- [Nov 2025] AgentSense is accepted to AAAI 2026.
- [Jun 2025] Paper available on arXiv.
If you use AgentSense in your research, please cite:
@inproceedings{leng2026agentsense,
title = {AgentSense: Virtual Sensor Data Generation Using LLM Agents in Simulated Home Environments},
author = {Leng, Zikang and Thukral, Megha and Liu, Yaqi and Rajasekhar, Hrudhai and Hiremath, Shruthi K. and He, Jiaman and Pl{\"o}tz, Thomas},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = {40},
number = {3},
pages = {1891--1899},
year = {2026}
}AgentSense builds on VirtualHome. We thank its authors for releasing the simulator.
Released under the MIT License.