This repository contains the prompt datasets, power collection tools, and analysis scripts for our EMNLP'26 main conference paper. The study examines how two prompt properties, cognitive load and phrasing pattern, shape the energy behavior of on-device LLM inference. We profile five lightweight LLMs (Gemma-2-2B, LLaMA-3.2-1B, Qwen2.5-0.5B, Qwen2.5-1.5B, SmolLM2-360M) on Pixel 7 and Pixel 8 Pro phones, separating prefill and decode energy at the phase level.
The on-device runtime used to run the models and log latency traces is our fork of MLC-LLM with LM-Meter instrumentation, available at LM-Meter. This repository provides the prompts, host-side power collection tools, and analysis pipeline.
docs/ Static paper project page for GitHub Pages
datasets/ Prompt datasets used in the experiments
cognitive_load/ Our constructed dataset: SVAMP, BoolQ, and AI2-ARC base prompts,
each with intrinsic, extraneous, and germane load variants
phrasing_pattern/ Phrasing variants from the CLEF 2025 ELOQUENT robustness dataset,
covering seven sub-properties (aggressive, conversational, CoT,
format, persona, polite, technical)
power_related/ Host-side measurement tools
power_collection.py Samples battery current and voltage over ADB while the phone
runs inference, and saves timestamped power readings
cpu_gpu_freq_monitor.sh Optional live monitor for CPU and GPU frequencies
data_analysis/ Analysis notebooks, staged to mirror the paper pipeline
stage1_power_latency_data_align.ipynb Aligns power traces with inference latency
logs into per-request energy records
stage2_evaluate_single_run_*.ipynb Scores response quality for one run:
ai2_arc, boolq, svamp (accuracy against
ground truth), phrasing_pattern (six GEval
rubrics, no gold answers)
stage3_merge_3_single_runs.ipynb Merges the three repeated runs per model
stage3_*_data_analysis.ipynb Produces the RQ1, RQ2, and RQ3 figures
stage3_*_table_organize.ipynb Renders the LaTeX summary tables
ground_truth_info/ Ground truth tables for the accuracy tasks
- Stage 1 aligns latency events with power traces and computes phase-level energy.
- Stage 2 evaluates response quality or task accuracy for each inference run.
- Stage 3 merges the three repeated runs for each model and prompt setting.
- The analysis and table notebooks generate the RQ1, RQ2, and RQ3 figures and LaTeX tables.
Start Jupyter inside data_analysis/ so the relative paths resolve, update the exemplar input paths for your setup, and run the notebooks in stage order. The stage 2 notebooks for AI2-ARC, BoolQ, and phrasing patterns use Gemini-2.5-Pro wrapped in the DeepEval model interface, so set GENAI_API_KEY before running them; the SVAMP scorer is purely local.
The on-device inference runs use fixed sampling parameters across all experiments: temperature=0.6 and top_p=0.8. The Gemini judge uses a separate low-temperature setting (temperature=0.1) for stable scoring.
Main dependencies: pandas, numpy, matplotlib, seaborn, deepeval, google-generativeai.
power_collection.py reads current_now and voltage_now from the battery driver through ADB at a 10 ms interval and computes instantaneous power. Start it on the host right before launching inference on the phone, and stop it with Ctrl+C to write the JSON trace. Phase-level energy is later obtained in stage 1 by integrating these samples over the prefill and decode windows recorded by the runtime.
@misc{hu2026promptvariationsaffectenergy,
title = {How Do Prompt Variations Affect Energy Consumption in On-Device LLMs?},
author = {Wei Hu and Xiaolong Tu and Dawei Chen and Yitao Chen and Kyungtae Han and Haoxin Wang},
year = {2026},
eprint = {2609.01798},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2609.01798}
}