Experimental gesture recognition for spellcasting in virtual reality.
Spellcaster VR explores whether hand-motion sequences can be represented as numerical features and classified reliably enough to drive gesture-based spell casting in a VR environment.
The current prototype focuses on the recognition pipeline: collecting gesture samples, extracting features, classifying known spell gestures, and rejecting movements that do not correspond to a supported spell.
The prototype currently recognises four spell gestures:
- Fireball
- Lightning
- Shield
- Telekinesis
It also includes an invalid / unknown class so that arbitrary motion is not automatically forced into one of the known spells.
This makes the problem closer to open-set recognition than ordinary multiclass classification.
At a high level:
VR hand-motion sample
|
v
gesture recording
|
v
feature extraction
|
v
fixed-length feature vector
|
v
gesture classifier
|
+----> known spell
|
+----> invalid / unknown
Each recorded gesture is transformed into a fixed-length representation containing 64 features.
The classifier then predicts one of the supported spells or rejects the gesture as invalid.
The initial experiment used:
- 25 gesture samples
- 64 features per sample
- 4 recognised spell classes
- 1 invalid / unknown class
- 5-fold cross-validation
The prototype evaluation achieved:
- 100% overall accuracy
- 100% valid-spell accuracy
- 100% invalid-gesture recall
- 1.00 macro F1
These results should be interpreted as an early proof of concept rather than evidence of production-level generalisation.
The dataset is intentionally small at this stage, and future evaluation needs substantially more gestures, users, variation and noisy real-world motion.
A standard classifier will normally choose the closest known class even when the user performs a completely unrelated movement.
That behaviour is undesirable for gesture-controlled interaction.
For example:
random hand movement
↓
ordinary classifier
↓
"fireball"
Instead, Spellcaster VR is designed around:
gesture
|
+---- sufficiently similar to known gesture --> spell
|
+---- outside accepted region -------------> invalid
This reduces unintended spell activations and makes the recognition system better suited to interactive VR use.
The project is being used to explore:
- gesture representation
- motion-feature engineering
- multiclass classification
- open-set recognition
- confidence and rejection thresholds
- robustness to variations in gesture execution
- integration of ML predictions with interactive VR systems
Spellcaster VR is an active experimental side project.
The recognition pipeline currently works on the prototype dataset. The next stages of the project focus on making the system substantially harder to fool and testing whether performance holds when gestures vary across repetitions and users.
Potential extensions include:
- larger gesture datasets
- multiple users
- noisy and incomplete gestures
- temporal sequence models
- stronger unknown-gesture rejection
- real-time inference
- direct VR spell activation
- additional spell classes
A high validation score on a small dataset is not sufficient to establish a reliable gesture-recognition system.
The interesting question is whether the model remains accurate when:
- the same spell is performed differently
- gestures are faster or slower
- starting positions change
- motion is incomplete
- users perform movements that were never part of training
- entirely new users interact with the system
Those cases are the focus of the next stages of the project.
Aishwarya Singh