Towards Transfer-Efficient Multi-modal Sequential Recommendation with State Space Duality

1. Introduction

Towards Transfer-Efficient Multi-modal Sequential Recommendation with State Space Duality
Hao Fan, Qingyang Liu, Hongjiu Liu, Yanrong Hu, Kai Fang
Accepted by ACM Multimedia 2026 (ACM MM 2026)
ArXiv Preprint: https://arxiv.org/abs/2506.02916

We propose MMM4Rec(Multi-Modal Mamba for Sequential Recommendation), a transfer-efficient multimodal sequential recommendation model. By establishing intrinsic algebraic constraints that align with sequential recommendation principles, MMM4Rec eliminates the need for complex optimization objectives required by other multimodal sequential recommendation models. During both pre-training and fine-tuning, MMM4Rec is optimized solely using the standard cross-entropy loss. This unified and simplified optimization objective enables MMM4Rec to achieve rapid convergence when transferred to new domains.

In the following, we will guide you how to use this repository step by step. 🤗

2. Preparation

2.1 Environment Requirements

The following are the main runtime environment dependencies for running the repository：

• linux (We use Ubuntu 22.0.4)
• cuda 11.8
• python 3.10.15
• pytorch 2.3.1
• numpy 1.26.4
• pandas 2.2.3
• jsonlines 4.0.0
• pytorch-lightning 2.4.0
• lightning 2.4.0
• transformers 4.47.0
• sentencepiece 0.2.0
• tabulate 0.9.0
• tensorboardX 2.6.2
• tensorboard 2.19.0
• casual-conv1d 1.4.0
• mamba-ssm 2.2.2

You can also view detailed environment information in file environment.yaml.

2.2 DataSets

This study focuses on the 🛒 Amazon Review 2018 dataset.

You can download our preprocessed dataset directly via the link https://figshare.com/s/f7603ea556c23c2aef88 and extract the subfolders (e.g., Scientific) from the downloaded archive into the 📁 dataset/📁 amazon-2018/📁 processed folder.

For detailed dataset preprocessing steps and descriptions, please refer to 📁 dataset/Ⓜ️ READEM.md.

2.3 Project Structure

In this section, you can learn about our project structure.

You can click on the directory below to expand and view the project structure:

📁 MMM4Rec

• 📁 baseline | (The baseline model in the paper)
• 📁 BSARec
• 📜 config.yaml
• 🐍 run.py
• 📁 ...
• 📁 configs | (Configuration file for MMM4Rec)
• 📁 finetune
• 📜 config_mmm4rec_scientific.yaml
• 📜 ...
• 📜 config_mmm4rec_FHCKM.yaml
• 📁 data | (Dataset class pytorch implementation)
• 🐍 amazon_dataset.py
• 📁 dataset | (Store dataset files)
• 📁 amazon-2018
• 📁 preprocess
• 📁 processed
• 📁 raw
• 📁 misc | (Store readme related images)
• 📁 model | (Python implementation of the model)
• 📁 encoder | (Kernel Implementation)
• 🐍 ssd_kernel.py
• 🐍 ...
• 🐍 mmm4rec.py
• 🐍 ...
• 📁 pre_weights | (Pre-trained weight files)
• 📀 pretrained_weight.ckpt
• 📁 reference_log | (Reference log file)
• 📁 scientific
• 📁 with_id | (With ID Feature)
• 📁 without_id | (Without ID Feature)
• 📁 ...
• 📁 saved | (Store the training logs and weights)
• 📁 MMM4Rec
• 📁 {time}
• 📄 output.log
• 📀 best_epoch.ckpt
• 📁 ...
• 📁 ...
• 📁 script | (Model fine-tuning script)
• 📁 finetune
• 🚅 scientific.sh
• 🚅 ...
• 📁 trainer | (Python Implementation of Trainer)
• 🐍 pretrain_trainer.py
• 🐍 trainer.py
• 🐍 utils.py
• 📜 environment.yaml
• 🐍 callback.py
• 🐍 main.py
• 🐍 pretrain.py
• 🐍 test.py

3. Run

Ok, congratulations 🎇, you have finished all the preparation 👍, let's start training the model! 😄

This section will introduce the training methods of the model.

3.1 Pre-train

After preparing our provided pretraining dataset, you can directly pretrain MMM4Rec using the following approach:

python pretrained.py

🚀 3.2 Fine-tune

As described in our paper, our work primarily focuses on model transfer efficiency. To help you quickly verify our results, we have provided pre-trained model weights in the 📁 pre_weights folder, along with a quick-start shell script in 📁 script .

To fine-tune MMM4Rec in the Scientific domain, simply run the following command:

cd ./script/finetune
/bin/bash scientific.sh
cd ../../

You can also directly examine the training logs in the 📁 reference_log folder to verify our work's effectiveness.

For example, check the output.log file to see MMM4Rec's fine-tuning logs in the Scientific domain.

Alternatively, you can directly download our fine-tuned model weights and complete log files for downstream datasets via the link https://figshare.com/s/f7603ea556c23c2aef88.

4. Results

This section presents the main experimental results of this work. We evaluate MMM4Rec on five downstream domains and report performance comparisons under different input types, comparisons without ID features, and the efficiency of model transfer.

4.1 Performance Comparison with ID Features

This subsection compares MMM4Rec with the baselines using different input types (ID, T+ID, and T+V+ID). Bold and underlined values denote the best and second-best results, respectively.

Input Type & Model →		ID				T+ID			T+V+ID					Improv. w/ ID
		SASRec	Mamba4Rec	TiM4Rec	BSARec	FDSA	S³-Rec	UniSRec	MISSRec	M³Rec	HM4SR	ATHWE	MMM4Rec
Scientific	R@10	0.1080	0.1040	0.1079	0.1102	0.0899	0.0525	0.1235	0.1360	0.1105	0.0937	0.1070	0.1348	-
	R@50	0.2042	0.2030	0.2021	0.2106	0.1732	0.1418	0.2473	0.2431	0.2142	0.1686	0.2072	0.2627	6.23%
	N@10	0.0553	0.0598	0.0605	0.0605	0.0580	0.0275	0.0634	0.0753	0.0616	0.0651	0.0711	0.0724	-
	N@50	0.0760	0.0814	0.0810	0.0824	0.0759	0.0468	0.0904	0.0983	0.0842	0.0814	0.0802	0.1002	1.93%
Pantry	R@10	0.0501	0.0487	0.0504	0.0531	0.0395	0.0444	0.0693	0.0779	0.0495	0.0437	0.0573	0.0984	26.32%
	R@50	0.1322	0.1377	0.1360	0.1408	0.1151	0.1315	0.1827	0.1875	0.1407	0.1156	0.1414	0.2127	13.44%
	N@10	0.0218	0.0223	0.0229	0.0234	0.0209	0.0214	0.0311	0.0365	0.0222	0.0232	0.0314	0.0481	31.78%
	N@50	0.0394	0.0415	0.0411	0.0423	0.0370	0.0400	0.0556	0.0598	0.0418	0.0388	0.0494	0.0729	21.91%
Instruments	R@10	0.1118	0.1113	0.1113	0.1156	0.1070	0.1056	0.1267	0.1300	0.1145	0.1079	0.1193	0.1330	2.31%
	R@50	0.2106	0.2034	0.2071	0.2114	0.1890	0.1927	0.2387	0.2370	0.2114	0.1881	0.2088	0.2525	5.78%
	N@10	0.0612	0.0751	0.0683	0.0649	0.0796	0.0713	0.0748	0.0843	0.0764	0.0807	0.0872	0.0822	-
	N@50	0.0826	0.0950	0.0890	0.0857	0.0972	0.0901	0.0991	0.1071	0.0975	0.0979	0.1066	0.1082	1.03%
Arts	R@10	0.1108	0.1089	0.1096	0.1105	0.1002	0.1003	0.1239	0.1314	0.1098	0.1011	0.1123	0.1307	-
	R@50	0.2030	0.2036	0.2027	0.2102	0.1779	0.1888	0.2347	0.2410	0.2027	0.1745	0.2007	0.2486	3.15%
	N@10	0.0587	0.0628	0.0630	0.0660	0.0714	0.0601	0.0712	0.0767	0.0636	0.0715	0.0769	0.0777	1.04%
	N@50	0.0788	0.0834	0.0832	0.0877	0.0883	0.0793	0.0955	0.1002	0.0838	0.0874	0.0971	0.1034	3.19%
Office	R@10	0.1056	0.1234	0.1227	0.1194	0.1118	0.1030	0.1280	0.1275	0.1217	0.1142	0.1223	0.1337	4.45%
	R@50	0.1627	0.1886	0.1892	0.1878	0.1665	0.1613	0.2016	0.2005	0.1864	0.1664	0.1797	0.2132	5.75%
	N@10	0.0710	0.0874	0.0876	0.0817	0.0868	0.0653	0.0831	0.0856	0.0858	0.0887	0.0947	0.0906	-
	N@50	0.0835	0.1016	0.1021	0.0966	0.0987	0.0780	0.0991	0.1012	0.0999	0.1001	0.1071	0.1080	0.84%

4.2 Performance Comparison without ID Features

This subsection compares MMM4Rec with the baselines using model inputs without ID, where "T" and "V" denote text and visual features. Bold and underlined values denote the best and second-best results, respectively.

Input Type & Model →		T						T+V			Improv. w/o ID
		SASRec	Mamba4Rec	TiM4Rec	ZESRec	UniSRec	VQRec	MMSRec	MISSRec	MMM4Rec
Scientific	R@10	0.0994	0.1118	0.1086	0.0851	0.1188	0.1211	0.1054	0.1278	0.1278	-
	R@50	0.2162	0.2149	0.2127	0.1746	0.2394	0.2369	0.2296	0.2375	0.2549	6.47%
	N@10	0.0561	0.0605	0.0587	0.0475	0.0641	0.0643	0.0548	0.0658	0.0668	1.52%
	N@50	0.0815	0.0829	0.0813	0.0670	0.0903	0.0897	0.0815	0.0893	0.0929	2.88%
Pantry	R@10	0.0585	0.0586	0.0575	0.0454	0.0636	0.0660	0.0666	0.0771	0.0885	14.79%
	R@50	0.1647	0.1521	0.1546	0.1141	0.1658	0.1753	0.1801	0.1833	0.1878	2.45%
	N@10	0.0285	0.0282	0.0287	0.0230	0.0306	0.0293	0.0309	0.0345	0.0431	24.93%
	N@50	0.0523	0.0484	0.0496	0.0378	0.0527	0.0527	0.0554	0.0571	0.0646	13.13%
Instruments	R@10	0.1127	0.1170	0.1150	0.0783	0.1189	0.1222	0.1119	0.1201	0.1293	5.81%
	R@50	0.2104	0.2040	0.2084	0.1387	0.2255	0.2343	0.2219	0.2218	0.2426	3.54%
	N@10	0.0661	0.0769	0.0741	0.0497	0.0680	0.0758	0.0732	0.0771	0.0847	9.86%
	N@50	0.0873	0.0988	0.0940	0.0627	0.0912	0.1002	0.0970	0.0988	0.1092	8.98%
Arts	R@10	0.0977	0.1010	0.1026	0.0664	0.1066	0.1189	0.1147	0.1119	0.1219	2.52%
	R@50	0.1916	0.1939	0.1953	0.1323	0.2049	0.2249	0.2205	0.2100	0.2319	3.11%
	N@10	0.0562	0.0598	0.0595	0.0375	0.0586	0.0703	0.0719	0.0625	0.0739	2.78%
	N@50	0.0766	0.0799	0.0796	0.0518	0.0799	0.0935	0.0950	0.0836	0.0979	3.05%
Office	R@10	0.0929	0.1075	0.1063	0.0641	0.1013	0.1236	0.1175	0.1038	0.1252	1.29%
	R@50	0.1580	0.1654	0.1659	0.1113	0.1702	0.1957	0.1859	0.1701	0.1999	2.15%
	N@10	0.0582	0.0729	0.0708	0.0391	0.0619	0.0814	0.0864	0.0666	0.0859	-
	N@50	0.0723	0.0855	0.0837	0.0493	0.0769	0.0972	0.1013	0.0808	0.1022	0.89%

4.3 Efficiency Comparison

This subsection presents the comprehensive efficiency comparison across the five downstream domains, where MMSRec, MISSRec, and MMM4Rec are compared separately for each dataset. FLOPs are measured per user instance; Fwd., Ret., and Total denote sequence forward encoding, full-corpus candidate retrieval, and their sum, respectively. GPU memory is the peak allocated memory during fine-tuning; Train and Infer. denote the wall-clock time per training and evaluation epoch, while Conv. is the total fine-tuning time to the best early-stopped checkpoint. Lower values are better, and the best result in each comparison is in bold.

Dataset	Model	Param. ↓ (M)	Fwd. FLOPs ↓ (M)	Ret. FLOPs ↓ (M)	Total FLOPs ↓ (M)	GPU Memory ↓ (GB)	Train Time ↓ (s)	Conv. Time ↓ (s)	Infer. Time ↓ (s)
		Scientific	MMSRec	9.78	371.2	4.49	375.7	4.63	2.72	68.0	0.32
MISSRec	7.53		146.8	7.89	154.7	3.29	2.21	168.0	0.44
MMM4Rec	4.24		105.2	7.86	113.1	3.01	2.07	26.9	0.19
Pantry	MMSRec	10.30	382.3	5.02	387.4	4.61	6.43	128.6	0.41
	MISSRec	8.21	160.4	8.82	169.2	3.55	5.97	191.0	0.67
	MMM4Rec	4.51	105.2	8.78	114.0	3.06	5.58	55.8	0.30
Instruments	MMSRec	15.50	453.4	10.20	463.6	5.72	12.25	612.5	0.89
	MISSRec	14.91	185.0	17.94	203.0	4.47	10.55	685.8	1.35
	MMM4Rec	7.10	105.2	17.86	123.1	3.11	9.08	63.6	0.58
Arts	MMSRec	26.84	650.8	21.52	672.3	8.27	33.81	2265.3	1.37
	MISSRec	29.55	224.3	37.84	262.1	5.31	25.15	4174.9	2.64
	MMM4Rec	12.76	105.2	37.68	142.9	3.72	14.92	74.6	1.05
Office	MMSRec	22.40	556.9	17.09	574.0	7.42	33.57	1745.6	2.54
	MISSRec	36.13	234.8	46.78	281.6	5.42	41.06	6282.2	5.12
	MMM4Rec	15.30	105.2	46.58	151.8	3.91	27.93	139.7	2.01

4.4 More Experiments

Due to the 8-page limit of ACM Multimedia, additional experiments are provided in the appendix of the arXiv version: https://arxiv.org/abs/2506.02916.

5. Acknowledgements

Our implementation is built upon Pytorch and Pytorch Lightning - we gratefully acknowledge their excellent work.

For dataset processing, we referenced approaches from MMSRec, UniSRec, and MISSRec. Our trainer implementation draws inspiration from RecBole.

Notably, we mathematically implemented an SSD kernel attention form (🐍 ssd_kernel.py) equivalent to Mamba's approach.

6. References

This work has been accepted by ACM Multimedia 2026 (ACM MM 2026). The official proceedings citation will be updated after publication; for now, if you find this code useful or use the toolkit in your work, please consider citing:

@misc{fan2025mmm4rec,
      title={Towards Transfer-Efficient Multi-modal Sequential Recommendation with State Space Duality}, 
      author={Hao Fan and Qingyang Liu and Hongjiu Liu and Yanrong Hu and Kai Fang},
      year={2025},
      eprint={2506.02916},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2506.02916}, 
}