[userspace LL] scheduling: LLEXT, DP & multicore by lyakh · Pull Request #10945 · thesofproject/sof

lyakh · 2026-06-18T16:08:25Z

This includes #10558 and my patches on top to enable LLEXT and multicore. Current status: passes simple tests with nocodec with both core 0 and core 1 streaming. 2 streams simultaneously run into a problem when the first of them terminates. WiP.

Update: now that #10558 has mostly been merged, this PR still carries remaining fixes for multicore, LLEXT, cold sections. At its peak this PR contained more than a 100 commits, as of today Aug 28th only 12 patches remain

Update: as of today Sep 2nd only 7 patches remain and this is now considered to be the final step (short of making userspace LL on by default on ACE3+)

lyakh · 2026-08-20T13:22:21Z

After multiple PRs got merged we're down from 108 commits to 32

lyakh · 2026-08-27T13:40:34Z

let me mark it "ready" to have some on-device testing

Copilot

Pull request overview

This PR is a WiP/PoC that extends the SOF Zephyr userspace low-latency (LL) work to support LLEXT and multi-core operation, including new syscalls and userspace IPC/LL thread infrastructure per core.

Changes:

Adds a userspace-accessible vregion syscall interface and vregion-to-mem-domain mapping support.
Introduces per-core userspace IPC threads and secondary-core initialization plumbing for userspace LL.
Refactors DP/LL scheduler interactions for multicore and userspace execution contexts, plus library manager changes to support userspace/module-loading flow.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
zephyr/syscall/vregion.c	Adds Zephyr syscall verifiers/marshalling for vregion operations.
zephyr/lib/vregion.c	Implements z_impl_* syscall backends, vregion mapping into a mem domain, and userspace verification.
zephyr/lib/userspace_helper.c	Adjusts mailbox partition mapping logic for IPC4 userspace access.
zephyr/Kconfig	Adds/selects userspace vregion interface Kconfig option.
zephyr/CMakeLists.txt	Registers new syscall headers/sources for vregion and other headers needed by syscalls.
uuid-registry.txt	Adds UUID entry used for secondary-core init task.
src/schedule/zephyr_ll.c	Updates LL tick path and scheduler data source for userspace LL.
src/schedule/zephyr_dp_schedule.h	Removes internal prototypes no longer intended for this header.
src/schedule/zephyr_dp_schedule.c	Converts scheduler_dp_ll_tick to a syscall-capable z_impl/z_vrfy API; uses userspace scheduler data.
src/schedule/zephyr_dp_schedule_application.c	Updates DP IPC paths and adds syscall verifier for DP internal free.
src/schedule/zephyr_domain.c	Adds helper to retrieve the LL domain thread for current core.
src/library_manager/llext_manager.c	Changes userspace LL domain add/remove behavior for LLEXT mappings.
src/library_manager/lib_manager.c	Refactors module creation to a reusable helper and adds syscall for module free; adjusts userspace/module interactions.
src/ipc/ipc4/helper.c	Adds helper to pre-load modules in kernel context for userspace LL flows.
src/ipc/ipc4/handler-user.c	Routes more IPC4 messages to per-core userspace handling and adjusts init path for LL vs DP.
src/ipc/ipc-common.c	Implements per-core userspace IPC threads, per-core stacks/events/threads, and secondary-core userspace init.
src/init/init.c	Hooks secondary-core init to start userspace IPC/LL infrastructure when enabled.
src/include/sof/schedule/schedule.h	Adds scheduler_list_get_data() and scheduler_get_user_data() helper for userspace schedulers.
src/include/sof/schedule/ll_schedule_domain.h	Exposes zephyr_ll_domain_thread().
src/include/sof/schedule/dp_schedule.h	Exposes scheduler_dp_ll_tick and scheduler_dp_internal_free as syscalls for Zephyr full-app builds.
src/include/sof/lib/vregion.h	Exposes vregion syscalls (create_map/get/put/alloc/free/set_interim) and verification helper.
src/include/sof/lib_manager.h	Exposes lib_manager_mod_create_priv() and lib_manager_free_module syscall/z_impl mapping.
src/include/sof/ipc/common.h	Extends ipc_user struct for per-core threads/events/audio threads; updates ipc_user_forward_cmd signature; declares secondary init helpers.
src/include/ipc4/handler.h	Declares ipc4_user_module_load().
src/audio/pipeline/pipeline-graph.c	Adds LL domain include needed by userspace LL changes.
src/audio/mux/mux_ipc4.c	Tightens IPC4 mux blob size validation.
src/audio/module_adapter/module_adapter.c	Switches DP module heap to vregion_create_map() under userspace constraints.
src/audio/buffers/comp_buffer.c	Fixes alloc context free path when vregion-backed allocations are used.
app/overlays/ptl/ll_userspace_overlay.conf	Updates overlay config for the evolving userspace LL setup.

Suppressed comments (2)

zephyr/lib/vregion.c:279

vregion_unmap(vr) runs after vpage_free(vr->base). This leaves a window where userspace can still access the partition while the backing memory is already freed/reusable, and unmapping may also rely on the region still being valid. Unmap partitions before freeing the backing pages.

	LOG_DBG(" lifetime used %zu free count %d", vr->lifetime.used, vr->lifetime.free_count);
	vpage_free(vr->base);

	vregion_unmap(vr);
	rfree(vr);

src/ipc/ipc-common.c:342

ipc_user_forward_cmd() uses 'core' to index init_needed[] and event[] with no bounds checking. core ultimately comes from IPC fields and can be out of range, leading to out-of-bounds access and memory corruption.

int ipc_user_forward_cmd(uint32_t primary, uint32_t extension, unsigned int core)
{
	struct ipc *ipc = ipc_get();
	struct ipc_user *pdata = ipc->ipc_user_pdata;
	k_spinlock_key_t key;

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+	struct k_mem_domain *domain = zephyr_ll_mem_domain();
+	struct k_mem_partition part = {
+		.start = *vreg_start,
+		.size = *vreg_size,
+		.attr = K_MEM_PARTITION_P_RW_U_RW | XTENSA_MMU_CACHED_WB,
+	};
+	int ret = k_mem_domain_add_partition(domain, &part);
+
+	if (ret < 0) {
+		vregion_put(vr);
+		return NULL;
+	}
+
+	part.start = (uintptr_t)sys_cache_uncached_ptr_get((void *)part.start);
+	part.attr = K_MEM_PARTITION_P_RW_U_RW;
+
+	ret = k_mem_domain_add_partition(domain, &part);
+	if (ret < 0) {
+		vregion_put(vr);
+		return NULL;
+	}
+
+	vr->domain = domain;


+
+	size_t vr_size = 0;
+	uintptr_t vr_start;
+
+	vregion_mem_info(vr, &vr_size, &vr_start);


+struct vregion *z_vrfy_vregion_create_map(uintptr_t *vreg_start, size_t *vreg_size)
+{
+	K_OOPS(K_SYSCALL_MEMORY_WRITE(vreg_start, sizeof(*vreg_start)));
+	K_OOPS(K_SYSCALL_MEMORY_WRITE(vreg_size, sizeof(*vreg_size)));
+	return z_impl_vregion_create_map(vreg_start, vreg_size);
+}


intel-sofci · 2026-09-01T13:05:40Z

PR 10945: test results

Run date: 2026-09-01 12:50 UTC

Tested commit: 34ad06a7b887f838b9b15df1590facb835519e87

Extract a privileged LLEXT-related part from lib_manager_module_create() into a separate function to be called from kernel context. At the same time lib_manager_mod_free_priv() already executes privileged operations; to make it callable in userspace, convert lib_manager_free_module() to a system call. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

If LL runs in userspace, it needs access to loaded LLEXT modules, running in DP more too. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

Make scheduling LL thread and synchronisation objects per-core and forward IPCs and scheduling events accordingly. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

Checking pointer for non-NULL after dereferencing it makes no sense. Swap the order. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

In case of userspace LL scheduling the (also userspace) IPC thread needs access rights to DP assets like the thread itself and its stack and synchronisation primitives. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

When running in syscall context on behalf of a userspace thread dynamically mapped memory doesn't automatically become accessible. To make it accessible it has to be added to the thread memory domain. This is a problem for loadable modules with executable cold sections. To be able to execute them they have to be mapped to threads with the executable bit set. While for linking that memory has to be mapped writable. To solve the problem we perform linking from the kernel IPC context before forwarding to the userspace IPC thread. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

Re-enable LLEXT, DRAM execution and data, the DP scheduler and boot time testing with userspace LL with a small restriction for the latter - userspace allocation failure tests aren't currently working. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

Copilot

🟡 Changes recommended

Confirmed issues include unsafe userspace LL domain mapping for DP modules and uninitialized/per-core IPC state leading to potential deadlocks or invalid memory access.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/ipc/ipc-common.c:619

ipc_user is allocated with sof_heap_alloc() but never zero-initialized; init_needed[] and the per-core thread/event pointers start as uninitialized heap garbage. This can cause ipc_user_forward_cmd() to deadlock (spurious init_needed[core]) or dereference invalid per-core pointers.

	struct ipc *ipc = ipc_get();
	struct ipc_user *ipc_user = sof_heap_alloc(sof_sys_user_heap_get(),
						   SOF_MEM_FLAG_USER | SOF_MEM_FLAG_COHERENT,
						   sizeof(*ipc_user), 0);
	int ret;

Files reviewed: 14/14 changed files
Comments generated: 2
Review effort level: Lite

@@ -431,13 +434,22 @@ int ipc_user_forward_cmd(uint32_t primary, uint32_t extension)
 	pdata->ipc_msg_ext = extension;
 	pdata->ipc = ipc;

+	/*
+	 * Forwarding the first IPC to this core, wait for its userspace IPC
+	 * thread to start
+	 */
+	if (pdata->init_needed[core]) {
+		pdata->init_needed[core] = false;
+		k_sem_take(pdata->sem, K_FOREVER);
+	}


+	ret = llext_manager_add_mod_domain(mctx, zephyr_ll_mem_domain());
+	if (ret < 0) {
+		tr_err(&lib_manager_tr, "failed to add domain: %d", ret);
+		goto e_data;
 	}


kv2019i

Looks good, some minor comments and usual nagging about z_vrfy code ...

kv2019i · 2026-09-02T10:12:07Z

+
+static int z_vrfy_lib_manager_free_module(const uint32_t component_id)
+{
+	return z_impl_lib_manager_free_module(component_id);


@lyakh We should have ability to limit which user-space threads can free modules. E.g. especially with non-LL-user builds, Dp modules should not be able to unload arbitrary components. I know we can mitigate this currently by enforcing signing on all loaded code, but we should at least have a TODO here, but ideally this would be only allowed from the LL thread handling user IPCs.

kv2019i · 2026-09-02T10:20:04Z

 #define IPC_TASK_POWERDOWN      BIT(3)

+#ifdef CONFIG_CORE_COUNT
+#define CORE_COUNT CONFIG_CORE_COUNT


Isn't CORE_COUNT a bit generic (and short) definition in a public header file?

Most places in SOF just use CONFIG_CORE_COUNT (defined in sof/src/platform/Kconfig), so this should be available for all builds. Maybe "ifndef CONFIG_CORE_COUNT" and set it 1 for the failing builds (library, cmocka, ztests?)

kv2019i · 2026-09-02T10:21:56Z

 	if (!ret) {
 		/* Wait for completion */
-		ret = k_sem_take(&dp_sync[cpu_get_id()], DP_THREAD_IPC_TIMEOUT);
+		ret = k_sem_take(&dp_sync[core], DP_THREAD_IPC_TIMEOUT);


These changes in scheduler_dp_thread_ipc() seem unrelated to the commit....

kv2019i · 2026-09-02T10:34:26Z

+	 * That one calls lib_manager_mod_create_priv(), then
+	 * lib_manager_allocate_module() and eventually
+	 * llext_manager_allocate_module() for LLEXT modules.
+	 */


This comment is a bit hard to follow. What do you mean "IPC handling thead would call"? Would or will? As far as I understand, the user-space IPC thread (maybe mention ipc_user_thread_dispatch() explicitly) will call comp_new_ipc4_user(), and do the steps you describe above. This function is run before that (in kernel space). Right?

kv2019i · 2026-09-02T10:35:13Z

    userspace/ksem.c
  )
-  if(CONFIG_USERSPACE AND CONFIG_SOF_USERSPACE_INTERFACE_ALLOC)
+  if(CONFIG_USERSPACE AND CONFIG_SOF_USERSPACE_INTERFACE_ALLOC AND NOT CONFIG_SOF_USERSPACE_LL)


Ack, filed #11147 to track this,

lyakh force-pushed the llext-ull branch from 0c4e579 to 0f8ce76 Compare June 22, 2026 10:53

lyakh mentioned this pull request Jun 22, 2026

userspace: preparatory patches for multicore LL scheduler #10949

Merged

lyakh mentioned this pull request Jul 10, 2026

subset of #10945 #10984

Merged

lyakh force-pushed the llext-ull branch 2 times, most recently from a462fcf to 9f6770d Compare July 14, 2026 09:58

This was referenced Jul 14, 2026

ipc: add support for userspace IPC handling #10994

Merged

userspace: a subset of #10945 that can be applied already now #11017

Merged

lyakh force-pushed the llext-ull branch from 4408350 to c20c623 Compare August 5, 2026 07:27

lyakh mentioned this pull request Aug 5, 2026

[Userspace LL] scattered commits from #10945 without dependencies #11065

Merged

This was referenced Aug 12, 2026

[Userspace LL] app: overlays: ptl: ll_userspace: disable DP scheduler #11085

Merged

[DRAFT/TESTING] userspace LL/audio test PR (latest ver: V40) #10558

Open

lyakh force-pushed the llext-ull branch from c20c623 to 4aebe2b Compare August 20, 2026 13:21

lyakh force-pushed the llext-ull branch from 4aebe2b to c2041bb Compare August 20, 2026 13:34

This was referenced Aug 20, 2026

[userspace LL] vregion related syscalls #11108

Merged

schedule: add a function for finding userspace scherulers #11109

Merged

scattered fixes and cosmetic improvements #11110

Merged

[userspace LL] LLEXT part of #10945 #11112

Merged

lyakh force-pushed the llext-ull branch from c2041bb to 389f5a5 Compare August 27, 2026 13:39

lyakh added the DNM Do Not Merge tag label Aug 27, 2026

lyakh marked this pull request as ready for review August 27, 2026 13:40

Copilot AI lite review requested due to automatic review settings August 27, 2026 13:40

lyakh requested review from LaurentiuM1234, abonislawski, dbaluta, iuliana-prodan, kv2019i, marcinszkudlinski and pblaszko as code owners August 27, 2026 13:40

lyakh requested review from lbetlej, lgirdwood, mmaka1 and plbossart as code owners August 27, 2026 13:40

lyakh changed the title ~~[DNM][WiP][PoC] userspace LL scheduling: LLEXT & multicore~~ [DNM][Do Not Review][WiP][PoC] userspace LL scheduling: LLEXT & multicore Aug 27, 2026

Copilot started reviewing on behalf of lyakh August 27, 2026 13:42 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

lyakh force-pushed the llext-ull branch 5 times, most recently from e62b4cd to 69f09d7 Compare August 28, 2026 08:11

This was referenced Aug 28, 2026

[userspace LL] IPC part of #10945 #11137

Merged

[userspace LL] Scheduling part of #10945 #11138

Merged

[userspace LL] library manager commits from #10945 #11139

Draft

lyakh force-pushed the llext-ull branch from 69f09d7 to 524d832 Compare August 28, 2026 15:33

lyakh added 7 commits September 2, 2026 10:00

llext: with userspace let LL thread access DP modules too

95693d1

If LL runs in userspace, it needs access to loaded LLEXT modules, running in DP more too. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

schedule: ll: enable multicore userspace

17cab14

Make scheduling LL thread and synchronisation objects per-core and forward IPCs and scheduling events accordingly. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

schedule: dp: application: fix NULL check order

eedc4a1

Checking pointer for non-NULL after dereferencing it makes no sense. Swap the order. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

lyakh force-pushed the llext-ull branch from 34ad06a to 4f17b68 Compare September 2, 2026 08:04

lyakh changed the title ~~[DNM][Do Not Review][WiP][PoC] userspace LL scheduling: LLEXT & multicore~~ [userspace LL] scheduling: LLEXT, DP & multicore Sep 2, 2026

lyakh requested a lite review from Copilot September 2, 2026 08:26

Copilot started reviewing on behalf of lyakh September 2, 2026 08:28 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

kv2019i reviewed Sep 2, 2026

View reviewed changes

Navigation Menu

Uh oh!

[userspace LL] scheduling: LLEXT, DP & multicore - #10945

[userspace LL] scheduling: LLEXT, DP & multicore#10945
lyakh wants to merge 7 commits into
thesofproject:mainfrom
lyakh:llext-ull

lyakh commented Jun 18, 2026 •

edited

Loading

Uh oh!

lyakh commented Aug 20, 2026

Uh oh!

lyakh commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

intel-sofci commented Sep 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

kv2019i left a comment

Uh oh!

kv2019i Sep 2, 2026

Uh oh!

kv2019i Sep 2, 2026

Uh oh!

kv2019i Sep 2, 2026

Uh oh!

kv2019i Sep 2, 2026

Uh oh!

kv2019i Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

lyakh commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lyakh commented Aug 20, 2026

Uh oh!

lyakh commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

intel-sofci commented Sep 1, 2026

PR 10945: test results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

kv2019i left a comment

Choose a reason for hiding this comment

Uh oh!

kv2019i Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

kv2019i Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

kv2019i Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

kv2019i Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

kv2019i Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lyakh commented Jun 18, 2026 •

edited

Loading