Description

Summary

We would like to propose a new, highly flexible application loading mode for retro-go. This approach bypasses the rigid limitations of traditional ESP-IDF OTA partition tables, allowing the launcher to dynamically load and execute arbitrary application binaries stored in a dedicated custom Flash partition (apps), managed by a metadata partition (appm).

A proof-of-concept has been implemented in the branch: https://github.com/zxfishhack/retro-go/tree/lite-pix.

Motivation & Background

The native ESP-IDF OTA mechanism requires predefined partition sizes and counts at compile time. This poses significant constraints for a gaming/application ecosystem like retro-go:

1. Users cannot easily install an arbitrary number of third-party apps or homebrews without re-flashing the partition table.
2. Apps are heavily coupled with the specific version of the SDK or retro-go core libraries.

By introducing a slot-based flash management system and leverage the bootloader to read boot targets from RTC memory, we can decouple the apps from the launcher, allowing the platform to act more like a classic operating system that treats applications as independent executable files.

Proposed Architecture & Mechanism

1. Flash Partition Layout:

   • appm (Application Management): A small metadata partition used to track the layout, occupancy, and allocation status of the apps partition.
   • apps (Applications Storage): A large storage area managed in fixed-size blocks called Slots (e.g., 256 KB per slot). An application can dynamically occupy multiple contiguous slots depending on its binary size.

2. Boot & Loading Flow:

   • Instead of switching OTA partitions via NVS, the launcher communicates the target application's physical Flash address to the bootloader using BOOTLOADER_RESERVE_RTC_MEM.
   • The bootloader reads the starting flash base address from the reserved RTC memory and maps it via MMU (ensuring 64KB page alignment), enabling it to boot an application from an arbitrary flash address.

Current Progress (What Works)

• Decoupled Binary Execution: Successfully running arbitrary .bin files, even those compiled with different ESP-IDF versions or without referencing the retro-go library.
• Unified File Browsing: Implemented a custom loading menu utilizing the native file browsing style to list and trigger these apps.
• RTC Memory Communication: Native apps can now communicate using the RTC memory protocol, preventing temporary or standalone applications from corrupting or requiring modifications to the core OTA configurations.

Work In Progress & Open Discussion (WIP)

We are seeking feedback and ideas from the community regarding the following challenges currently being tackled:

1. Storage Management Optimization: The implementation of the appm indexing system is under active development. We need to formalize the directory/metadata structure to efficiently query installed applications.
2. Flash Memory Fragmentation (The Crucial Challenge): Since applications must occupy contiguous slots due to the hardware MMU alignment requirements, repeated installation and uninstallation of apps will inevitably lead to flash fragmentation (where total free space is sufficient, but no single continuous block is large enough). Currently, the fallback is a naive "uninstall all and reinstall" approach. We welcome cleaner compaction or defragmentation algorithm ideas suitable for resource-constrained microcontrollers.
3. Always-On / Default App Handling: Migrating default/always-on background programs to use the same RTC MEM boot mechanism. The goal is to point the main OTA partition exclusively to the launcher. Upon boot, the launcher will check NVS configurations; if a default app is set, it writes the corresponding RTC MEM flag and triggers a soft reset to let the bootloader jump straight into that app.

Anything else?

No response