feat(storage): paginate remote storage listings with a "Load more" button#9834
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
6 tasks
Contributor
There was a problem hiding this comment.
2 issues found across 15 files
Architecture diagram
sequenceDiagram
participant User
participant UI as Storage Inspector (React)
participant State as Jotai State (Atom/Reducer)
participant Backend as Runtime Commands
participant Storage as Storage Backend (Obstore/Fsspec)
participant External as Object Store (S3/GCS/...)
Note over UI,External: Paginated storage listing flow (NEW)
UI->>State: useStorageEntries(namespace, prefix)
State-->>UI: cached entries & pageMetadata (if any)
alt Cache miss (first fetch)
UI->>Backend: ListStorageEntries (pageToken: null)
Backend->>Storage: list_entries(limit=150, page_token=None)
Storage->>External: list entries (e.g., list_with_delimiter)
External-->>Storage: raw entries (may be provider-truncated)
alt Obstore: result length >= 1000
Storage->>Storage: set may_have_more=true (provider limit)
end
Storage-->>Backend: StorageListResult(entries, next_page_token, may_have_more)
Backend-->>UI: StorageEntriesNotification(entries, next_page_token, may_have_more)
UI->>State: setEntries({entries, nextPageToken, mayHaveMore, append: false})
State-->>UI: entriesByPath + pageMetadataByPath updated
end
UI->>UI: Render entries + "Load more" button (if next_page_token != null)
UI->>UI: Render "More files may exist" hint (if may_have_more && !next_page_token)
User->>UI: Click "Load more"
UI->>UI: loadMore() – guard: skip if isLoadingMoreRef or no token
UI->>Backend: ListStorageEntries (pageToken: next_page_token)
Backend->>Storage: list_entries(limit=150, page_token)
Storage->>Storage: slice entries[offset:offset+limit]
Storage-->>Backend: StorageListResult(new_entries, new_next_page_token, may_have_more)
Backend-->>UI: StorageEntriesNotification(new_entries, new_next_page_token, ...)
UI->>State: setEntries({append: true, entries: new_entries, nextPageToken, mayHaveMore})
State-->>UI: entries appended, pageMetadata updated
UI->>UI: Re-render button (hidden if new_next_page_token null)
UI->>UI: Show "More files may exist" if applicable
opt Error during loadMore
UI->>UI: setLoadMoreError(error)
UI->>UI: Display error message next to button
end
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Light2Dark
force-pushed
the
Light2Dark/feat-storage-pagination
branch
from
June 9, 2026 17:35
fe2773c to
e446002
Compare
Contributor
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end pagination support for remote storage listings, enabling the frontend storage inspector to fetch additional pages of entries via a “Load more” interaction while also signaling when a provider may have truncated results that cannot be paged further.
Changes:
- Introduces
StorageListResult(entries +next_page_token+may_have_more) and threadspage_tokenthrough backend implementations, runtime command handling, and notifications. - Updates the frontend storage state/hook layer to track pagination metadata per namespace/prefix and append subsequent pages into the cached listing.
- Adds UI affordances in the storage inspector for “Load more” and a “More files may exist” hint, with corresponding test updates.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/_runtime/test_runtime_external_storage.py | Updates runtime storage listing tests to exercise page_token and next_page_token. |
| tests/_data/_external_storage/test_storage_models.py | Updates backend model tests for StorageListResult and adds pagination/may-have-more coverage. |
| packages/openapi/src/api.ts | Extends generated OpenAPI TS types for new pagination fields. |
| packages/openapi/api.yaml | Extends OpenAPI schema for pageToken request field and pagination fields on notifications. |
| marimo/_server/models/models.py | Threads page_token from HTTP request model into the runtime command. |
| marimo/_runtime/commands.py | Adds page_token to StorageListEntriesCommand. |
| marimo/_runtime/callbacks/external_storage.py | Calls backend list_entries(..., page_token=...) and includes pagination metadata in StorageEntriesNotification. |
| marimo/_messaging/notification.py | Adds next_page_token and may_have_more to StorageEntriesNotification. |
| marimo/_data/_external_storage/storage.py | Changes storage backends to return StorageListResult and implements page slicing + “may have more” detection for obstore listings. |
| marimo/_data/_external_storage/models.py | Introduces StorageListResult and updates the StorageBackend.list_entries contract accordingly. |
| frontend/src/core/storage/types.ts | Extends storage state types with per-path pagination metadata. |
| frontend/src/core/storage/state.ts | Implements page fetching/append logic and exposes loadMore and pagination state from useStorageEntries. |
| frontend/src/core/storage/tests/useStorageEntries.test.tsx | Adds coverage for load-more behavior, empty-string page tokens, and duplicate load-more suppression. |
| frontend/src/core/storage/tests/state.test.ts | Adds reducer/state coverage for pagination metadata + append behavior. |
| frontend/src/components/storage/storage-inspector.tsx | Adds “Load more” button and “More files may exist” hint UI using new hook outputs. |
Light2Dark
requested review from
dmadisetti,
kirangadhave and
mscolnick
and removed request for
dmadisetti
June 11, 2026 04:41
kirangadhave
left a comment
Member
There was a problem hiding this comment.
looks great, some nits and questions
Bundle ReportChanges will increase total bundle size by 11.9kB (0.05%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: marimo-esmAssets Changed:
Files in
Files in
|
…tton Thread page tokens through the storage backends, runtime command, and notification, and surface a "Load more" control (plus a "may have more" hint when a provider truncates a listing) in the storage inspector. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… more" tooltip - loadMore now guards on `nextPageToken == null` instead of falsy, so an empty-string token (treated as a valid page by hasMore and the page fetcher) is no longer silently dropped. - Make the "More files may exist" info tooltip keyboard-focusable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous label duplicated the adjacent visible text; give the focusable icon a concise accessible name instead, leaving the detail to the tooltip description. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Light2Dark
force-pushed
the
Light2Dark/feat-storage-pagination
branch
from
June 15, 2026 03:33
167f472 to
0ca831a
Compare
mscolnick
reviewed
Jun 15, 2026
| /** @default null */ | ||
| error?: string | null; | ||
| /** @default false */ | ||
| may_have_more?: boolean; |
Contributor
There was a problem hiding this comment.
what about just has_more?
Member
Author
There was a problem hiding this comment.
I've removed this field entirely. It's very rarely to hit, (user needs to scroll to 1000 items within a directory) + click "loads more" a few times.
Collapse pagination to a single concept: next_page_token. The may_have_more flag was a non-actionable "more files may exist" hint for the rare >1000-entries-per-directory case (which obstore directory listing can't page past), and reviewers found the dual signal confusing. Properly supporting >1000 entries needs a larger listing redesign and is deferred. Removes the flag and the LIMIT_ENTRIES boundary check end-to-end: backend result/notification, runtime callback, frontend state/hook, the "More files may exist" UI, regenerated OpenAPI, and the related tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
2 issues found across 12 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="frontend/src/core/storage/state.ts">
<violation number="1" location="frontend/src/core/storage/state.ts:155">
P2: Dropping `may_have_more` here prevents the inspector from surfacing truncated listings that cannot be paged through.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mscolnick
approved these changes
Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Summary
Split out from #9708. Part of the work for #9662.
Adds pagination to remote storage listings:
page_tokenthrough the storage backends, theStorageListEntriesCommand, and theStorageEntriesNotification.list_entriesnow returns aStorageListResult(entries +next_page_token+may_have_more).Screen.Recording.2026-06-10.at.11.56.00.AM.mov
📋 Pre-Review Checklist
✅ Merge Checklist