Object Storage Direct
Each service publishes directly to object storage. Competing writers coordinate through conditional updates to the log manifest.
Service → object storage
How it works
Your software owns its state and behavior. Mainframe connects replicas to one ordered history.
Architecture
Your handler, application logic, and local data share a process. Replicas coordinate at the log backend.
| Part | Responsibility |
|---|---|
| Your software | The API, command format, data structures, indexes, and rules for changing state. |
| Mainframe core | Compose engines, submit commands to the chosen log, and apply ordered entries while tracking progress. |
| Log backend | Establish one order, publish entries durably, serve retained history, and coordinate competing writers. |
| Your service integration | Request deduplication, atomic state persistence, snapshots, read freshness, and external effects. |
Where agreement comes from
The backend establishes the log’s order and durability. Saving an object alone does not decide which concurrent command comes first.
Object Storage Broker
Services send appends to an HTTP broker, which batches and publishes them. Readers fetch the log from object storage. The broker is an additional process to operate.
Write: service → broker → object storage
Walk through publication
The manifest identifies the batches that belong to the log. An uploaded object alone is not an accepted append. If a response is lost, reconcile against fresh metadata before deciding whether to retry. Backend correctness depends on the store honoring the required conditional writes, reads, and durability. Read the validation approach ↗
The object storage backends are prototypes. An in-memory log is available for local development; it loses its history when the process stops.
Read and recover from a known position
Read from local state
- For a read that allows lag, return a consistent local view and its applied position.
- For a fresh read, obtain the latest position using a backend with the required consistency.
- Apply all entries before that boundary, then read a consistent view of the resulting state.
If the required barrier cannot be reached, wait or return an error. Returning older data would change the read contract.
Watch a replica catch up ↗Recover a replica
- Restore a compatible snapshot of the data, request results, and next-to-apply position.
- Replay the retained log from that position, applying entries in order.
- Serve requests after reaching the required boundary. Keep history until every required recovery path can do the same.
Saving a progress number does not save application state. A missing entry or incompatible snapshot must stop recovery.
Watch recovery ↗A recorded command can still be rejected by your application.
Acceptance into the log establishes an order. During apply, your code checks stock, record versions, or another rule against the state at that position. Record the result and advance progress even when the application rejects the command. Use stable request IDs so retries can return the saved result.
Follow two conflicting updates ↗Engine composition, API boundaries, and checkpoints +
The current Engine API is Rust. A JSON-RPC handler can expose your service to clients in other languages; this implementation does not provide Go or TypeScript engine SDKs.
The examples import mainframe_core from the mainframe-core package. Place the example crate beside your mainframe checkout.
SharedLog- Append, read by position, inspect the tail, seal, and trim. A backend supplies these operations and their ordering contract.
Engineproposecan transform or reject a command before publication.applyprocesses an ordered entry.syncis the engine's synchronization hook.EngineStack- Proposals flow from the top engine toward the log; application flows back from the bottom engine. Application progress is serialized within a stack.
// Submit a command and learn its log position.
p = stack.propose(command).await?;
// Apply entry p using an exclusive upper boundary.
stack.sync_and_apply(p.next()).await?;
// Read the application result saved while applying it.
return local_results[request_id];This is pseudocode. Your integration must persist state, results, and progress
consistently and tolerate retries of failed or cancelled apply calls.
checkpoint() returns a common engine boundary; the caller still
saves the engine snapshots. Restore every engine to that boundary before
constructing the stack with from_checkpoint.
Implementation basis: core/src/lib.rs and the
backend sources. Inspect a complete local example ↗
Changing the backend with a virtual log +
A loglet owns one segment of history. VirtualLog presents a chain
of those segments as one shared log. Reconfiguration seals the current segment,
creates an empty successor at the same boundary, and publishes the chain update
through MetaStore. Earlier history remains addressable.
This separates the application's logical positions from the backend serving each segment. Safe reconfiguration still depends on the loglets' seal behavior and the metadata store's conditional update contract.
Implementation basis: core/src/virtual_log.rs.
Read the virtual consensus explanation and paper ↗
Guarantees & testing
These are the shared-log requirements. A persistent deployment must validate its backend against them.
| The log must provide | Your application must handle |
|---|---|
| One ordered history with unique positions. | Deterministic application, including stock, version, and transaction rules. |
| Retention of acknowledged entries on a persistent backend, until safely trimmed. | Atomic persistence of state, request results, and progress; compatible snapshots. |
| Consistent tail and positional reads under its documented contract. | A read barrier and catch-up when a caller needs fresh data. |
| A fixed final boundary after sealing; explicit errors for unavailable history. | Retry reconciliation, request deduplication, and retention of everything recovery needs. |
The in-memory log is for local development and loses history on exit. Object storage backends are prototypes; their guarantees depend on the store’s conditional writes, reads, and durability.
How the core is tested
The suites exercise individual log operations, engine recovery, competing writers, and failures between processes.
These descriptions document test coverage, not a live test result. Public CI reports and source browsing are not currently provided on this site; reproducing the suites requires access to the core repository.
Check every operation.
Append entries, read them back, close the log to new writes, and remove an old prefix. Shared tests check ordering, unique positions, concurrent reads and writes, and randomized workloads.
For example Race an append against closing the log. An accepted entry must fall before the final boundary; later appends must fail.
core/src/tests/ · core/src/lib.rs
Resume at the right entry.
Tests exercise repeated and concurrent catch-up calls, checkpoint recovery, and failures partway through an engine stack. They also check that switching log segments preserves the sequence.
For example Make one engine fail during apply. On retry, resume at that engine without calling engines that already completed the entry again.
review/tests/regressions.rs · core/src/virtual_log.rs
Exercise the ambiguous cases.
Integration and regression tests use disposable MinIO storage. They cover independent writers, failed metadata reads, failed or lost commit responses, concurrent trimming, and interrupted segment transitions.
For example Lose the response to a metadata commit. Check that recovery does not publish a gap in the log or reuse an occupied position.
review/tests/regressions.rs · object-storage-direct/ · object-storage-broker/
Check what callers observed.
A separate runner starts concurrent clients, crashes and restarts processes, and reads back acknowledged writes. Its local fault mode interrupts the storage connection. It saves each operation and outcome for the Jepsen/Knossos checker.
For example Ask whether the recorded operations fit one legal sequential log history that respects real-time order. The checker also requires recovery-read coverage; an empty history cannot pass.
scripts/test_processes.py · jepsen-suite/
Run the tests
RUN THE CORE SUITE
Start without cloud storage.
From the Mainframe repository root, run the standalone core tests. These exercise the core implementation without provisioning S3 or MinIO.
cargo test --manifest-path core/Cargo.toml --lockedWhat runs in CI
On pushes and pull requests to main, the workflow runs core tests, workspace tests, review regressions against isolated MinIO, and tests for the storage-test runner. A separate job tests the history checker against known valid and invalid histories.
What runs separately
Independent-process fault runs and live S3/GCS validation are
separate from those CI jobs. Local results establish behavior
under the tested conditions; they do not establish the same
behavior on every cloud backend. The run commands and setup are
documented in review/README.md.
Performance
Local reads access your own state without a database hop. Writes pay for publication through the selected backend; fresh reads may also wait for catch-up. Measure those paths separately in your environment.
No measured throughput or latency results are published here yet.
Build a benchmark command
Benchmark configurationCLI builder
1 writer100 writers
Your workloadPER BACKEND
500 attempted appends · 31.3 KiB requested payload per backend
Run from the mainframe directory
Build with
cargo build --release --workspace. Set
S3_BUCKET, region, and credentials for a disposable
test bucket. For local MinIO, set AWS_ENDPOINT_URL.
The wrapper starts a broker on port 8080; use that address for
BROKER_URL.
How to make a useful comparison +
Keep the machine, endpoint, region, payload size, and concurrency identical. Record the source revision and environment alongside the output. Repeat runs to capture variation, and report failures with latency. The harness runs backends sequentially when “both” is selected; reads measure a different workload from concurrent appends. Use dedicated test storage: the benchmark writes data and does not clean it up.
Successful ops / sec
Successful operations divided by elapsed wall time. Failures are counted separately.
p50 & p99 latency
p50 is the median time for a successful operation; 99% finish within the p99 time. Read checks run one at a time.
Verified reads
Checks both position and payload for entries written during the current run.
Research
Meta’s Delos is the original research and production system. Mainframe is an independent implementation inspired by it. These publications describe Meta’s deployments and the ideas behind this library.
Meta implementations
DelosTable: a database above the log.
Delos: Simple, flexible control plane storage ↗Meta Engineering · June 2019Database design paper · SOSP 2021 ↗
Meta described DelosTable as a relational store for control-plane data, including the Resource Broker’s machine-allocation ledger. Its table API supports transactions, secondary indexes, and range queries. A shared log coordinates changes while replicas maintain their local representation.
Meta’s Delos supplies an ordered history and a replication framework. Meta reported replacing ZooKeeper-backed ordering with a native implementation without service downtime.
Follow a table update
Design lesson: keep database semantics separate from the consensus implementation. This illustration omits the full transaction protocol; it is not a SQL execution trace. Diagram: explanatory reconstruction based on the cited publication.
Zelos: the ZooKeeper API, on Meta’s Delos.
Introducing Zelos: A ZooKeeper API leveraging Delos ↗Meta Engineering · June 2022Engine design paper · SOSP 2021 ↗
Meta built Zelos to support ZooKeeper clients using Delos underneath. The API includes sessions, watches, and ephemeral nodes. Those behaviors require additional protocol logic above a shared log; a total log order alone does not preserve a client session’s issue order.
Meta’s Delos supplies the ordered log and composable engine framework. Replicas use reusable engines to interpret that history consistently.
Follow session ordering
Design lesson: reuse consensus, then explicitly implement the API’s additional semantics. Watches and ephemeral nodes also depend on session state, not just the ordering shown here. Diagram: explanatory reconstruction based on the cited publication.
Mainframe’s core ideas
Change consensus beneath the application.
Virtual Consensus in Delos ↗Mahesh Balakrishnan et al. · OSDI 2020
Virtual Consensus in Delos separates the application’s logical log from its underlying loglets. Sealing one segment and installing its successor lets the application continue across changes in the replication implementation.
VirtualLog and Loglet model the segment boundary and reconfiguration. Recovery and competing transitions require configuration coordination.
Follow a loglet transition
Source: paper §3–4. This is a reconstruction of the boundary transition, not a full failure-recovery protocol. Diagram: explanatory reconstruction based on the cited publication.
Reuse more than the consensus protocol.
Log-structured Protocols in Delos ↗Mahesh Balakrishnan et al. · SOSP 2021
Log-structured Protocols in Delos describes composable engines that execute above a shared log. Different databases can share protocol machinery while keeping their own APIs and application state.
Engine exposes propose, apply, and sync; EngineStack connects engines to SharedLog. The examples of batching, leases, and sessions here describe the research, not a catalog of shipped engines.
Follow the engine stack
Source: paper Figure 1 and §3–4. The arrows show the two directions of processing, not a network hop for every engine. Diagram: explanatory reconstruction based on the cited publication.
Foundations and related work
A history many applications can share.
CORFU: A Shared Log Design for Flash Clusters ↗Mahesh Balakrishnan et al. · NSDI 2012
CORFU presents a shared log over distributed flash storage. Clients use a sequencer and a mapping from log positions to storage units. Applications consume the history to construct their own state.
SharedLog exposes positional reads and appends. The S3 backends do not implement CORFU’s flash-cluster protocol or imply that a standalone sequencer provides consensus.
Follow a CORFU append
Incomplete writes need explicit resolution. The diagram simplifies chain replication, filling, and reconfiguration; follow the publication for the complete protocol. Diagram: explanatory reconstruction based on the cited publication.
The rules underneath the diagrams.
Replicated state machines
Start from the same state. Deliver the same ordered inputs. Apply a deterministic transition function. Each replica then computes the same result. A live clock or random choice must not make replicas disagree during application.
Same initial state→ Same ordered inputs→ Same resulting state
Implementing Fault-Tolerant Services Using the State Machine
Approach: A Tutorial ↗Fred B. Schneider · ACM Computing Surveys, 1990
Linearizability
Operations must fit a legal sequential history that respects real-time order. If a write finishes before a read begins, that read cannot return the earlier value when no later write intervenes. Concurrent operations may have either legal order.
Write x = 1 completes→ Read begins→ Return 1
Linearizability: A Correctness Condition for Concurrent
Objects ↗Maurice Herlihy & Jeannette M. Wing · ACM TOPLAS,
1990
These are correctness foundations, not storage backends. The fresh-read example and history checker connect these ideas to this project.
Scalog: scale the shared log.
Scalog separates data replication from global ordering and studies how a shared log can scale and reconfigure. It is useful context for understanding that a shared-log API can have very different implementations underneath.
Scalog: Seamless Reconfiguration and Total Order in a Scalable
Shared Log ↗Cong Ding, David Chu, Evan Zhao, Xiang Li, Lorenzo Alvisi &
Robbert van Renesse · NSDI 2020
Related work cited in the repository; Scalog’s protocol is not implemented by these S3 backends.