joedb
  • 1. Introduction
  • 2. User Guide
  • 3. Reference
    • 3.1. API reference
    • 3.2. File Format
    • 3.3. Network Protocol
    • 3.4. Tools
    • 3.5. Testing
    • 3.6. TODO
    • 3.7. Links
    • 3.8. Release Checklist
    • 3.9. History
    • 3.10. License
joedb
  • 3. Reference
  • 3.7. Links

3.7. Links

Database software with history

  • https://redis.io/ (the AOF persistence option has similarities to joedb)

  • https://kafka.apache.org/intro Kafka is an event streaming system

  • https://samza.apache.org/

  • https://prevayler.org/ (inactive)

  • https://www.datomic.com/

  • https://github.com/attic-labs/noms (inactive) a git-like database

  • https://github.com/dolthub/dolt “git for data”

Streaming replication, continuous archiving, point-in-time recovery

  • https://www.postgresql.org/docs/current/warm-standby.html

  • https://www.postgresql.org/docs/current/continuous-archiving.html

  • https://github.com/benbjohnson/litestream

Structured data storage (no history, no durable incremental update):

  • https://developers.google.com/protocol-buffers/

  • https://google.github.io/flatbuffers/

  • https://www.json.org/

  • https://en.wikipedia.org/wiki/XML

  • https://www.boost.org/doc/libs/1_85_0/libs/serialization/doc/index.html

  • http://www.garret.ru/post/readme.htm

On-disk containers:

  • http://stackoverflow.com/questions/149488/disk-backed-stl-container-classes

  • http://stxxl.sourceforge.net/

Using SQL from C++

  • https://sqlite.org/ (non-durable transactions: https://www.sqlite.org/wal.html#fast)

  • https://github.com/rbock/sqlpp11

  • https://www.webtoolkit.eu/wt/doc/tutorial/dbo.html

  • https://codesynthesis.com/products/odb/

  • https://sourceforge.net/projects/litesql/

  • http://en.wikipedia.org/wiki/Active_record_pattern

  • C++20 ❤ SQL - John Bandela - CppCon 2021

    • https://www.youtube.com/watch?v=-P9FyevnL6U

    • https://cppcon.digital-medium.co.uk/wp-content/uploads/2021/10/Cpp20SQL.pdf

    • https://github.com/google/cpp-from-the-sky-down/tree/master/meta_struct_20/cppcon_version

Compression

  • https://github.com/google/snappy (used by LevelDB)

  • https://github.com/google/brotli

  • https://facebook.github.io/zstd/

  • https://lz4.org/

  • https://www.percona.com/blog/evaluating-database-compression-methods/

Fast IO:

  • https://github.com/ned14/llfio

  • https://stackoverflow.com/questions/1201261/what-is-the-fastest-method-for-high-performance-sequential-file-i-o-in-c

  • https://devblogs.microsoft.com/oldnewthing/20221130-00/?p=107505

File Locking, atomicity, durability:

  • https://gavv.net/articles/file-locks/

  • https://man7.org/linux/man-pages/man2/flock.2.html

  • https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-lockfileex

  • https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_07

  • https://utcc.utoronto.ca/%7Ecks/space/blog/unix/WriteNotVeryAtomic

  • https://stackoverflow.com/questions/35595685/write2-read2-atomicity-between-processes-in-linux

  • https://danluu.com/deconstruct-files/

  • https://danluu.com/file-consistency/

Crash resistance:

[Rosenblum-1991]

The Design and Implementation of a Log-Structured File System

[Redis-2016a]

Redis persistence demystified

[Redis-2016b]

fsync() on a different thread: apparently a useless trick

[SQLite-AC]

Atomic Commit In SQLite

[Moyer-2011]

Ensuring data reaches disk

[serverfault-2009]

SATA Disks that handle write caching properly?

[PG-2018]

PostgreSQL’s fsync() surprise

[PG-2019]

How is it possible that PostgreSQL used fsync incorrectly for 20 years, and what we’ll do about it

[Miller-2024]

Userland Disk I/O

Other links:

  • https://stackoverflow.com/questions/39281/database-design-for-revisions

  • https://stackoverflow.com/questions/750782/database-design-for-text-revisions

  • https://stackoverflow.com/questions/2724187/where-are-all-the-native-revisioned-databases

  • https://stackoverflow.com/questions/9484714/how-to-store-datas-change-history

  • https://stackoverflow.com/questions/9852703/store-all-data-changes-with-every-details-like-stackoverflow

  • https://stackoverflow.com/questions/12321200/database-row-snapshots-revisions

  • https://stackoverflow.com/questions/68323888/how-to-implement-a-real-time-data-logging-backup-system

  • https://opendata.stackexchange.com/questions/748/is-there-a-git-for-data

  • https://blogs.oracle.com/connect/post/managing-history

  • https://zodb.org/en/latest/

  • https://ayende.com/blog/162792/worlds-smallest-no-sql-database-persistent-transaction-logs

  • https://database-programmer.blogspot.fr/2008/07/history-tables.html

  • https://www.reddit.com/r/databasedevelopment/

  • https://www.reddit.com/r/programming/comments/3tfkdq/immutability_in_db_might_be_the_next_big_thing/

  • https://www.youtube.com/watch?v=fU9hR3kiOK0

  • I Want Decentralized Version Control for Structured Data!

  • The SQLite team is preparing an efficient remote replication tool

Papers

  • The End of an Architectural Era (It’s Time for a Complete Rewrite)

  • Are You Sure You Want to Use MMAP in Your Database Management System?, YouTube presentation of the paper

Data Loss Stories

  • https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/

  • https://www.reddit.com/r/ExperiencedDevs/comments/1j2wrdv/wiped_my_companys_production_db_last_week/

  • https://www.brentozar.com/archive/2015/02/9-ways-to-lose-your-data/

Previous Next

© Copyright Rémi Coulom.

Built with Sphinx using a theme provided by Read the Docs.