2.7. Concurrency

Joedb offers mechanisms to access a single database from multiple processes on the same machine, or from remote machines over the network.

2.7.1. Principle

A joedb client is made of two parts: a file, and a connection. The file stores a replica of the database, and the connection is used for synchronization. A connection provides 3 operations:

  • pull: update the file with new journal entries from the connection.

  • lock_pull: get exclusive write access to the connection, and pull.

  • push_unlock: write new modifications of the file to the connection, and release the lock.

So it works a bit like git, with the significant difference that merging branches is not possible. History must be linear, and a central mutex is used to prevent branches from diverging.

Locking the central mutex is not strictly necessary: offline changes can be made to a local database without any synchronization with the remote server. It will still be possible to push those changes to the server when connecting later, but the push will succeed only if there is no conflict.

2.7.2. Example

The compiler produces code that ensures that locks and unlocks are correctly paired, and modifications to the database can only occur during a lock. This is done with transaction function that takes a lambda as parameter, and executes it between a lock_pull and a push_unlock.

#include "tutorial/Client.h"
#include "joedb/concurrency/File_Connection.h"
#include "joedb/journal/Memory_File.h"

/////////////////////////////////////////////////////////////////////////////
int main()
/////////////////////////////////////////////////////////////////////////////
{
 //
 // This sets up a configuration with a server and 2 clients.
 //
 joedb::Memory_File server_file;
 joedb::Memory_File client1_file;
 joedb::Memory_File client2_file;

 joedb::File_Connection connection(server_file);

 tutorial::Client client1(client1_file, connection);
 tutorial::Client client2(client2_file, connection);

 //
 // The databases are empty. client1 will add a few cities.
 //
 // All write operations are performed via the transaction function.
 // The transaction function takes a lambda as parameter.
 // The lock_pull operation is performed before the lambda, and the push_unlock
 // operation is performed after the lambda, if no exception was thrown.
 // If any exception was thrown during the lambda, then the changes
 // are not pushed to the connection, and the connection is unlocked.
 // Writes that occured in a transaction before an exception are not sent to
 // the connection, but they are written to the file.
 //
 client1.transaction([](tutorial::Writable_Database &db)
 {
  db.new_city("Paris");
  db.new_city("New York");
  db.new_city("Tokyo");
 });

 //
 // client1.get_database() gives a read-only access to the client file
 //
 std::cout << "Number of cities for client1: ";
 std::cout << client1.get_database().get_city_table().get_size() << '\n';

 //
 // Client1 added cities, and they were pushed to the central database.
 // They have not yet reached client2.
 //
 std::cout << "Number of cities for client2 before pulling: ";
 std::cout << client2.get_database().get_city_table().get_size() << '\n';

 //
 // Let's pull to update the database of client2
 //
 client2.pull();
 std::cout << "Number of cities for client2 after pulling: ";
 std::cout << client2.get_database().get_city_table().get_size() << '\n';

 return 0;
}

It produces this output:

Number of cities for client1: 3
Number of cities for client2 before pulling: 0
Number of cities for client2 after pulling: 3

2.7.3. Connections

The constructor of the tutorial::Client class takes two parameters: a file for storing the database journal, and a connection. The connection is an object of the Connection class, that provides the synchronization operations (pull, lock_pull, push_unlock). This section presents the different kinds of available connections.

2.7.3.1. Plain Connection

The Connection superclass does not connect to anything. Such a connection can be used in a client to handle concurrent access to a local file. If the file was opened with Open_Mode::shared_write, clients can start write transactions simultaneously, and the connection will use file locking to synchronize them.

joedbc produces a convenient tutorial::File_Client class that creates the connection and the client in a single line of code. Here is an example:

#include "tutorial/File_Client.h"

#include "joedb/ui/main_exception_catcher.h"

#include <chrono>
#include <thread>

/////////////////////////////////////////////////////////////////////////////
static int local_concurrency(int argc, char **argv)
/////////////////////////////////////////////////////////////////////////////
{
 tutorial::File_Client client("local_concurrency.joedb");

 while (true)
 {
  client.transaction([](tutorial::Writable_Database &db)
  {
   db.new_person();
  });

  std::cout << "I have just added one person. Population: ";
  std::cout << client.get_database().get_person_table().get_size() << '\n';
  std::this_thread::sleep_for(std::chrono::seconds(1));
 }

 return 0;
}

/////////////////////////////////////////////////////////////////////////////
int main(int argc, char **argv)
/////////////////////////////////////////////////////////////////////////////
{
 return joedb::main_exception_catcher(local_concurrency, argc, argv);
}

Multiple instances of this program can safely write to the same database concurrently.

2.7.3.2. File_Connection

File_Connection creates a connection to a file. Here are some typical use cases:

  • File_Connection can be used to make a safe and clean copy of a database that contains a transaction that was not checkpointed, either because the database is currently being written to, or because of a previous crash.

  • File_Connection can be used to convert between different file formats. For instance, pushing a plain joedb file to a Brotli_File will create a compressed database.

  • A File_Connection to an SFTP_File can be a convenient way to pull from a remote database without running a joedb server on the remote machine. Performance will be inferior to running a joedb server, though. Similarly, a File_Connection to a CURL_File can be used to pull from a joedb database served by a web server.

2.7.3.3. Server_Connection

Server_Connection allows connecting to a running joedb_server using the joedb network protocol.

The constructor of Server_Connection takes a Channel parameter. Two channel classes are provided:

2.7.4. Combining Local and Remote Concurrency

A client can handle concurrency for both its file and its connection simultaneously: it is possible for two different clients running on the same machine to share a connection to the same remote server, and also share the same local file. For this to work, the local file must be opened with Open_Mode::shared_write.

2.7.5. Using a Client_Lock instead of a Lambda

The transaction function is a simple way to handle the lock-pull-write-push-unlock sequence, but may not be flexible enough to handle some more complex use cases. The Client_Lock object allows:

  • starting the transaction in one function, and finishing it in another one,

  • pushing multiple times in the middle of a transaction, without unlocking the connection,

  • writing data in one thread, and asynchronously pushing from time to time in another one (use a mutex).

Client_Lock performs Connection::lock_pull() in its constructor, and you have to explicitly call either Client_Lock::push_unlock() or Client_Lock::unlock() right before its destruction.

#include "joedb/ui/main_exception_catcher.h"
#include "tutorial/File_Client.h"

static int client_lock(int argc, char **argv)
{
 tutorial::File_Client client("tutorial.joedb");

 {
  tutorial::Client_Lock lock(client);

  lock.get_database().write_comment("Hello");
  lock.push();
  lock.get_database().write_comment("Goodbye");
  lock.get_database().write_timestamp();
  lock.push_unlock();
 }

 return 0;
}

int main(int argc, char **argv)
{
 return joedb::main_exception_catcher(client_lock, argc, argv);
}