GitHub - bw-hro/sqlitemap: sqlitemap is a lightweight C++ wrapper around SQLite that provides a simple, map-like interface.

sqlitemap — Persistent Map Backed by SQLite

sqlitemap is a lightweight C++ wrapper around SQLite that provides a simple, map-like interface.
It’s designed to make working with key-value storage easy and convenient — similar to how sqlitedict works in the Python world. The library is implemented as a single-header file, making integration easy and is licensed under the MIT License.

Features

Persistent key-value storage using SQLite
Easy-to-use map-like interface in C++
Transactions
Custom encoding/decoding
Minimal dependencies (just requires SQLite)
Please refer to the Usage section below for more details

Installation

Just add the sqlitemap.hpp header to your project include files:
Make sure that SQLite is available

#include <bw/sqlitemap/sqlitemap.hpp>

CMake

Example of a minimal CMake setup using FetchContent to obtain sqlitemap

cmake_minimum_required(VERSION 3.15)
project(sqlitemap-consumer)
set(CMAKE_CXX_STANDARD 17)

# Declare and fetch sqlitemap library
include(FetchContent)
FetchContent_Declare(
        sqlitemap
        GIT_REPOSITORY https://github.com/bw-hro/sqlitemap.git
        GIT_TAG v1.2.0 # replace with desired version / branch
        SOURCE_SUBDIR "include" # sqlitemap is single header only
)
FetchContent_MakeAvailable(sqlitemap)

# First install SQLite3 on your system
# e.g. 'sudo apt install libsqlite3-dev'
find_package(SQLite3 REQUIRED)

add_executable(sqlitemap-consumer main.cpp)
target_include_directories(sqlitemap-consumer PRIVATE "${sqlitemap_SOURCE_DIR}/include")
target_link_libraries(sqlitemap-consumer PRIVATE SQLite::SQLite3)

Example of a minimal CMake setup using CPM.cmake to obtain sqlitemap

cmake_minimum_required(VERSION 3.15)
project(sqlitemap-consumer)
set(CMAKE_CXX_STANDARD 17)

# make CPM.cmake available
set(CPM_DIR "${CMAKE_CURRENT_BINARY_DIR}/cmake/CPM.cmake")
file(DOWNLOAD https://github.com/cpm-cmake/CPM.cmake/releases/latest/download/get_cpm.cmake ${CPM_DIR})
include(${CPM_DIR})

# add sqlitemap and SQLite dependencies
CPMAddPackage(URI "gh:bw-hro/sqlitemap@1.2.0" DOWNLOAD_ONLY YES)
CPMAddPackage("gh:sjinks/sqlite3-cmake@3.49.1")

add_executable(sqlitemap-consumer main.cpp)
target_include_directories(sqlitemap-consumer PRIVATE "${sqlitemap_SOURCE_DIR}/include")
target_link_libraries(sqlitemap-consumer PRIVATE SQLite::SQLite3)

vcpkg

You can also use the vcpkg port bw-sqlitemap which handles all dependencies for you

vcpkg install bw-sqlitemap

or add dependency to vcpkg.json manifest file

{
  "name": "your-project",
  "version-string": "1.0.1",
  "dependencies": [
    "bw-sqlitemap"
  ]
}

conan

You can also use the conan recipe sqlitemap which handles all dependencies for you. Assuming you use cmake, just add the following to conanfile.txt

[requires]
sqlitemap/[>=1.2.0]

[generators]
CMakeDeps
CMakeToolchain

and to CMakeLists.txt

cmake_minimum_required(VERSION 3.15)
project(sqlitemap-consumer)

find_package(sqlitemap REQUIRED)

add_executable(${PROJECT_NAME} src/main.cpp)
target_link_libraries(${PROJECT_NAME} PRIVATE sqlitemap::sqlitemap)

Usage

Write

#include <bw/sqlitemap/sqlitemap.hpp>

int main()
{
    bw::sqlitemap::sqlitemap db("example.sqlite");

    db["a"] = "first-item";
    db["b"] = "second-item";
    db["c"] = "third-item";

    // Commit to save items
    db.commit();

    db["d"] = "yet-another-item";
    // Forgot to commit here, that item will never be saved.
    db.close();

    // Always remember to commit, or enable autocommit with
    //
    //     bw::sqlitemap::sqlitemap db(bw::sqlitemap::config()
    //        .file("example.sqlite")
    //        .auto_commit(true));
    //
    // Autocommit is off by default for performance.


    db.connect(); // reconnect, after close
    // some additional helpful write methods, among others:
    db.set("x", "draft");     // set or update value
    db.del("x");              // delete value
    db.emplace("y", "draft"); // insert value if not exists
    db.erase("y");            // delete value returns number 0 if not exists, 1 otherwise
    db.insert(std::pair{"z", "draft"}); // insert value, returns iterator to the inserted item
                                        // attention: iterator can not be advanced,
                                        // it can only be used to access the value
    db.clear(); // clear all items
}

Read

#include <bw/sqlitemap/sqlitemap.hpp>

int main()
{
    bw::sqlitemap::sqlitemap db("example.sqlite");
    std::cout << "There are " << db.size() << " items in the database" << std::endl;
    // There are 3 items in the database

    // Standard map like interface. operator[], iterator, ...

    for (auto& [key, value] : db)
    {
        std::cout << key << " = " << value << std::endl;
    }
    // a = first-item
    // b = second-item
    // c = third-item

    std::string value_a = db["a"];
    std::cout << "value of a: " << value_a << ", value of b: " << db["b"] << std::endl;
    // value of a: first-item, value of b: second-item

    // some additional helpful access methods, among others:
    db.get("a");      // returns value or throws sqlitemap_error if not found
    db.try_get("b");  // returns std::optional containing value or empty if not found
    db.find("c");     // returns iterator to the found item or end() if not found, attention: iterator
                      // can not be advanced, it can only be used to access the value
    db.contains("d"); // returns true if key is found, false otherwise
    db.count("e");    // returns number of items with the given key, 0 or 1
}

Database Connection Lifecycle

The sqlitemap object manages the lifecycle of the SQLite database connection. When the object is created, it automatically connects to the database. When the object goes out of scope and is destroyed, it ensures that the database connection is properly closed.

{
    bw::sqlitemap::sqlitemap db("example.sqlite");
    // The database connection is opened and ready for work
    db["key"] = "value";
    db.commit();
    // The database connection is automatically closed when `db` goes out of scope.
    // db.connect() and db.close() can be used for manual connection management.
}

Transactions

sqlitemap provides transaction management to ensure data consistency and performance. By default, sqlitemap does not auto-commit changes for performance reasons, allowing you to group multiple operations into a single transaction. When autocommit is used, the beginning of a multiple command spanning transaction must be stated explicitly.

Explicit Transactions

You can explicitly control transactions using the begin_transaction(), commit() and rollback() methods:

begin_transaction(): Starts a new transaction explicitly. (required when autocommit is used)
commit(): Saves all changes made during the current transaction to the database.
rollback(): Discards all changes made during the current transaction.

#include <bw/sqlitemap/sqlitemap.hpp>

int main()
{
    bw::sqlitemap::sqlitemap db("example.sqlite"); // no autocommit active

    try
    {
        // Write operations implicitly begins a new transaction
        // db.begin_transaction() can be used to  make it explicit
        db["key1"] = "value1";
        db["key2"] = "value2";

        // Commit the transaction to save changes
        db.commit();

        db["key3"] = "value3";
        throw std::runtime_error("An error occurred");

        // This line will not be reached, and the changes to "key3" will not be saved
        db.commit();
    }
    catch (const std::exception& e)
    {
        std::cerr << "Exception: " << e.what() << std::endl;

        // Rollback the transaction to discard uncommitted changes
        db.rollback();

        // Use alternative value in error case
        db["key3"] = "alt_value3";
        db.commit();
    }
}

If you prefer not to manage transactions manually, you can enable auto-commit mode. In this mode, every write operation is immediately committed to the database until you explicitly state a new transaction.

#include <bw/sqlitemap/sqlitemap.hpp>

int main()
{
    bw::sqlitemap::sqlitemap db(bw::sqlitemap::config()
        .file("example.sqlite")
        .auto_commit(true)); // Enable auto-commit mode

    db["key1"] = "value1"; // Automatically committed
    db["key2"] = "value2"; // Automatically committed

    // begin new transaction explicitly
    db.begin_transaction();
    db["key3"] = "value3";
    db["key4"] = "value4";

    // Rollback the transaction to discard uncommitted changes
    db.rollback();
}

Tables

A database file can store multiple tables. The default table "unnamed" is used when no table name is specified.

#include <bw/sqlitemap/sqlitemap.hpp>

int main()
{
    bw::sqlitemap::sqlitemap db(bw::sqlitemap::config()
        .file("example.sqlite")
        .auto_commit(true));
    
    db["k1"] = "v1";
    db["k2"] = "v2";

    bw::sqlitemap::sqlitemap cities(bw::sqlitemap::config()
        .file("example.sqlite")
        .table("cities")
        .auto_commit(true));

    cities["rostock"] = "https://en.wikipedia.org/wiki/Rostock";
    cities["raleigh"] = "https://en.wikipedia.org/wiki/Raleigh";
    cities["dalian"] = "https://en.wikipedia.org/wiki/Dalian";

    bw::sqlitemap::sqlitemap countries(bw::sqlitemap::config()
        .file("example.sqlite")
        .table("countries")
        .auto_commit(true));

    countries["germany"] = "https://en.wikipedia.org/wiki/Germany";
    countries["usa"] = "https://en.wikipedia.org/wiki/United_States";
    countries["china"] = "https://en.wikipedia.org/wiki/China";

    auto tables = bw::sqlitemap::get_tablenames("example.sqlite");
    // tables contains {"cities", "countries", "unnamed"}
}

Operation modes

sqlitemap supports several operation modes that define how the database and its tables are handled. These modes can be configured using the operation_mode enum.

Available Modes

operation_mode::c (Create/Default): Opens the database in read-write mode. If the database or table does not exist, it will be created.
operation_mode::r (Read-Only): Opens the database in read-only mode. No modifications are allowed.
operation_mode::w (Write/Drop): Opens the database in read-write mode but drops the contents of the specified table before use.
operation_mode::n (New): Creates a new database, erasing all existing tables.

#include <bw/sqlitemap/sqlitemap.hpp>

int main()
{
    using namespace bw::sqlitemap;

    // Open database in read-only mode
    sqlitemap db_readonly(config()
        .file("example.sqlite")
        .mode(operation_mode::r));
}

In addition to the application-level operation_mode, the behavior of underlying SQLite database can be configured using PRAGMA statements. sqlitemap provides a convenient API to set these pragmas, which are executed when the connection to the database is established.

An example of configuring sqlitemap to optimize for high-concurrency workloads.

#include <bw/sqlitemap/sqlitemap.hpp>

int main()
{
    using namespace bw::sqlitemap;

    sqlitemap sm(config()
        .filename("example.sqlite")
        .pragma("journal_mode", "WAL") // DELETE | TRUNCATE | PERSIST | MEMORY | WAL | OFF
        .pragma("cache_size", -64000)   // -64000 = 64000KiB, 64000 = number of pages
        .pragma("temp_store = 2")      // 0 = DEFAULT, 1 = FILE, 2 = MEMORY
        .pragma("PRAGMA synchronous = NORMAL")); // 0 = OFF, 1 = NORMAL, 2 = FULL

        // configuration.pragma(...) accepts a whole statement, or a flag-value pair
        // it prepends statements with "PRAGMA " when this prefix is missing
}

Encoding/Decoding

sqlitemap supports custom encoding and decoding mechanisms for both keys and values to handle complex data types. By default, sqlitemap works with simple key-value pairs of std::string. However, you can define custom codecs to serialize and deserialize more complex types, such as structs or user-defined objects.

#include <bw/sqlitemap/sqlitemap.hpp>

// point, feature definded in test/catch2/unit_tests/custom.hpp

int main()
{
    using namespace bw::testhelper;

    auto kc = key_codec([](point p) { return point::to_string(p); },
                        [](std::string s) { return point::from_string(s); });

    auto vc = value_codec([](feature f) { return feature::to_string(f); },
                          [](std::string s) { return feature::from_string(s); });

    // configure custom key and value codecs
    sqlitemap db(config(kc, vc));

    // working with point/feature directly encoding/decoding will be applied automatically
    db.set({0, 0, 0}, {"origin", 5});
    db[{0, 0, 0}] == feature{"origin", 5}; // true

    db[{1, 0, 0}] = {"x-direction", 1};
    db.get({1, 0, 0}) == feature{"x-direction", 1}; // true
}

sqlitemap_cereal.cpp demonstrates how to use sqlitemap with custom data types stored as blob using cereal.
sqlitemap_json.cpp demonstrates how to use sqlitemap with custom data types stored as json string using nlohmann::json.
sqlitemap_tiles.cpp demonstrates how to use sqlitemap with custom data types stored as blobs.
sqlitemap_zlib.cpp demonstrates how to use sqlitemap to store compressed values using zlib.
Please make sure to also inspect sqlitemap_codecs_tests.cpp were further details regarding encoding/decoding using codecs are covered.

Using `sqlitemap_t` for convenience with sqlitemap type construction

To reduce boilerplate when creating sqlitemap instances, the helper alias sqlitemap_t automatically selects the appropriate codec configuration based on the template arguments you provide.

#include <bw/sqlitemap/sqlitemap.hpp>

using namespace bw::sqlitemap;

// Default key/value types (std::string identity)
sqlitemap_t<> db; // same as just sqlitemap

// Specify only the value type (key codec defaults to std::string identity)
sqlitemap_t<int> db;
sqlitemap_t<value_codec_t<my_class, std::string>> db;

// Specify only the key type (value codec defaults to std::string identity)
sqlitemap_t<key_codec_t<my_class, std::string>> db;

// Specify both key and value types (identity codecs inferred)
sqlitemap_t<int, double> db;

// Use explicit codec types directly
sqlitemap_t<key_codec_t<int>, value_codec_t<my_class, std::string> db;

Example showing how sqlitemap_t, key_codec_t and value_codec_t support convenient application-specific type aliases and improve readability:

using namespace bw::sqlitemap;

template<typename T> blob to_blob(const T& data){...}
template<typename T> T from_blob(blob blob){...}

struct tile_location ...
struct tile_bitmap ...

class tiles
{
    public:
    using location_codec = key_codec_t<tile_location, blob>;
    using bitmap_codec = value_codec_t<tile_bitmap, blob>;
    using tiles_db = sqlitemap_t<location_codec, bitmap_codec>;

    tiles()
        : data(std::make_unique<tiles_db>(
                config(location_codec{to_blob<tile_location>, from_blob<tile_location>},
                       bitmap_codec{to_blob<tile_bitmap>, from_blob<tile_bitmap>})
                    .filename("tiles.sqlite")
                    .table("tiles")))
    {
    }

    void save(const tile_location& location, const tile_bitmap& bitmap)
    {
        data->set(location, bitmap);
    }

    tile_bitmap tile_for(const tile_location& location) const
    {
        return data->get(location);
    }

    private:
    std::unique_ptr<tiles_db> data;
};

Tests / Examples / Additional Documentation

sqlitemap is extensively covered by unit tests, which also serve as documentation and usage examples.
Additionally, sqlitemap_client is a command-line wrapper around sqlitemap that demonstrates and covers its most important features and how to embed it into your own project. An executable can be found in the release section.