MemoryBackend

Store data in an in-memory store built in to clearskies.

Overview
silent_on_missing_tables

Overview

Usage

Since the memory backend is built into clearskies, there’s no configuration necessary to make it work: simply attach it to any of your models and it will manage data for you. If you want though, you can declare a binding named memory_backend_default_data which you fill with records for your models to pre-populate the memory backend. This can be helpful for tests as well as tables with fixed values.

Testing

A primary use case of the memory backend is for building unit tests of your code. You can use the dependency injection system to override other backends with the memory backend. You can still operate with model classes in the exact same way, so this can be an easy way to mock out databases/api endpoints/etc… Of course, there can be behavioral differences between the memory backend and other backends, so this isn’t always perfect. Hence why this works well for unit tests, but can’t replace all testing, especially integration tests or end-to-end tests. Here’s an example of that. Note that the UserPreference model uses the cursor backend, but when you run this code it will actually end up with the memory backend, so the code will run even without attempting to connect to a database.

import clearskies


class UserPreference(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.CursorBackend()
    id = clearskies.columns.Uuid()


cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        lambda user_preferences: user_preferences.create(no_data=True).id,
    ),
    classes=[UserPreference],
    class_overrides={
        clearskies.backends.CursorBackend: clearskies.backends.MemoryBackend(),
    },
)

if __name__ == "__main__":
    cli()

Note that the model requests the CursorBackend, but then we switch that for the memory backend via the class_overrides kwarg to clearskies.contexts.Cli. Therefore, the above code works regardless of whether or not a database is running. Since the models are used to interact with the backend (rather than using the cursor directly), the above code works the same despite the change in backend.

Again, this is helpful as a quick way to manage tests - swap out a database backend (or any other backend) for the memory backend, and then pre-populate records to test your application logic. Obviously, this won’t cach backend-specific issues (e.g. forgetting to add a column to your database, mismatches between column schema and backend schema, missing indexes, etc…), which is why this helps for unit tests but not for integration tests.

Production Usage

You can use the memory backend in production if you want, although there are some important caveats to keep in mind:

There is limited attempts at performance optimization, so you should test it before putting it under substantial loads.
There’s no concept of replication. If you have multiple workers, then write operations won’t be persisted between them.

So, while there may be some cases where this is useful in production, it’s by no means a replacement for more typical in-memory data stores.

Predefined Records

You can declare a binding named memory_backend_default_data to seed the memory backend with records. This can be helpful in testing to setup your tests, and is occassionally helpful for keeping track of data in fixed, read-only tables. Here’s an example:

import clearskies


class Owner(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String()
    phone = clearskies.columns.Phone()


class Pet(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String()
    species = clearskies.columns.String()
    owner_id = clearskies.columns.BelongsToId(Owner, readable_parent_columns=["id", "name"])
    owner = clearskies.columns.BelongsToModel("owner_id")


wsgi = clearskies.contexts.WsgiRef(
    clearskies.endpoints.List(
        model_class=Pet,
        readable_column_names=["id", "name", "species", "owner"],
        sortable_column_names=["id", "name", "species"],
        default_sort_column_name="name",
    ),
    classes=[Owner, Pet],
    bindings={
        "memory_backend_default_data": [
            {
                "model_class": Owner,
                "records": [
                    {"id": "1-2-3-4", "name": "John Doe", "phone": "555-123-4567"},
                    {"id": "5-6-7-8", "name": "Jane Doe", "phone": "555-321-0987"},
                ],
            },
            {
                "model_class": Pet,
                "records": [
                    {"id": "a-b-c-d", "name": "Fido", "species": "Dog", "owner_id": "1-2-3-4"},
                    {"id": "e-f-g-h", "name": "Spot", "species": "Dog", "owner_id": "1-2-3-4"},
                    {
                        "id": "i-j-k-l",
                        "name": "Puss in Boots",
                        "species": "Cat",
                        "owner_id": "5-6-7-8",
                    },
                ],
            },
        ],
    },
)

if __name__ == "__main__":
    wsgi()

And if you invoke it:

$ curl 'http://localhost:8080' | jq
{
    "status": "success",
    "error": "",
    "data": [
        {
            "id": "a-b-c-d",
            "name": "Fido",
            "species": "Dog",
            "owner": {
                "id": "1-2-3-4",
                "name": "John Doe"
            }
        },
        {
            "id": "i-j-k-l",
            "name": "Puss in Boots",
            "species": "Cat",
            "owner": {
                "id": "5-6-7-8",
                "name": "Jane Doe"
            }
        },
        {
            "id": "e-f-g-h",
            "name": "Spot",
            "species": "Dog",
            "owner": {
                "id": "1-2-3-4",
                "name": "John Doe"
            }
        }
    ],
    "pagination": {
        "number_results": 3,
        "limit": 50,
        "next_page": {}
    },
    "input_errors": {}
}

silent_on_missing_tables

Optional