ManyToManyIdsWithData

A column to represent a many-to-many relationship with information stored in the relationship itself.

1. Overview
2. related_model_class
3. pivot_model_class
4. own_column_name_in_pivot
5. related_column_name_in_pivot
6. readable_related_columns
7. readable_pivot_column_names
8. setable_column_names
9. persist_unique_lookup_column_to_pivot_table
10. default
11. setable
12. is_readable
13. is_writeable
14. is_temporary
15. validators
16. on_change_pre_save
17. on_change_post_save
18. on_change_save_finished
19. created_by_source_type
20. created_by_source_key
21. created_by_source_strict

Overview

This is an extention of the many-to-many column, but with one important addition: data about the relationship is stored in the pivot table. This creates some differences, which are best explained by example:

import clearskies


class ThingyWidgets(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    # these could also be belongs to relationships, but the pivot model
    # is rarely used directly, so I'm being lazy to avoid having to use
    # model references.
    thingy_id = clearskies.columns.String()
    widget_id = clearskies.columns.String()
    name = clearskies.columns.String()
    kind = clearskies.columns.String()


class Thingy(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String()


class Widget(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String()
    thingy_ids = clearskies.columns.ManyToManyIdsWithData(
        related_model_class=Thingy,
        pivot_model_class=ThingyWidgets,
        readable_pivot_column_names=["id", "thingy_id", "widget_id", "name", "kind"],
    )
    thingies = clearskies.columns.ManyToManyModels("thingy_ids")
    thingy_widgets = clearskies.columns.ManyToManyPivots("thingy_ids")


def my_application(widgets: Widget, thingies: Thingy):
    thing_1 = thingies.create({"name": "Thing 1"})
    thing_2 = thingies.create({"name": "Thing 2"})
    thing_3 = thingies.create({"name": "Thing 3"})
    widget = widgets.create({
        "name": "Widget 1",
        "thingy_ids": [
            {"thingy_id": thing_1.id, "name": "Widget Thing 1", "kind": "Special"},
            {"thingy_id": thing_2.id, "name": "Widget Thing 2", "kind": "Also Special"},
        ],
    })

    return widget


cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        my_application,
        model_class=Widget,
        return_records=True,
        readable_column_names=["id", "name", "thingy_widgets"],
    ),
    classes=[Widget, Thingy, ThingyWidgets],
)

if __name__ == "__main__":
    cli()

As with setting ids in the ManyToManyIds class, any items left out will result in the relationship (including all its related data) being removed. An important difference with the ManyToManyWithData column is the way you specify which record is being connected. This is easy for the ManyToManyIds column because all you provide is the id from the related model. When working with the ManyToManyWithData column, you provide a dictionary for each relationship (so you can provide the data that goes in the pivot model). To let it know what record is being connected, you therefore explicitly provide the id from the related model in a dictionary key with the name of the related model id column in the pivot (e.g. {"thingy_id": id} in the first example. However, if there are unique columns in the related model, you can provide those instead. If you execute the above example you’ll get:

{
    "status": "success",
    "error": "",
    "data": {
        "id": "c4be91a8-85a1-4e29-994a-327f59e26ec7",
        "name": "Widget 1",
        "thingy_widgets": [
            {
                "id": "3a8f6f14-9657-49d8-8844-0db3452525fe",
                "thingy_id": "db292ebc-7b2b-4306-aced-8e6d073ec264",
                "widget_id": "c4be91a8-85a1-4e29-994a-327f59e26ec7",
                "name": "Widget Thing 1",
                "kind": "Special",
            },
            {
                "id": "480a0192-70d9-4363-a669-4a59f0b56730",
                "thingy_id": "d469dbe9-556e-46f3-bc48-03f8cb8d8e44",
                "widget_id": "c4be91a8-85a1-4e29-994a-327f59e26ec7",
                "name": "Widget Thing 2",
                "kind": "Also Special",
            },
        ],
    },
    "pagination": {},
    "input_errors": {},
}

related_model_class

Required

The model class for the model that we are related to.

pivot_model_class

Required

The model class for the pivot table - the table used to record connections between ourselves and our related table.

own_column_name_in_pivot

Optional

The name of the column in the pivot table that contains the id of records from the model with this column.

A default name is created by taking the model class name, converting it to snake case, and then appending _id. If you name your columns according to this standard then you don’t have to specify this column name.

related_column_name_in_pivot

Optional

The name of the column in the pivot table that contains the id of records from the related table.

A default name is created by taking the name of the related model class, converting it to snake case, and then appending _id. If you name your columns according to this standard then you don’t have to specify this column name.

readable_related_columns

Optional

readable_pivot_column_names

Optional

The list of columns in the pivot model that will be included when returning records from an endpoint.

setable_column_names

Optional

The list of columns in the pivot model that can be set when saving data from an endpoint.

persist_unique_lookup_column_to_pivot_table

Optional

Complicated, but probably should be false.

Sometimes you have to provide data from the related model class in your save data so that clearskies can find the right record. Normally, this lookup column is not persisted to the pivot table, because it is assumed to only exist in the related table. In some cases though, you may want it in both, in which case you can set this to true.

default

Optional

A default value to set for this column.

The default is only used when creating a record for the first time, and only if a value for this column has not been set.

import clearskies

class Widget(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String(default="Jane Doe")

cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        lambda widgets: widgets.create(no_data=True),
        model_class=Widget,
        readable_column_names=["id", "name"]
    ),
    classes=[Widget],
)

if __name__ == "__main__":
    cli()

Which when invoked returns:

{
    "status": "success",
    "error": "",
    "data": {
        "id": "03806afa-b189-4729-a43c-9da5aa17bf14",
        "name": "Jane Doe"
    },
    "pagination": {},
    "input_errors": {}
}

setable

Optional

is_readable

Optional

Whether or not this column can be converted to JSON and included in an API response.

If this is set to False for a column and you attempt to set that column as a readable_column in an endpoint, clearskies will throw an exception.

is_writeable

Optional

is_temporary

Optional

Whether or not this column is temporary. A temporary column is not persisted to the backend.

Temporary columns are useful when you want the developer or end user to set a value, but you use that value to trigger additional behavior, rather than actually recording it. Temporary columns often team up with actions or are used to calculate other values. For instance, in our setable example above, we had both an age and a date of birth column, with the date of birth calculated from the age. This obviously results in two columns with similar data. One could be marked as temporary and it will be available during the save operation, but it will be skipped when saving data to the backend:

import clearskies

class Pet(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String()
    date_of_birth = clearskies.columns.Date(is_temporary=True)
    age = clearskies.columns.Integer(
        setable=lambda data, model, now:
            (now-dateparser.parse(model.latest("date_of_birth", data))).total_seconds()/(86400*365),
    )
    created = clearskies.columns.Created()

cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        lambda pets: pets.create({"name": "Spot", "date_of_birth": "2020-05-03"}),
        model_class=Pet,
        readable_column_names=["id", "age", "date_of_birth"],
    ),
    classes=[Pet],
)

if __name__ == "__main__":
    cli()

Which will return:

{
    "status": "success",
    "error": "",
    "data": {
        "id": "ee532cfa-91cf-4747-b798-3c6dcd79326e",
        "age": 5,
        "date_of_birth": null
    },
    "pagination": {},
    "input_errors": {}
}

e.g. the date_of_birth column is empty. To be clear though, it’s not just empty - clearskies made no attempt to set it. If you were using an SQL database, you would not have to put a date_of_birth column in your table.

validators

Optional

on_change_pre_save

Optional

Actions to take during the pre-save step of the save process if the column has changed during the active save operation.

Pre-save happens before the data is persisted to the backend. Actions/callables in this step must return a dictionary. The data in the dictionary will be included in the save operation. Since the save hasn’t completed, any data in the model itself reflects the model before the save operation started. Actions in the pre-save step must NOT make any changes directly, but should ONLY return modified data for the save operation. In addition, they must be idempotent - they should always return the same value when called with the same data. This is because clearskies can call them more than once. If a pre-save hook changes the save data, then clearskies will call all the pre-save hooks again in case this new data needs to trigger further changes. Stateful changes should be reserved for the post_save or save_finished stages.

Callables and actions can request any dependencies provided by the DI system. In addition, they can request two named parameters:

1. model - the model involved in the save operation
2. data - the new data being saved

The key here is that the defined actions will be invoked regardless of how the save happens. Whether the model.save() function is called directly or the model is creatd/modified via an endpoint, your business logic will always be executed. This makes for easy reusability and consistency throughout your application.

Here’s an example where we want to record a timestamp anytime an order status becomes a particular value:

import clearskies

class Order(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    status = clearskies.columns.Select(
        ["Open", "On Hold", "Fulfilled"],
        on_change_pre_save=[
            lambda data, utcnow: {"fulfilled_at": utcnow} if data["status"] == "Fulfilled" else {},
        ],
    )
    fulfilled_at = clearskies.columns.Datetime()

wsgi = clearskies.contexts.WsgiRef(
    clearskies.endpoints.Create(
        model_class=Order,
        writeable_column_names=["status"],
        readable_column_names=["id", "status", "fulfilled_at"],
    ),
)
wsgi()

You can then see the difference depending on what you set the status to:

$ curl http://localhost:8080 -d '{"status":"Open"}' | jq
{
    "status": "success",
    "error": "",
    "data": {
        "id": "a732545f-51b3-4fd0-a6cf-576cf1b2872f",
        "status": "Open",
        "fulfilled_at": null
    },
    "pagination": {},
    "input_errors": {}
}

$ curl http://localhost:8080 -d '{"status":"Fulfilled"}' | jq
{
    "status": "success",
    "error": "",
    "data": {
        "id": "c288bf43-2246-48e4-b168-f40cbf5376df",
        "status": "Fulfilled",
        "fulfilled_at": "2025-05-04T02:32:56+00:00"
    },
    "pagination": {},
    "input_errors": {}
}

on_change_post_save

Optional

Actions to take during the post-save step of the process if the column has changed during the active save.

Post-save happens after the data is persisted to the backend but before the full save process has finished. Since the data has been persisted to the backend, any data returned by the callables/actions will be ignored. If you need to make data changes you’ll have to execute a separate save operation. Since the save hasn’t finished, the model is not yet updated with the new data, and any data you fetch out of the model will refelect the data in the model before the save started.

Callables and actions can request any dependencies provided by the DI system. In addition, they can request three named parameters:

1. model - the model involved in the save operation
2. data - the new data being saved
3. id - the id of the record being saved

Here’s an example of using a post-save action to record a simple audit trail when the order status changes:

import clearskies

class Order(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    status = clearskies.columns.Select(
        ["Open", "On Hold", "Fulfilled"],
        on_change_post_save=[
            lambda model, data, order_histories: order_histories.create({
                "order_id": model.latest("id", data),
                "event": "Order status changed to " + data["status"]
            }),
        ],
    )

class OrderHistory(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    event = clearskies.columns.String()
    order_id = clearskies.columns.BelongsToId(Order)

    # include microseconds in the created_at time so that we can sort our example by created_at
    # and they come out in order (since, for our test program, they will all be created in the same second).
    created_at = clearskies.columns.Created(date_format="%Y-%m-%d %H:%M:%S.%f")

def test_post_save(orders: Order, order_histories: OrderHistory):
    my_order = orders.create({"status": "Open"})
    my_order.status = "On Hold"
    my_order.save()
    my_order.save({"status": "Open"})
    my_order.save({"status": "Fulfilled"})
    return order_histories.where(OrderHistory.order_id.equals(my_order.id)).sort_by("created_at", "asc")

cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        test_post_save,
        model_class=OrderHistory,
        return_records=True,
        readable_column_names=["id", "event", "created_at"],
    ),
    classes=[Order, OrderHistory],
)
cli()

Note that in our on_change_post_save lambda function, we use model.latest("id", data). We can’t just use data["id"] because data is a dictionary containing the information present in the save. During the create operation data["id"] will be populated, but during the subsequent edit operations it won’t be - only the status column is changing. model.latest("id", data) is basically just short hand for: data.get("id", model.id). On the other hand, we can just use data["status"] because the on_change hook is attached to the status field, so it will only fire when status is being changed, which means that the status key is guaranteed to be in the dictionary when the lambda is executed.

Finally, the post-save action has a named parameter called id, so in this specific case we could use:

lambda data, id, order_histories: order_histories.create("order_id": id, "event": data["status"])

When we execute the above script it will return something like:

{
    "status": "success",
    "error": "",
    "data": [
        {
        "id": "c550d714-839b-4f25-a9e1-bd7e977185ff",
        "event": "Order status changed to Open",
        "created_at": "2025-05-04T14:09:42.960119+00:00"
        },
        {
        "id": "f393d7b0-da21-4117-a7a4-0359fab802bb",
        "event": "Order status changed to On Hold",
        "created_at": "2025-05-04T14:09:42.960275+00:00"
        },
        {
        "id": "5b528a10-4a08-47ae-938c-fc7067603f8e",
        "event": "Order status changed to Open",
        "created_at": "2025-05-04T14:09:42.960395+00:00"
        },
        {
        "id": "91f77a88-1c38-49f7-aa1e-7f97bd9f962f",
        "event": "Order status changed to Fulfilled",
        "created_at": "2025-05-04T14:09:42.960514+00:00"
        }
    ],
    "pagination": {},
    "input_errors": {}
}

on_change_save_finished

Optional

Actions to take during the save-finished step of the save process if the column has changed in the save.

Save-finished happens after the save process has completely finished and the model is updated with the final data. Any data returned by these actions will be ignored, since the save has already finished. If you need to make data changes you’ll have to execute a separate save operation.

Callables and actions can request any dependencies provided by the DI system. In addition, they can request the following parameter:

1. model - the model involved in the save operation

Unlike pre_save and post_save, data is not provided because this data has already been merged into the model. If you need some context from the completed save operation, use methods like was_changed and previous_value.

created_by_source_type

Optional

created_by_source_key

Optional

created_by_source_strict

Optional