ManyToManyIds

A column that represents a many-to-many relationship.

1. Overview
2. related_model_class
3. pivot_model_class
4. own_column_name_in_pivot
5. related_column_name_in_pivot
6. readable_related_column_names
7. default
8. setable
9. is_readable
10. is_writeable
11. is_temporary
12. validators
13. on_change_pre_save
14. on_change_post_save
15. on_change_save_finished
16. created_by_source_type
17. created_by_source_key
18. created_by_source_strict

Overview

This is different from belongs to/has many because with those, every child has only one parent. With a many-to-many relationship, both models can have multiple relatives from the other model class. In order to support this, it’s necessary to have a third model (the pivot model) that records the relationships. In general this table just needs three columns: it’s own id, and then one column for each other model to store the id of the related records. You can specify the names of these columns but it also follows the standard naming convention by default: take the class name, convert it to snake case, and append _id.

Note, there is a variation on this (ManyToManyIdsWithData) where additional data is stored in the pivot table to record information about the relationship.

This column is writeable. You would set it to a list of ids from the related model that denotes which records it is related to.

The following example shows usage. Normally the many-to-many column exists for both related models, but in this specific example it only exists for one of the models. This is done so that the example can fit in a single file and therefore be easy to demonstrate. In order to have both models reference eachother, you have to use model references to avoid circular imports. There are examples of doing this in the BelongsTo column class.

import clearskies


class ThingyToWidget(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    # these could also be belongs to relationships, but the pivot model
    # is rarely used directly, so I'm being lazy to avoid having to use
    # model references.
    thingy_id = clearskies.columns.String()
    widget_id = clearskies.columns.String()


class Thingy(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String()


class Widget(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String()
    thingy_ids = clearskies.columns.ManyToManyIds(
        related_model_class=Thingy,
        pivot_model_class=ThingyToWidget,
    )
    thingies = clearskies.columns.ManyToManyModels("thingy_ids")


def my_application(widgets: Widget, thingies: Thingy):
    thing_1 = thingies.create({"name": "Thing 1"})
    thing_2 = thingies.create({"name": "Thing 2"})
    thing_3 = thingies.create({"name": "Thing 3"})
    widget = widgets.create({
        "name": "Widget 1",
        "thingy_ids": [thing_1.id, thing_2.id],
    })

    # remove an item by saving without it's id in place
    widget.save({"thingy_ids": [thing.id for thing in widget.thingies if thing.id != thing_1.id]})

    # add an item by saving and adding the new id
    widget.save({"thingy_ids": [*widget.thingy_ids, thing_3.id]})

    return widget.thingies


cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        my_application,
        model_class=Thingy,
        return_records=True,
        readable_column_names=["id", "name"],
    ),
    classes=[Widget, Thingy, ThingyToWidget],
)

if __name__ == "__main__":
    cli()

And when executed:

{
    "status": "success",
    "error": "",
    "data": [
        {"id": "741bc838-c694-4624-9fc2-e9032f6cb962", "name": "Thing 2"},
        {"id": "1808a8ef-e288-44e6-9fed-46e3b0df057f", "name": "Thing 3"},
    ],
    "pagination": {},
    "input_errors": {},
}

Of course, you can also create or remove individual relationships by using the pivot model directly, as shown in these partial code snippets:

def add_items(thingy_to_widgets):
    thingy_to_widgets.create({
        "thingy_id": "some_id",
        "widget_id": "other_id",
    })


def remove_item(thingy_to_widgets):
    thingy_to_widgets.where("thingy_id=some_id").where("widget_id=other_id").first().delete()

related_model_class

Required

The model class for the model that we are related to.

pivot_model_class

Required

The model class for the pivot table - the table used to record connections between ourselves and our related table.

own_column_name_in_pivot

Optional

The name of the column in the pivot table that contains the id of records from the model with this column.

A default name is created by taking the model class name, converting it to snake case, and then appending _id. If you name your columns according to this standard then you don’t have to specify this column name.

related_column_name_in_pivot

Optional

The name of the column in the pivot table that contains the id of records from the related table.

A default name is created by taking the name of the related model class, converting it to snake case, and then appending _id. If you name your columns according to this standard then you don’t have to specify this column name.

readable_related_column_names

Optional

The list of columns to be loaded from the related models when we are converted to JSON.

default

Optional

A default value to set for this column.

The default is only used when creating a record for the first time, and only if a value for this column has not been set.

import clearskies

class Widget(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String(default="Jane Doe")

cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        lambda widgets: widgets.create(no_data=True),
        model_class=Widget,
        readable_column_names=["id", "name"]
    ),
    classes=[Widget],
)

if __name__ == "__main__":
    cli()

Which when invoked returns:

{
    "status": "success",
    "error": "",
    "data": {
        "id": "03806afa-b189-4729-a43c-9da5aa17bf14",
        "name": "Jane Doe"
    },
    "pagination": {},
    "input_errors": {}
}

setable

Optional

is_readable

Optional

Whether or not this column can be converted to JSON and included in an API response.

If this is set to False for a column and you attempt to set that column as a readable_column in an endpoint, clearskies will throw an exception.

is_writeable

Optional

is_temporary

Optional

Whether or not this column is temporary. A temporary column is not persisted to the backend.

Temporary columns are useful when you want the developer or end user to set a value, but you use that value to trigger additional behavior, rather than actually recording it. Temporary columns often team up with actions or are used to calculate other values. For instance, in our setable example above, we had both an age and a date of birth column, with the date of birth calculated from the age. This obviously results in two columns with similar data. One could be marked as temporary and it will be available during the save operation, but it will be skipped when saving data to the backend:

import clearskies

class Pet(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    name = clearskies.columns.String()
    date_of_birth = clearskies.columns.Date(is_temporary=True)
    age = clearskies.columns.Integer(
        setable=lambda data, model, now:
            (now-dateparser.parse(model.latest("date_of_birth", data))).total_seconds()/(86400*365),
    )
    created = clearskies.columns.Created()

cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        lambda pets: pets.create({"name": "Spot", "date_of_birth": "2020-05-03"}),
        model_class=Pet,
        readable_column_names=["id", "age", "date_of_birth"],
    ),
    classes=[Pet],
)

if __name__ == "__main__":
    cli()

Which will return:

{
    "status": "success",
    "error": "",
    "data": {
        "id": "ee532cfa-91cf-4747-b798-3c6dcd79326e",
        "age": 5,
        "date_of_birth": null
    },
    "pagination": {},
    "input_errors": {}
}

e.g. the date_of_birth column is empty. To be clear though, it’s not just empty - clearskies made no attempt to set it. If you were using an SQL database, you would not have to put a date_of_birth column in your table.

validators

Optional

on_change_pre_save

Optional

Actions to take during the pre-save step of the save process if the column has changed during the active save operation.

Pre-save happens before the data is persisted to the backend. Actions/callables in this step must return a dictionary. The data in the dictionary will be included in the save operation. Since the save hasn’t completed, any data in the model itself reflects the model before the save operation started. Actions in the pre-save step must NOT make any changes directly, but should ONLY return modified data for the save operation. In addition, they must be idempotent - they should always return the same value when called with the same data. This is because clearskies can call them more than once. If a pre-save hook changes the save data, then clearskies will call all the pre-save hooks again in case this new data needs to trigger further changes. Stateful changes should be reserved for the post_save or save_finished stages.

Callables and actions can request any dependencies provided by the DI system. In addition, they can request two named parameters:

1. model - the model involved in the save operation
2. data - the new data being saved

The key here is that the defined actions will be invoked regardless of how the save happens. Whether the model.save() function is called directly or the model is creatd/modified via an endpoint, your business logic will always be executed. This makes for easy reusability and consistency throughout your application.

Here’s an example where we want to record a timestamp anytime an order status becomes a particular value:

import clearskies

class Order(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    status = clearskies.columns.Select(
        ["Open", "On Hold", "Fulfilled"],
        on_change_pre_save=[
            lambda data, utcnow: {"fulfilled_at": utcnow} if data["status"] == "Fulfilled" else {},
        ],
    )
    fulfilled_at = clearskies.columns.Datetime()

wsgi = clearskies.contexts.WsgiRef(
    clearskies.endpoints.Create(
        model_class=Order,
        writeable_column_names=["status"],
        readable_column_names=["id", "status", "fulfilled_at"],
    ),
)
wsgi()

You can then see the difference depending on what you set the status to:

$ curl http://localhost:8080 -d '{"status":"Open"}' | jq
{
    "status": "success",
    "error": "",
    "data": {
        "id": "a732545f-51b3-4fd0-a6cf-576cf1b2872f",
        "status": "Open",
        "fulfilled_at": null
    },
    "pagination": {},
    "input_errors": {}
}

$ curl http://localhost:8080 -d '{"status":"Fulfilled"}' | jq
{
    "status": "success",
    "error": "",
    "data": {
        "id": "c288bf43-2246-48e4-b168-f40cbf5376df",
        "status": "Fulfilled",
        "fulfilled_at": "2025-05-04T02:32:56+00:00"
    },
    "pagination": {},
    "input_errors": {}
}

on_change_post_save

Optional

Actions to take during the post-save step of the process if the column has changed during the active save.

Post-save happens after the data is persisted to the backend but before the full save process has finished. Since the data has been persisted to the backend, any data returned by the callables/actions will be ignored. If you need to make data changes you’ll have to execute a separate save operation. Since the save hasn’t finished, the model is not yet updated with the new data, and any data you fetch out of the model will refelect the data in the model before the save started.

Callables and actions can request any dependencies provided by the DI system. In addition, they can request three named parameters:

1. model - the model involved in the save operation
2. data - the new data being saved
3. id - the id of the record being saved

Here’s an example of using a post-save action to record a simple audit trail when the order status changes:

import clearskies

class Order(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    status = clearskies.columns.Select(
        ["Open", "On Hold", "Fulfilled"],
        on_change_post_save=[
            lambda model, data, order_histories: order_histories.create({
                "order_id": model.latest("id", data),
                "event": "Order status changed to " + data["status"]
            }),
        ],
    )

class OrderHistory(clearskies.Model):
    id_column_name = "id"
    backend = clearskies.backends.MemoryBackend()

    id = clearskies.columns.Uuid()
    event = clearskies.columns.String()
    order_id = clearskies.columns.BelongsToId(Order)

    # include microseconds in the created_at time so that we can sort our example by created_at
    # and they come out in order (since, for our test program, they will all be created in the same second).
    created_at = clearskies.columns.Created(date_format="%Y-%m-%d %H:%M:%S.%f")

def test_post_save(orders: Order, order_histories: OrderHistory):
    my_order = orders.create({"status": "Open"})
    my_order.status = "On Hold"
    my_order.save()
    my_order.save({"status": "Open"})
    my_order.save({"status": "Fulfilled"})
    return order_histories.where(OrderHistory.order_id.equals(my_order.id)).sort_by("created_at", "asc")

cli = clearskies.contexts.Cli(
    clearskies.endpoints.Callable(
        test_post_save,
        model_class=OrderHistory,
        return_records=True,
        readable_column_names=["id", "event", "created_at"],
    ),
    classes=[Order, OrderHistory],
)
cli()

Note that in our on_change_post_save lambda function, we use model.latest("id", data). We can’t just use data["id"] because data is a dictionary containing the information present in the save. During the create operation data["id"] will be populated, but during the subsequent edit operations it won’t be - only the status column is changing. model.latest("id", data) is basically just short hand for: data.get("id", model.id). On the other hand, we can just use data["status"] because the on_change hook is attached to the status field, so it will only fire when status is being changed, which means that the status key is guaranteed to be in the dictionary when the lambda is executed.

Finally, the post-save action has a named parameter called id, so in this specific case we could use:

lambda data, id, order_histories: order_histories.create("order_id": id, "event": data["status"])

When we execute the above script it will return something like:

{
    "status": "success",
    "error": "",
    "data": [
        {
        "id": "c550d714-839b-4f25-a9e1-bd7e977185ff",
        "event": "Order status changed to Open",
        "created_at": "2025-05-04T14:09:42.960119+00:00"
        },
        {
        "id": "f393d7b0-da21-4117-a7a4-0359fab802bb",
        "event": "Order status changed to On Hold",
        "created_at": "2025-05-04T14:09:42.960275+00:00"
        },
        {
        "id": "5b528a10-4a08-47ae-938c-fc7067603f8e",
        "event": "Order status changed to Open",
        "created_at": "2025-05-04T14:09:42.960395+00:00"
        },
        {
        "id": "91f77a88-1c38-49f7-aa1e-7f97bd9f962f",
        "event": "Order status changed to Fulfilled",
        "created_at": "2025-05-04T14:09:42.960514+00:00"
        }
    ],
    "pagination": {},
    "input_errors": {}
}

on_change_save_finished

Optional

Actions to take during the save-finished step of the save process if the column has changed in the save.

Save-finished happens after the save process has completely finished and the model is updated with the final data. Any data returned by these actions will be ignored, since the save has already finished. If you need to make data changes you’ll have to execute a separate save operation.

Callables and actions can request any dependencies provided by the DI system. In addition, they can request the following parameter:

1. model - the model involved in the save operation

Unlike pre_save and post_save, data is not provided because this data has already been merged into the model. If you need some context from the completed save operation, use methods like was_changed and previous_value.

created_by_source_type

Optional

created_by_source_key

Optional

created_by_source_strict

Optional