How-to — Serializer Combinations

---

The Basics

In order to extend the use of serializers, it is possible to use serializers that wrap other serializers. The goal is to customize the behavior of the serialized data without even knowing what is in it.

For example, you could:

• compress the data

• add a checksum to verify data integrity

• encode/encrypt the data

• and much more.

Such a class is called a serializer wrapper. The easynetwork.serializers.wrapper package provides some useful classes to handle these cases.

Use Cases

To illustrate why this concept exists and can be useful to you, let’s take a concrete example of an existing serializer.

Let’s say we have a serializer named… PickleSerializer.

Enable The Stream Context

There is no efficient way to determine the end of a pickle-serialized object without calling pickle.loads() and praying that it works. And since we’re running Python code to recursively deserialize objects, it’s even less safe to potentially deserialize by half.

Therefore, this class does not implement the AbstractIncrementalPacketSerializer class. But if we want to use stream pipes (at random, a TCP stream), how can we do that?

Fortunately, it is possible to encode pickle data in another format: base64. Using an alphabet, we can divide the received packets into lines. And this is exactly what the Base64EncoderSerializer already does:

def main() -> None:
    from easynetwork.serializers.wrapper import Base64EncoderSerializer

    # Use of Base64EncoderSerializer as an incremental serializer wrapper
    # Each line (by default delimited with \r\n) is a base64 encoded pickle object.
    serializer = Base64EncoderSerializer(PickleSerializer())
    protocol = StreamProtocol(serializer)

    with TCPNetworkClient(("remote_address", 12345), protocol) as endpoint:
        # Sends b'gASVDQAAAAAAAAB9lIwEZGF0YZRLKnMu\r\n' over the TCP socket.
        endpoint.send_packet({"data": 42})

        ...

Ensure Data Integrity

The pickle protocol is incredibly handy, but not very secure. Aside from the fact that malicious data can be processed, it can also be corrupted in transit, which is even more problematic since pickle relies on data to execute code, not the other way around.

To overcome this, we can tell the serializer to add an extra layer of verification:

def main() -> None:
    from easynetwork.serializers.wrapper import Base64EncoderSerializer

    # Add checksum parameter to append a unique hash to verify data integrity
    serializer = Base64EncoderSerializer(PickleSerializer(), checksum=True)
    protocol = StreamProtocol(serializer)

    with TCPNetworkClient(("remote_address", 12345), protocol) as endpoint:
        endpoint.send_packet({"data": 42})

        ...

Use Less Bandwidth

Let’s say {"data": 42} produces a BIG packet (e.g., 1Mb size).

This is often the case with large projects that handle a lot of data. It’s common practice to reduce the size of packets sent as much as possible to reduce bandwidth usage and network traffic.

The first thing to do would be to reduce the pickle data itself:

def main() -> None:
    from easynetwork.serializers.wrapper import Base64EncoderSerializer

    # Reduce pickle size by adding pickler_optimize=True
    serializer = Base64EncoderSerializer(PickleSerializer(pickler_optimize=True), checksum=True)
    protocol = StreamProtocol(serializer)

    with TCPNetworkClient(("remote_address", 12345), protocol) as endpoint:
        endpoint.send_packet({"data": 42})

        ...

This will do the trick, but it’s clearly not enough. So, last option: compress the data using zlib.

def main() -> None:
    from zlib import Z_BEST_COMPRESSION

    from easynetwork.serializers.wrapper import Base64EncoderSerializer, ZlibCompressorSerializer

    # Add ZlibCompressorSerializer to compress pickle output
    serializer = Base64EncoderSerializer(
        ZlibCompressorSerializer(PickleSerializer(pickler_optimize=True), compress_level=Z_BEST_COMPRESSION),
        checksum=True,
    )
    protocol = StreamProtocol(serializer)

    with TCPNetworkClient(("remote_address", 12345), protocol) as endpoint:
        endpoint.send_packet({"data": 42})

        ...

Deserialize Trusted Data

Adding a checksum to verify the data is all well and good, but it’s not enough in terms of security if there’s no authentication protocol to ensure the provenance of the data, and even less so when it comes to pickle.

The last thing to do would be to add a signature with a key known only to the two communicators.

def main() -> None:
    from zlib import Z_BEST_COMPRESSION

    from easynetwork.serializers.wrapper import Base64EncoderSerializer, ZlibCompressorSerializer

    # Sign the checksum hash with a key known only to both sides of the pipe to verify the origin of the data.
    verification_key = b"very secret key"
    serializer = Base64EncoderSerializer(
        ZlibCompressorSerializer(PickleSerializer(pickler_optimize=True), compress_level=Z_BEST_COMPRESSION),
        checksum=verification_key,
    )
    protocol = StreamProtocol(serializer)

    with TCPNetworkClient(("remote_address", 12345), protocol) as endpoint:
        endpoint.send_packet({"data": 42})

        ...

Now we use this StreamProtocol for both server and client, and we can have secure data transfer.