At SumUp I work on the point-of-sale app, written in Kotlin Multiplatform. A POS exchanges a lot of messages, and with Protobuf the code on the other side of the wire is not necessarily Kotlin: it can be any language, with classes generated by protoc from .proto files.

So the requirement is simple to state: a message encoded in Kotlin must be decoded exactly the same way by protoc-generated code, and vice versa. Not “mostly the same”. Exactly.

That’s where things got interesting, and why I ended up writing k2pb on my own time.

The obvious choice: kotlinx.serialization

kotlinx-serialization-protobuf is the natural candidate in KMP. Annotate with @Serializable, pick the ProtoBuf format, done. It’s a great library, but it’s Kotlin-first and has to support many formats (JSON, CBOR, Protobuf…). Some of its choices make sense for Kotlin and are subtly incompatible with the Protobuf spec. In a system where Kotlin and protoc share messages, subtle means bugs in production.

Two examples.

Default values

Kotlin lets you declare a default value:

@Serializable
data class MyMessage(val myInt: Int = 42)

The equivalent Protobuf message:

syntax = "proto3";

message MyMessage {
  int32 myInt = 1;
}

Protobuf has its own defaults, the same for everyone: 0 for an int32, "" for a string, the first entry for an enum. And the spec is explicit: a field set to its default value is not written on the wire.

Now decode an empty message:

kotlinx.serialization gives you myInt = 42 (the Kotlin default)
protoc-generated Java gives you myInt = 0 (the Protobuf default)

Same bytes, two different values. And encoding is not better. kotlinx.serialization lets you pick encodeDefaults:

encodeDefaults = false: MyMessage() is encoded as an empty message, because 42 is the Kotlin default. The Java side reads 0. Two interpretations of the same message.
encodeDefaults = true: everything is written, both sides agree… but you’re sending bytes Protobuf was designed not to send.

There’s an open issue about it.

Nullability

Protobuf has no null. A scalar field is either set, or unset and then reads as its default value. In Kotlin, "" and null are two different things, and we rely on that difference all the time.

The classic trap is schema evolution. You add a new nullable enum to a message. Old messages don’t contain it:

kotlinx.serialization has no notion of null in Protobuf, so you give the field a Kotlin default (= null) and get null.
protoc code reads the same old message and gets… the first enum entry.

Again: same bytes, two different values.

What k2pb does differently

k2pb (Kotlin to ProtoBuf) starts from your Kotlin (or KMP) models and uses KSP to generate, at compile time:

the serializers (one codec per class) for runtime encoding/decoding, in common code,
the .proto files, so any other language can talk to you with protoc.

The models are the source of truth. The .proto files are an output you publish to other teams.

Setup

plugins {
    kotlin("multiplatform")
    id("com.google.devtools.ksp")
    id("com.glureau.k2pb") version "0.9.35"
}

Annotate your classes:

@ProtoMessage
data class MyDataClass(val myInt: Int)

Each module gets a generated register<Module>Codecs() function. You pick the modules you need when creating the serializer:

val serializer = K2PB {
    registerSampleLibCodecs()
    registerSampleAppCodecs()
}

val bytes = serializer.encodeToByteArray(MyDataClass(42))
val decoded = serializer.decodeFromByteArray<MyDataClass>(bytes)

Defaults are Protobuf defaults

k2pb follows the spec: a field equal to its Protobuf default is not encoded, and a missing field decodes to its Protobuf default. A Kotlin default value is a convenience for your constructors, not a wire-format decision. Kotlin and protoc always agree.

Nullability is explicit

For each nullable field, k2pb adds a small companion field to the message, so that “null” survives the trip:

@ProtoMessage
data class Order(val note: String?)

message Order {
  string note = 1;
  com.glureau.k2pb.ExplicitNullability isNoteNull = 2;
}

ExplicitNullability is an enum with three values: UNKNOWN (the field wasn’t there, typically an older message), NULL and NOT_NULL. Kotlin users never see it. protoc users get an explicit accessor and a documented rule. And because UNKNOWN exists, k2pb can let you decide what an old message means when you add a field. That deserves its own article.

Readable output

Since KSP runs at compile time, k2pb sees things a runtime schema generator can’t:

your KDoc is copied into the .proto files, so the teams consuming them get the documentation for free,
value classes are encoded as their underlying value, not as a useless wrapper message,
nested classes are supported,
sealed classes become oneof,
custom converters (for BigDecimal, Uuid, dates…) are declared on the field:

class BigDecimalConverter : NullableStringConverter<BigDecimal> {
    override fun encode(value: BigDecimal): String? = value.toPlainString()
    override fun decode(data: String?): BigDecimal? = data?.takeIf { it.isNotBlank() }?.toBigDecimal()
}

@ProtoMessage
data class Price(@ProtoField(converter = BigDecimalConverter::class) val amount: BigDecimal)

And the generated codecs are plain Kotlin you can read and debug. No reflection, no magic.

How I know it’s compatible

Compatibility is the whole point, so it’s what the tests check:

the generated .proto files and codecs are covered by approval tests: any change in the output shows up in the diff,
the generated .proto files are compiled with protoc, and each test builds the same message twice, once in Kotlin and once with the protoc-generated Java classes, then checks that:
- both produce the exact same bytes,
- k2pb decodes the Java bytes into the original Kotlin instance,
- Java decodes the k2pb bytes into the original Java instance.

inline fun <reified Kt : Any> assertCompatibleSerialization(ktInstance: Kt, protocInstance: GeneratedMessage) {
    val encodedViaProtoc = protocInstance.toByteArray()
    val encodedViaK2PB = serializer.encodeToByteArray<Kt>(ktInstance)
    assertContentEquals(expected = encodedViaK2PB, actual = encodedViaProtoc)

    assertEquals(ktInstance, serializer.decodeFromByteArray<Kt>(encodedViaProtoc))
    assertEquals(protocInstance, protocInstance.parserForType.parseFrom(encodedViaK2PB))
}

Byte-for-byte equality is stricter than “it decodes fine”: it also catches the bandwidth waste of encoding defaults.

In production

k2pb started as a side project, written on my own time. It now handles the whole serialization layer of the SumUp point of sale, a Kotlin Multiplatform app where a message decoded differently on each side of the wire is not an option.

That’s the best test suite I could ask for.

Status & what’s next

k2pb is still young (0.9.x). Proto3 only, no proto2. The next big topic is checking schema upgrades: compare the newly generated .proto files with the published ones and fail the build if a change is not backward compatible.

If you use Protobuf in a KMP project, give it a try and open an issue with whatever you run into.