**Access date for every source below: 2026-08-09.** (All fetches performed 2026-08-09.)
Two extraction paths were used, and they do not have the same fidelity. Every quote is marked:
Anything I could not find is written as **NOT FOUND** with the searches performed. There are eleven such gaps and they are real gaps, not padding.
**Answer: no untagged unions. Protobuf's only union is `oneof`, and its variant is identified by the field number on the wire, never by which fields are present.**
Protobuf has no construct where the reader infers the variant from the shape of the payload. A `oneof` is encoded as ordinary fields with ordinary tags; the runtime records which tag arrived last. The evolution rules confirm the union is tag-identified, because moving a *pre-existing* field into a `oneof` is unsafe (its tag is already in flight and old readers do not know it participates in a discriminated group):
"Moving fields into an existing `oneof` is not safe." [WF] — https://protobuf.dev/programming-guides/proto3/
"Changing a single explicit presence field or extension into a member of a **new** `oneof` is safe." [WF] — https://protobuf.dev/programming-guides/proto3/
"Changing a `oneof` which contains only one field to an explicit presence field is safe." [WF] — https://protobuf.dev/programming-guides/proto3/
The known evolution hazard of `oneof` is that its "unset" state is **indistinguishable from "set to a variant I do not know"**, because the unknown variant's tag lands in the unknown-field set rather than in the oneof:
"If checking the value of a oneof returns `None`/`NOT_SET`, it could mean that the oneof has not been set or it has been set to a field in a different version of the oneof." [WF] — https://yokota.blog/2021/08/26/understanding-protobuf-compatibility/
"removing a field from a oneof is considered a backward incompatible change. Likewise, adding a field to a oneof is considered a forward incompatible change" [WF] — https://yokota.blog/2021/08/26/understanding-protobuf-compatibility/
**Stated rationale for not having untagged unions: NOT FOUND.** Searched protobuf.dev proto3/proto2/editions guides, dos-donts, and the editions design docs. Protobuf never had untagged unions to reject, so no rejection rationale exists. The nearest thing to a rationale is Cap'n Proto's and JTD's, in §1.4 and §1.5.
**Answer: NO. proto3 cannot distinguish an empty repeated field from an absent one, and this was never fixed — the 3.15 presence restoration covers singular scalars only, not `repeated` or `map`.**
"When serializing, fields with _implicit presence_ are not serialized if they contain their default value." [WF] — https://protobuf.dev/programming-guides/field_presence/
The JSON mapping makes the collapse explicit and names the empty-list case directly:
"If the field doesn't support field presence and has the default value (for example any empty repeated field) serializers should omit it from the output." [WF] — https://protobuf.dev/programming-guides/json/
The presence document is candid that the collapsed state is genuinely three-ways ambiguous under implicit presence:
"The default value may mean: the field was explicitly set to its default value, which is valid in the application-specific domain of values; the field was notionally 'cleared' by setting its default; or the field was never set." [WF] — https://protobuf.dev/programming-guides/field_presence/
There is a JSON-only escape hatch: JSON has `null`, which the wire format does not, and the presence doc flags the mismatch:
"JSON may include fields that are 'not present,' unlike the _implicit presence_ discipline for other formats: JSON defines a `null` value, which may be used to represent a _defined but not-present field_." [WF] — https://protobuf.dev/programming-guides/field_presence/
But ProtoJSON deliberately throws that information away on parse — `null` is normalised to "unset":
"Parsers accept `null` as a legal value for any field" where "The field should remain unset, as though it was not present in the input at all." [WF] — https://protobuf.dev/programming-guides/json/
**Was it ever changed?** Yes for singular scalars (see §3, Reversal R2). No for repeated/map — those remain no-presence in all of proto2, proto3, and editions. The `optional` label cannot be applied to a repeated field.
**Answer: proto3 and editions use OPEN enums; proto2 used CLOSED. The switch was made specifically because closed enums misbehave.** This is the second-clearest reversal in the whole corpus (see §3, R4).
"Open enums will parse the value `2` and store it directly in the field. Accessor will report the field as being _set_ and will return something that represents `2`." [WF] — https://protobuf.dev/programming-guides/enum/
"Closed enums will parse the value `2` and store it in the message's unknown field set. Accessors will report the field as being _unset_ and will return the enum's default value." [WF] — https://protobuf.dev/programming-guides/enum/
"Prior to the introduction of `syntax = \"proto3\"` all enums were _closed_." [WF] — https://protobuf.dev/programming-guides/enum/
The rationale sentence — this is the money quote:
"Proto3 and editions use _open_ enums specifically because of the unexpected behavior that _closed_ enums cause." [WF] — https://protobuf.dev/programming-guides/enum/
**The documented incident class** (closed enums silently reorder repeated fields):
"When a `repeated Enum` field is parsed, all unknown values will be placed in the unknown field set. When it is serialized those unknown values will be written again, _but not in their original place in the list_." [WF] — https://protobuf.dev/programming-guides/enum/
"A wire format containing the values `[0, 2, 1, 2]` for field 1 will parse so that the repeated field contains `[0, 1]` and the value `[2, 2]` will end up stored as an unknown field. After reserializing the message, the wire format will correspond to `[0, 1, 2, 2]`." [WF] — https://protobuf.dev/programming-guides/enum/
"Maps with _closed_ enums for their value will place entire entries (key and value) in the unknown fields whenever the value is unknown." [WF] — https://protobuf.dev/programming-guides/enum/
**Second documented incident — closed enums interact catastrophically with `required`:**
"A second issue with required fields appears when someone adds a value to an enum. In this case, the unrecognized enum value is treated as if it were missing, which also causes the required value check to fail." [WF] — https://protobuf.dev/programming-guides/proto2/
**The UNSPECIFIED/UNKNOWN sentinel convention:**
"In proto3, the first value defined in an enum definition **must** have the value zero and should have the name `ENUM_TYPE_NAME_UNSPECIFIED` or `ENUM_TYPE_NAME_UNKNOWN`." [WF] — https://protobuf.dev/programming-guides/proto3/
"Enums should include a default `FOO_UNSPECIFIED` value as the first value in the declaration." [WF] — https://protobuf.dev/best-practices/dos-donts/
"the first declared enum value should be a default `FOO_UNSPECIFIED` value and should use tag 0." [WF] — https://protobuf.dev/best-practices/dos-donts/
Editions makes the open/closed choice a per-file/per-type *feature* rather than a syntax-wide law:
"In edition 2023, the first value defined in an enum definition **must** have the value zero and should have the name `ENUM_TYPE_NAME_UNSPECIFIED` or `ENUM_TYPE_NAME_UNKNOWN`." [WF] — https://protobuf.dev/programming-guides/editions/
"If an enum type has been migrated from proto2 using `option features.enum_type = CLOSED;` there is no restriction on the first value in the enum." [WF] — https://protobuf.dev/programming-guides/editions/
And **adding enum values is a wire-safe change**:
"Adding additional values to an enum is safe." [WF] — https://protobuf.dev/programming-guides/proto3/
**Answer: protobuf is asymmetric BY ENCODING, not by role — the binary codec is tolerant-and-preserving, the JSON codec is strict-and-lossy. This is the single most important finding for a JSON-in-a-repo format.**
Binary:
"Proto3 messages preserve unknown fields and include them during parsing and in the serialized output, which matches proto2 behavior." [WF] — https://protobuf.dev/programming-guides/proto3/
JSON — the opposite default:
"The protobuf JSON parser should reject unknown fields by default but may provide an option to ignore unknown fields in parsing." [WF] — https://protobuf.dev/programming-guides/json/
So the same schema, encoded in the two blessed encodings, gives you opposite unknown-field policies. Protobuf's own advice is to prefer binary precisely to keep the tolerant behaviour:
"Use binary; avoid using text formats for data exchange." [WF] — https://protobuf.dev/programming-guides/proto3/
**"Be liberal in what you accept" and its modern critique.** Protobuf does not cite Postel. The authoritative modern critique is IETF RFC 9413 (*Maintaining Robust Protocols*):
"Be strict when sending and tolerant when receiving. Implementations must follow specifications precisely when sending to the network, and tolerate faulty input from the network." [WF, §2] — https://www.rfc-editor.org/rfc/rfc9413.html
"An implementation that reacts to variations in the manner recommended in the robustness principle enters a pathological feedback cycle. Over time: Implementations progressively add logic to constrain how data is transmitted or to permit variations in what is received." [WF, §4.1] — https://www.rfc-editor.org/rfc/rfc9413.html
"A flaw can become entrenched as a de facto standard. Any implementation of the protocol is required to replicate the aberrant behavior, or it is not interoperable." [WF, §4.1] — https://www.rfc-editor.org/rfc/rfc9413.html
"If non-compliance is tolerated by existing implementations, non-compliant implementations can be deployed successfully. Newer implementations then have a strong incentive to tolerate any existing non-compliance in order to be successfully deployed." [WF, §4.2] — https://www.rfc-editor.org/rfc/rfc9413.html
"Choosing to generate fatal errors for unspecified conditions instead of attempting error recovery can ensure that faults receive attention." [WF, §5.1] — https://www.rfc-editor.org/rfc/rfc9413.html
**Crucial distinction the RFC forces you to make** (and which the naive reading of "tolerant reader" blurs): tolerating *unknown extension points that the spec declared extensible* is not the same as tolerating *malformed or non-conformant data*. Protobuf's unknown-field preservation is the former. RFC 9413 attacks the latter.
**Answer: protobuf has NO schema version in the data. Evolution is entirely field-level. Editions version the LANGUAGE, not the message.**
There is no version number anywhere on the wire, no schema id, no `$schema`. Compatibility is a property of each field's tag/type/label history, which is why the entire evolution surface is a list of per-field rules ("Updating A Message Type"). The `edition = "2024"` marker is a compiler directive about which language defaults apply to a `.proto` file; it never appears in a serialized message.
"The last radical change to Protobuf (`syntax = \"proto3\";`) split the ecosystem." [WF] — https://raw.githubusercontent.com/protocolbuffers/protobuf/main/docs/design/editions/what-are-protobuf-editions.md
"Protobuf Editions replace the proto2 and proto3 designations" [WF] — https://protobuf.dev/news/2023-06-29/
"Instead of the hardcoded behaviors in older versions, editions will represent a collection of 'features'" [WF] — https://protobuf.dev/news/2023-06-29/
**Why field-level rather than versioned:** NOT FOUND as an explicit design statement. Inferable from the Cap'n Proto FAQ's message-bus story (§1.6 / §3 R1): a version-gated format requires every intermediary to know the version, which is exactly the coupling protobuf's design avoids. But I found no protobuf document that says this in so many words.
**Answer: numeric tags are the identity. Names are decoration in binary and identity in JSON — a split that matters enormously for a JSON-at-rest format.**
"Never re-use a tag number. It messes up deserialization. Even if you think no one is using the field, don't re-use a tag number." [WF] — https://protobuf.dev/best-practices/dos-donts/
"When you delete a field that's no longer used, reserve its tag number so that no one accidentally re-uses it in the future." [WF] — https://protobuf.dev/best-practices/dos-donts/
"Changing field numbers for any existing field is not safe." [WF] — https://protobuf.dev/programming-guides/proto3/
**What breaks when the policy is violated** — this list is the best "consequences" enumeration in the whole corpus, because it names data-corruption and PII leakage, not just parse errors:
"Reusing a field number makes decoding wire-format messages ambiguous." [WF] — https://protobuf.dev/programming-guides/proto3/
"Encoding a field using one definition and then decoding that same field with a different definition can lead to: Developer time lost to debugging; A parse/merge error (best case scenario); Leaked PII/SPII; Data corruption" [WF] — https://protobuf.dev/programming-guides/proto3/
"If you [update] a message type by entirely deleting a field, or commenting it out, future developers can reuse the field number when making their own updates to the type. This can cause severe issues" [WF] — https://protobuf.dev/programming-guides/proto2/
**`reserved` has two halves with different force:**
"If you [update] a message type by entirely deleting a field...you **must** [reserve the deleted field number]." [WF] — https://protobuf.dev/programming-guides/proto3/
"Reserved field names affect only the protoc compiler behavior and not runtime behavior." [WF] — https://protobuf.dev/programming-guides/proto3/
That last sentence is the key asymmetry: reserving a **number** protects the *data*; reserving a **name** protects only the *build*. In a JSON format where names are the identity, only the weaker half exists on the wire — reserving a name is your only tool and it has no runtime force at all unless you build one.
The name-fragility of ProtoJSON is stated directly in the ecosystem literature:
"ProtoJSON format does not support unknown fields, and it puts field and enum value names into encoded messages which makes it much harder to change those names later." [WF] — via https://protobuf.dev/programming-guides/json/ discussion, surfaced in search; see also §6 re-fetch note
Other identity-adjacent don'ts:
"Almost never change the type of a field; it'll mess up deserialization, same as re-using a tag number." [WF] — https://protobuf.dev/best-practices/dos-donts/
"Almost never change the default value of a proto field. This causes version skew between clients and servers." [WF] — https://protobuf.dev/best-practices/dos-donts/
"Although it won't cause crashes, you'll lose data." (on repeated→scalar) [WF] — https://protobuf.dev/best-practices/dos-donts/
**Answer: binary preserves (since 3.5, after a removal-and-restoration — see §3 R1); JSON does not; and several innocent-looking operations silently destroy preservation even in binary.**
"Proto3 messages preserve unknown fields and include them during parsing and in the serialized output, which matches proto2 behavior." [WF] — https://protobuf.dev/programming-guides/proto3/
"Some actions can cause unknown fields to be lost. For example, if you do one of the following, unknown fields are lost: Serialize a proto to JSON. Iterate over all of the fields in a message to populate a new message." [WF] — https://protobuf.dev/programming-guides/proto3/
"Use message-oriented APIs, such as `CopyFrom()` and `MergeFrom()`, to copy data rather than copying field-by-field" [WF] — https://protobuf.dev/programming-guides/proto3/
**What concretely goes wrong when a reader drops what it did not understand.** The protobuf issue #272 thread (2015-04-07 → closed 2017-12-11, 69 comments) is the best corpus of real failure modes anywhere in this research. All of the following are [RAW], extracted from the GitHub REST API:
The proxy / intermediary case, stated with a diagram by `stevvooe`, 2016-11-18:
"Producer and Consumer could be updated with new fields, while intermediate can remain on the same version. If intermediate is a proxy of sorts, then this is important." [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
The silent-data-loss-is-indistinguishable-from-default case, `matthewrj`, 2016-08-31 — **this is the sharpest statement of the harm in the entire thread**:
"We have the same use case where A sends data to B which reads some fields and forwards the message to C. We don't want to have to constantly update B when the schema changes even though it doesn't read any of the new fields. The current behaviour is quite dangerous since C can't tell if one of the new fields was set to the default value or if B is just out of date and lost data." [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
The cryptographic-signature case, `chmod007`, 2016-10-05:
"Include a signature in the same protobuf as the payload to be signed. To verify the signature, I deserialize, extract and remove the signature, reserialize and verify the signature. This breaks if the signed message contains any new fields unknown to the process verifying the signature." [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
The stream-processing case, `Kaiserchen`, 2016-08-24:
"This allows the stream processor to continue working even when upstream schema changes happen, we do not need to redeploy our stream processing application, and the new fields end up in the output for free." [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
"To add some drama: I think loosing the unknown fields will force us to move to avro" [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
The deploy-ordering / topological-sort case, `Xorlev`, 2016-11-18:
"Depending on any cycles in data flows, there may be no topological order that produces valid schema updates without doing a 2-step deploy: 1) upgrade proto schema, redeploy all the (many) things that might rely on it 2) update producer to fill in field, deploy producer. Pray all the systems were updated." [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
The org-coupling case, `fducat`, 2016-12-12:
"The interest of using unknown fields is simply development efficiency by removing team dependencies. Usually one or two BE in the row are interested in the change. Forcing all 12 to update the version in coordination is what we cannot afford." [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
The silent-deletion objection, `jeremyong`, 2016-03-14:
"I have a lot of concerns about silently deleting data upon deserialization, to the point that even though we have internally been using proto3 for several months, I am considering changing things back to proto2." [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
**Answer: protobuf does NOT use backward/forward vocabulary. It uses a three-tier wire-safety taxonomy instead.** This is a genuine and under-appreciated finding: the backward/forward vocabulary people attribute to protobuf actually comes from Avro-and-Kafka-land.
Protobuf's own tiers, from "Updating A Message Type" [all WF, https://protobuf.dev/programming-guides/proto3/]:
Note that "safe" here is *symmetric* — protobuf's model is that the same change is fine in both directions, because unknown fields are preserved and missing fields default. The backward/forward split only becomes necessary once you have a format (like Avro or like a strict JSON reader) where the two directions genuinely differ. **Protobuf's taxonomy is bidirectional-by-construction; Avro's is not.** See §4.
The proto2 rule list adds many more conditional compatibilities [WF, https://protobuf.dev/programming-guides/proto2/]: "Integer type conversions (int32↔int64, etc.)", "sint32/sint64 compatibility with each other only", "string/bytes compatibility with valid UTF-8", "Embedded messages compatible with bytes", "fixed32↔sfixed32, fixed64↔sfixed64", "Singular↔repeated for string/bytes/message fields", "enum↔int32/uint32/int64/uint64", "map↔repeated message field conversions".
**Answer: no untagged unions. Avro unions are tagged by branch INDEX in binary and by TYPE NAME in JSON.** The JSON encoding is the interesting one for our purposes, because Avro faced exactly the "JSON union" problem and chose explicit tagging.
Binary:
"A union is encoded by first writing an `int` value indicating the zero-based position within the union of the schema of its value. The value is then encoded per the indicated schema within the union." [WF] — https://avro.apache.org/docs/1.11.1/specification/
JSON — **the explicit tagging decision**:
"if its type is _null_, then it is encoded as a JSON _null_; otherwise it is encoded as a JSON object with one name/value pair whose name is the type's name and whose value is the recursively encoded value." [WF] — https://avro.apache.org/docs/1.11.1/specification/
So `{"int": 5}` rather than `5`. Avro's own reason: the JSON encoding still cannot be read without the schema, and the wrapper is what keeps it unambiguous:
"The original schema is still required to correctly process JSON-encoded data" because the encoding cannot distinguish between semantically similar types like int versus long. [WF] — https://avro.apache.org/docs/1.12.0/specification/
Note the branch-index dependency: because binary unions are indexed positionally, **reordering union branches is a data-corrupting change**, exactly as reordering FlatBuffers union variants is. JSON's name-tagging removes that hazard — a rare case where the JSON encoding is *more* evolvable than the binary one.
**Answer: YES, cleanly, and this is Avro's structural advantage over protobuf. `[]` is an empty array; absence is expressed by `["null", {"type":"array",...}]` and encoded as a distinct union branch.** Avro has a real `null` type, so empty/null/absent are three different encodable states.
The mechanism that makes "absent" meaningful across versions is that defaults live **in the schema**, not in the code:
"A default value for this field, only used when reading instances that lack the field for schema evolution purposes. The presence of a default value does not make the field optional at encoding time. Avro encodes a field even if its value is equal to its default." [WF] — https://avro.apache.org/docs/1.11.1/specification/
That last sentence is important and frequently misread: **Avro always writes the field.** There is no "omit if default" optimisation, so there is no ambiguity of the protobuf implicit-presence kind. The default is purely a *reader-side* mechanism for fields the writer's schema never had.
**Answer: Avro enums were CLOSED and hard-fail, which was a five-year known-bad; a reader-side `default` was added in 1.9.0 to soften it — and it then didn't work for years.** This is a reversal-shaped story (§3, R5).
The base rule:
"if the writer's symbol is not present in the reader's enum and the reader has a default value, then that value is used, otherwise an error is signalled." [WF] — https://avro.apache.org/docs/1.11.1/specification/
"A default value for this enumeration, used during resolution when the reader encounters a symbol from the writer that isn't defined in the reader's schema (optional). The value provided here must be a JSON string that's a member of the symbols array." [WF] — https://avro.apache.org/docs/1.11.1/specification/
**The actual discussion (AVRO-1340, reported Jim Donofrio 25/May/13, resolved 02/May/18, fix version 1.9.0):**
"if the writer's symbol is not present in the reader's enum, then an error is signalled" [WF, quoting the pre-fix spec] — https://issues.apache.org/jira/browse/AVRO-1340
"makes it difficult to use enum's because you can never add a enum value and keep old reader's compatible" [WF] — https://issues.apache.org/jira/browse/AVRO-1340
**Five years from report to fix.** And then AVRO-3313 (affects 1.9.0, 1.9.1, 1.9.2, 1.10.0, 1.10.1, 1.10.2, 1.11.0) reported the fix did not actually work:
Writer schema (v2) enum `["A","B","C"]` default "A"; reader schema (v1) enum `["A","B"]` default "A"; expected the reader to deserialize unknown "C" as "A"; actual: `org.apache.avro.AvroTypeException: No match for C` [WF, paraphrase of the reproduction] — https://issues.apache.org/jira/browse/AVRO-3313
Resolved: **Not A Bug** (27/Sep/23) [WF] — https://issues.apache.org/jira/browse/AVRO-3313
The "Not A Bug" resolution is itself a finding: the enum default applies during *schema resolution* (reader schema explicitly supplied and differing from writer's), not to a naive single-schema decode, and users repeatedly hit the difference. **Practical takeaway: a reader-side fallback that only fires in an explicitly-two-schema code path will be missed by most users.**
Secondary/practitioner guidance found: add symbol defaults *pre-emptively* because older Avro versions tolerate-and-ignore them, so they become useful once everyone upgrades. [WF] — surfaced via https://medium.com/expedia-group-tech/safety-considerations-when-using-enums-in-avro-schemas-82e18baaa081 (secondary source, flagged as such).
**Answer: reader ignores writer's extra fields; reader's extra fields must have defaults or it is a hard error.** Avro is tolerant in one direction and *strict* in the other, and it is the strict direction that bites.
"if the writer's record contains a field with a name not present in the reader's record, the writer's value for that field is ignored." [WF] — https://avro.apache.org/docs/1.11.1/specification/
"if the reader's record schema has a field that contains a default value, and writer's schema does not have a field with the same name, then the reader should use the default value from its field." [WF] — https://avro.apache.org/docs/1.11.1/specification/
"if the reader's record schema has a field with no default value, and writer's schema does not have a field with the same name, an error is signalled." [WF] — https://avro.apache.org/docs/1.11.1/specification/
Note the word **"ignored"** in the first rule — not "preserved". See Q7.
**Answer: no version number; Avro versions by carrying the WRITER'S SCHEMA WITH THE DATA. This is the most important structural idea in Avro and the one that transfers best to files-at-rest.**
"a reader of Avro data, whether from an RPC or a file, can always parse that data because the original schema must be provided along with the data" [WF] — https://avro.apache.org/docs/1.12.0/specification/
"Binary encoded Avro data does not include type information or field names." [WF] — https://avro.apache.org/docs/1.12.0/specification/
Evolution is then a *pairwise function of two schemas*, not a linear version ladder. There is no "version 3"; there is only "resolve writer-S1 against reader-S2". That means compatibility is a relation, not an ordering — a point §4 develops.
**Answer: NAMES, not numbers — the opposite of protobuf/Thrift/Cap'n Proto/FlatBuffers. Renaming is therefore a breaking change, softened by `aliases`.**
"if the writer's record contains a field with a name not present in the reader's record, the writer's value for that field is ignored" [WF] — https://avro.apache.org/docs/1.11.1/specification/ (field matching is by name)
"Named types and fields may have aliases. An implementation may optionally use aliases to map a writer's schema to the reader's. This facilitates both schema evolution as well as processing disparate datasets." [WF] — https://avro.apache.org/docs/1.11.1/specification/
Note **"An implementation MAY optionally use aliases"** — alias support is not guaranteed. That is a materially weaker guarantee than protobuf's `reserved`, which is enforced by the compiler.
There is no `reserved` mechanism in Avro. **Avro `reserved` equivalent: NOT FOUND** — searched the 1.11.1 and 1.12.0 specifications; Avro has no construct for retiring a name so it cannot be reused with different semantics. Name reuse with a changed type is caught only if the reader happens to resolve against the old writer's schema and the types fail to match.
Practitioner-level rules found (secondary sources, flagged): you cannot rename a field (use aliases); you cannot change a field's data type (add a new field); "A non-union type may be changed to a union that contains only the original type, or vice-versa"; "If you do not provide a default value for a field, you cannot delete that field from your schema." [WF] — https://docs.oracle.com/cd/E26161_02/html/GettingStartedGuide/schemaevolution.html
**Answer: NO. Avro DROPS unknown fields — "ignored" is the spec's word — and this is the concrete reason people cited protobuf's unknown-field preservation as the reason to prefer protobuf over Avro (and, in issue #272, the reverse).**
"if the writer's record contains a field with a name not present in the reader's record, the writer's value for that field is ignored." [WF] — https://avro.apache.org/docs/1.11.1/specification/
Avro's mitigation is different in kind: because the writer's schema travels with the data (object container files), a reader that resolves against the *writer's* schema loses nothing — but any reader that projects onto its own reader schema and re-writes has performed a lossy transformation. **Avro trades "the reader can preserve what it doesn't understand" for "the data can always be re-read later with full fidelity, given the original file."** For files-at-rest that trade is arguably better; for read-modify-write pipelines it is worse.
**Avro's own spec defines NO backward/forward taxonomy — NOT FOUND in the specification.** Searched the 1.11.1 and 1.12.0 specs; they define *schema resolution* rules only. The backward/forward/full vocabulary that everyone attributes to Avro is Confluent Schema Registry's, layered on top. See §1.7 and §4.
**Answer: no untagged unions.** Thrift's `union` is a struct in which at most one field is set, and every field carries a numeric field id in its wire header, so the variant is identified by id. NOT FOUND: an authoritative Apache Thrift statement specifically about `union` semantics — https://thrift.apache.org/docs/types returned "Unions are not mentioned in this content" and https://thrift.apache.org/docs/idl documents field ids and requiredness but I did not obtain a union section. Searched thrift.apache.org/docs/idl, /docs/types, and the 2007 whitepaper.
**Answer: YES for scalars via the generated `__isset` struct; for collections it depends on requiredness. The `__isset` design is Thrift's explicit-presence mechanism and predates protobuf's rediscovery of the same idea.**
[RAW] from the 2007 Facebook whitepaper, §5:
"Essentially, the inner isset object of each Thrift struct contains a boolean value for each field which denotes whether or not that field is present in the struct. When a reader receives a struct, it should check for a field being set before operating directly on it." [RAW] — https://thrift.apache.org/static/files/thrift-20070401.pdf
The three requiredness levels [WF] — https://thrift.apache.org/docs/idl:
"Required fields are always written and are expected to be set." / "Required fields are always read and are expected to be contained in the input stream." "Optional fields are only written when they are set" / "Optional fields may, or may not be part of the input stream." (default/implicit) "In theory, the fields are always written. There are some exceptions to that rule." / "Like optional, the field may, or may not be part of the input stream."
"Default requiredness is a good starting point. The desired behaviour is a mix of optional and required." [WF] — https://thrift.apache.org/docs/idl
**Answer: Thrift enums are i32 on the wire; behaviour on unknown values is implementation-dependent and NOT specified authoritatively.** NOT FOUND in thrift.apache.org/docs/types (explicitly returned NOT FOUND) or /docs/idl.
The best statement I found is secondary, from *Thrift: The Missing Guide* (Diwaker Gupta), §1.3 [RAW, extracted from the PDF]:
"a field with an enum type can only have one of a specified set of constants as its value (if you try to provide a different value, the parser will treat it like an unknown field)" [RAW] — https://diwakergupta.github.io/thrift-missing-guide/thrift.pdf
"Enumerator constants MUST be in the range of postive 32-bit integers." [RAW, `postive` sic] — https://diwakergupta.github.io/thrift-missing-guide/thrift.pdf
Practitioner reports confirm the hazard: adding enum values can break jobs running an older Thrift schema. [WF] — surfaced via https://cwiki.apache.org/confluence/display/FLINK/FLIP-237:+Thrift+Format+Support (secondary).
**Answer: tolerant reader by design, stated in 2007 and unchanged.** [RAW] from the whitepaper §5.3 "Case Analysis" — this is the four-case table the question asks about:
"1. Added field, old client, new server. In this case, the old client does not send the new field. The new server recognizes that the field is not set, and implements default behavior for out-of-date requests." [RAW]
"2. Removed field, old client, new server. In this case, the old client sends the removed field. The new server simply ignores it." [RAW]
"3. Added field, new client, old server. The new client sends a field that the old server does not recognize. The old server simply ignores it and processes as normal." [RAW]
"4. Removed field, new client, old server. This is the most dangerous case, as the old server is unlikely to have suitable default behavior implemented for the missing field. It is recommended that in this situation the new server be rolled out prior to the new clients." [RAW]
— all https://thrift.apache.org/static/files/thrift-20070401.pdf
Case 4 is Thrift's own naming of the **forward-compatibility hazard** and its own prescription of an **upgrade ordering** — thirteen years before Confluent's tables said the same thing. Note it is a *deployment-order* remedy, not a format remedy.
**Answer: no schema version in the data; and Thrift explicitly separates protocol-level versioning from IDL-level versioning.** [RAW], whitepaper §5.4:
"The TProtocol abstractions are also designed to give protocol implementations the freedom to version themselves in whatever manner they see fit. Specifically, any protocol implementation is free to send whatever it likes in the writeMessageBegin() call. It is entirely up to the implementor how to handle versioning at the protocol level. The key point is that protocol encoding changes are safely isolated from interface definition version changes." [RAW] — https://thrift.apache.org/static/files/thrift-20070401.pdf
**This two-layer split is a genuinely transferable idea**: version the *envelope/encoding* explicitly and let the *content* evolve by field rules. See §5.
**Answer: numeric field identifiers, and the whitepaper already recommends always writing them explicitly.** [RAW], §5.1:
"Versioning in Thrift is implemented via field identifiers. The field header for every member of a struct in Thrift is encoded with a unique field identifier. The combination of this field identifier and its type specifier is used to uniquely identify the field. The Thrift definition language supports automatic assignment of field identifiers, but it is good programming practice to always explicitly specify field identifiers." [RAW] — https://thrift.apache.org/static/files/thrift-20070401.pdf
Note **"The combination of this field identifier and its type specifier"** — Thrift's identity is (id, type), so changing a field's type is a distinct field, not a redefinition. That is a subtly different (and arguably safer) identity model than protobuf's id-alone.
**Thrift `reserved` mechanism: NOT FOUND.** Searched thrift.apache.org/docs/idl and the whitepaper. Thrift has no reserved-id construct; the discipline is purely conventional.
**Answer: NO — unknown fields are skipped, not retained. Authoritative statement NOT FOUND, but the whitepaper's "simply ignores it" wording (cases 2 and 3 above) plus the absence of any unknown-field-set API in generated code is dispositive in practice.** Searched thrift.apache.org/docs/idl, /docs/types, the whitepaper, and the Missing Guide. There is no `UnknownFieldSet` analogue in Thrift's generated code model.
**Answer: no named taxonomy; the four-case analysis in §5.3 IS Thrift's taxonomy.** Thrift does not use the words backward/forward. NOT FOUND: any published Facebook retrospective on the required/optional decision. Searched for "Thrift retrospective", "Facebook Thrift lessons", "required fields harmful Thrift". The closest published critique is Thrift's own IDL documentation, which is unusually blunt for official docs:
"Because of this behaviour, required fields drastically limit the options with regard to soft versioning." [WF] — https://thrift.apache.org/docs/idl
"Because they must be present on read, the fields cannot be deprecated." [WF] — https://thrift.apache.org/docs/idl
"If a required field would be removed (or changed to optional), the data are no longer compatible between versions." [WF] — https://thrift.apache.org/docs/idl
**This is Thrift's own retrospective on `required`, in its official docs, reaching the same conclusion protobuf reached — without ever removing the keyword.** See §3, R1.
All quotes in this subsection are [RAW] — extracted from https://www.rfc-editor.org/rfc/rfc8927.txt, downloaded and read directly.
**Answer: JTD has ONLY tagged unions. Untagged unions are absent by design, and the RFC gives the rationale twice — once positively (code generation) and once negatively (ambiguity).**
The positive rationale, §1:
"JTD's niche is to focus on enabling code generation from schemas; to this end, JTD's expressiveness is intentionally limited to be no more powerful than what can be expressed in the type systems of mainstream programming languages." [RAW]
"Enable code generation from JTD schemas. JTD schemas are meant to be easy to convert into data structures idiomatic to mainstream programming languages." [RAW]
"JTD is intentionally designed as a rather minimal schema language. Thus, although JTD can describe some categories of JSON, it is not able to describe its own structure... By keeping the expressiveness of the schema language minimal, JTD makes code generation and standardized error indicators easier to implement." [RAW]
"A \"discriminator\" form of JSON objects, corresponding to a discriminated (or \"tagged\") union. The \"discriminator\" form of JSON objects is akin to a C++ \"std::variant\"." [RAW]
"JTD's feature set is designed to represent common patterns in JSON-using applications, while still having a clear correspondence to programming languages in widespread use." [RAW]
"The principle of clear correspondence to common programming languages is why JTD does not support, for example, a data type for integers up to 2**53-1." [RAW]
**The negative rationale — §2.2.8 is an entire section devoted to making tags unambiguous by construction. This is the most directly applicable text in the whole research corpus for a JSON-in-a-repo format:**
"To prevent ambiguous or unsatisfiable constraints on the \"discriminator\" property of a tagged union, an additional constraint on schemas of the \"discriminator\" form exists." [RAW]
"* For each member _P_ of _S_ whose name equals \"properties\" or \"optionalProperties\", _P_'s value, which must be an object, MUST NOT contain any members whose name equals _D_'s value." [RAW]
"JTD handles such possible ambiguity by disallowing, at the syntactic level, the possibility of contradictory specifications of discriminator \"tags\". Discriminator \"tags\" cannot be redefined in other parts of the schema." [RAW]
"JTD handles such possible ambiguity by disallowing, at the syntactic level, the possibility of contradictory specifications of whether an instance described by a schema of the \"discriminator\" form may be null. The schemas in a discriminator \"mapping\" cannot have \"nullable\" set to \"true\"; only the discriminator itself can use \"nullable\" in this way." [RAW]
Note the repeated phrase **"disallowing, at the syntactic level"** — JTD's method is to make the ambiguous schema *unwritable*, not to define a resolution rule for it. Contrast OpenAPI's discriminator (§1.6), which is a *hint* layered over a validation model that would work without it.
Confirmation that JTD refuses type-unions generally, from the Ajv implementation docs:
"Unlike JSON Schema, JTD does not allow defining values that can take one of several types, but they can be defined as `nullable`." [WF] — https://ajv.js.org/json-type-definition.html
**Answer: YES — JTD distinguishes all three cleanly, and it is the only format here that does so with three distinct, orthogonal, first-class mechanisms.**
And a sharp asymmetry worth copying: `nullable: false` is inert.
"it is not the case that putting a 'false' value for 'nullable' will ever override a 'nullable' member." [WF] — https://www.rfc-editor.org/rfc/rfc8927.html
**Answer: CLOSED and hard-rejecting. There is no open-enum, no sentinel convention, no fallback.**
"For a schema of the \"enum\" form to be correct, the value of the member named \"enum\" must be a nonempty array of strings, and that array must not contain duplicate values." [WF, §2.2.4] — https://www.rfc-editor.org/rfc/rfc8927.html
Validating a value not in the list produces an error with `schemaPath` pointing to `/enum` — unknown values are rejected. [WF, §3.3.4] — https://www.rfc-editor.org/rfc/rfc8927.html
**JTD provides no enum-evolution story at all.** This is a real limitation for a long-lived at-rest format and JTD does not pretend otherwise.
**Answer: STRICT BY DEFAULT — unknown members are rejected unless the schema opts in. JTD is the anti-Postel design point in this corpus, and unusually, the RFC acknowledges the disagreement explicitly rather than asserting a winner.**
"Some users may expect that {\"a\": \"foo\", \"b\": \"bar\"} satisfies the schema in Figure 2. Others may disagree, as \"b\" is not one of the properties described in the schema." [RAW, §3.1]
"Evaluation of a schema does not allow additional properties by default, but this can be overridden by having the schema include a member named \"additionalProperties\", where that member has a value of \"true\"." [RAW, §3.1]
"briefly, the schema { \"properties\": { \"a\": { \"type\": \"string\" }}} rejects { \"a\": \"foo\", \"b\": \"bar\" }" [RAW, §3.1]
**The one deliberate hole in the strictness — the "discriminator tag exemption"** [RAW, §3.3.6]:
"If the \"discriminator tag exemption\" is in effect on _I_ (see Section 3.3.8), then ignore _I_." [RAW]
That is: the tag property itself is exempted from the variant schema's "no additional properties" rule, because the variant schema doesn't declare the tag (it's declared once, on the union). A small but instructive piece of engineering — the strictness rule and the tagging rule would otherwise collide, and JTD carved the exemption at exactly one point rather than loosening either rule.
Confirmation from the implementation side:
"Unlike JSON Schema, all properties defined in `properties` schema member are required, the data instance must be JSON object (without using additional `type` keyword) and by default additional properties are not allowed (with the exception of discriminator tag)." [WF] — https://ajv.js.org/json-type-definition.html
**Answer: NONE. JTD has no versioning mechanism whatsoever, and no evolution rules. The only extension point is inert metadata.**
"Users MAY add metadata members to JTD schemas to convey information that is not pertinent to validation." [WF, §2.3] — https://www.rfc-editor.org/rfc/rfc8927.html
"Users SHOULD NOT expect metadata members to be understood by other parties. As a result, if consistent validation with other parties is a requirement, users MUST NOT use metadata members to affect how schema validation...works." [WF, §2.3] — https://www.rfc-editor.org/rfc/rfc8927.html
**JTD schema-evolution rules: NOT FOUND — because there are none.** Searched the full RFC text for "evolution", "compatib", "version". The RFC is a validation spec, not an evolution spec. This is a genuine and important gap: **JTD gives you a rigorous way to say what a document looks like today and no way at all to say how it may change.**
The RFC is also unusually honest about its own status:
"This document does not have IETF consensus and is presented here to facilitate experimentation with the concept of JTD. The purpose of the experiment is to gain experience with JTD and to possibly revise this work accordingly." [RAW, §1]
**Answer: names only. No numeric tags, no reserved mechanism, no aliases.** JTD is a validation language over JSON, and JSON member names are the only identity available. NOT FOUND: any reuse policy. Searched the full RFC.
**Answer: N/A for the validator; NEGATIVE for the code generators, which is the point.** JTD does not define a codec, so it neither preserves nor drops. But its raison d'être is code generation into `struct`s, and generated structs by construction drop what is not declared. `additionalProperties: true` permits unknown members to *pass validation*; it does not create a place to *store* them.
**Explicit statement about round-trip preservation in JTD: NOT FOUND.** Searched RFC 8927 in full and the Ajv JTD docs.
**NOT FOUND — JTD defines none.** Searched RFC 8927 in full.
**Ulysse Carion's own blog post articulating JTD's design goals vs JSON Schema: NOT FOUND.** Searched: "Ulysse Carion JSON Type Definition design why not JSON Schema blog rationale discriminator"; "jsontypedef JSON Type Definition Carion blog tagged unions why no untagged union design goal code generation"; and direct fetches of https://jsontypedef.com/ and https://jsontypedef.com/docs/. **Note: `jsontypedef.com` is no longer under the author's control — the domain now serves unrelated casino-affiliate content.** The canonical design rationale surviving in a citable, stable form is RFC 8927 §1 and Appendix A ("Rationale for Omitted Features", §A.1 on 64-bit numbers, §A.2 on non-root definitions) plus the `github.com/jsontypedef` org. **If this rationale matters to the decision, RFC 8927 Appendix A is the artifact to cite, not the website.**
**Q1 — unions:** explicitly tagged, with a stored discriminant.
"A union is two or more fields of a struct which are stored in the same location. Only one of these fields can be set at a time, and a separate tag is maintained to track which one is currently set." [WF] — https://capnproto.org/language.html
**Q2 — absent/empty/null:** Cap'n Proto deliberately has **no optional** and no presence.
"Cap'n Proto has no notion of 'optional' fields." [RAW] — https://capnproto.org/faq.html
"A primitive field always takes space on the wire whether you set it or not (although default-valued fields will be compressed away if you enable packing). Such a field can be made semantically optional by placing it in a union with a Void field" [RAW] — https://capnproto.org/faq.html
"A better approach may be to give the field a bogus default value and interpret that value to mean 'not present'." [RAW] — https://capnproto.org/faq.html
"Pointer fields are a bit different. They start out 'null', and you can check for nullness using the hasFoo() accessor. You could use a null pointer to mean 'not present'. Note, though, that calling getFoo() on a null pointer returns the default value, which is indistinguishable from a legitimate value, so checking hasFoo() is in fact the only way to detect nullness." [RAW] — https://capnproto.org/faq.html
**Q3 — enums:** new enumerants may be added at the end.
"New fields, enumerants, and methods may be added to structs, enums, and interfaces, respectively, as long as each new member's number is larger than all previous members." [WF] — https://capnproto.org/language.html
**Q4/Q7 — strictness and unknown-field retention:** Cap'n Proto **retains** unknown fields, and its author flagged proto3's removal of the same as a mistake at the time:
Feature matrix "Unknown field retention" — Cap'n Proto: **"yes"**; Protobuf: **"removed in proto3"**; SBE: **"no"**; FlatBuffers: **"no"**. [WF] — https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html
"Apparently, version 3 of Protocol Buffers, aka 'proto3', removes this feature. I honestly don't know what they're thinking." [WF] — https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html
"This feature has been absolutely essential in many of Google's internal systems." [WF] — https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html
**Q5 — version semantics:** none; ordinal-based field-level rules only.
**Q6 — field identity:** ordinals (`@N`) and type IDs. The allowed/forbidden list [WF] — https://capnproto.org/language.html:
Allowed: "New types, constants, and aliases can be added anywhere, since they obviously don't affect the encoding of any existing type." / "New parameters may be added to a method. The new parameters must be added to the end of the parameter list and must have default values." / "Members can be re-arranged in the source code, so long as their numbers stay the same." / "Any symbolic name can be changed, as long as the type ID / ordinal numbers stay the same." / "Type definitions can be moved to different scopes, as long as the type ID is declared explicitly." / "A field can be moved into a group or a union, as long as the group/union and all other fields within it are new."
Forbidden: "You cannot change a field, method, or enumerant's number" / "change a field or method parameter's type or default value" / "You cannot change a type's ID" / "move an existing field into or out of an existing union, nor can you form a new union containing more than one existing field."
**Q8 — taxonomy:** none named.
**The explicit comparison to protobuf's mistakes** (this is the requested item, and it is the single most vivid passage found in this entire research effort — all [RAW], https://capnproto.org/faq.html):
"You don't. You may find this surprising, but the 'required' keyword in Protocol Buffers turned out to be a horrible mistake." [RAW]
"The problem with this is, validation is sometimes more subtle than that. Sometimes, different applications – or different parts of the same application, or different versions of the same application – place different requirements on the same protocol. An application may want to pass around partially-complete messages internally. A particular field that used to be required might become optional. A new use case might call for almost exactly the same message type, minus one field, at which point it may make more sense to reuse the type than to define a new one." [RAW]
"A field declared required, unfortunately, is required everywhere. The validation is baked into the parser, and there's nothing you can do about it. Nothing, that is, except change the field from 'required' to 'optional'. But that's where the real problems start." [RAW]
"Imagine a production environment in which two servers, Alice and Bob, exchange messages through a message bus infrastructure running on a big corporate network. The message bus parses each message just to examine the envelope and decide how to route it, without paying attention to any other content. Often, messages from various applications are batched together and then split up again downstream." [RAW]
"Now, at some point, Alice's developers decide that one of the fields in a deeply-nested message commonly sent to Bob has become obsolete. To clean things up, they decide to remove it, so they change the field from 'required' to 'optional'. The developers aren't idiots, so they realize that Bob needs to be updated as well. They make the changes to Bob, and just to be thorough they run an integration test with Alice and Bob running in a test environment. The test environment is always running the latest build of the message bus, but that's irrelevant anyway because the message bus doesn't actually care about message contents; it only does routing. Protocols are modified all the time without updating the message bus." [RAW]
"Satisfied with their testing, the devs push a new version of Alice to prod. Immediately, everything breaks. And by 'everything' I don't just mean Alice and Bob. Completely unrelated servers are getting strange errors or failing to receive messages. The whole data center has ground to a halt and the sysadmins are running around with their hair on fire." [RAW]
"What happened? Well, the message bus running in prod was still an older build from before the protocol change. And even though the message bus doesn't care about message content, it does need to parse every message just to read the envelope. And the protobuf parser checks the entire message for missing required fields. So when Alice stopped sending that newly-optional field, the whole message failed to parse, envelope and all. And to make matters worse, any other messages that happened to be in the same batch also failed to parse, causing errors in seemingly-unrelated systems that share the bus." [RAW]
"Things like this have actually happened. At Google. Many times." [RAW]
"The right answer is for applications to do validation as-needed in application-level code. If you want to detect when a client fails to set a particular field, give the field an invalid default value and then check for that value on the server. Low-level infrastructure that doesn't care about message content should not validate it at all." [RAW]
"Oh, and also, Cap'n Proto doesn't have any parsing step during which to check for required fields. :)" [RAW]
**The generalisable law in that story: validation baked into the PARSER propagates failure to every party that must parse, including parties that do not care about the content.** For a JSON-file format read by third-party tools, the analogue is exact: a strict schema validator wired into the *load* path of a generic tool turns every schema addition into a breakage of tools that never looked at the new field.
**Q1 — unions:** tagged, with a discriminant, and the discriminant is positional.
"New union variants must be appended at the end to prevent discriminant mismatches. Adding variants mid-union causes 'CodeV1' and 'CodeV2' to misinterpret values. Using explicit discriminant values (e.g., `A = 1`) allows middle insertion safely by overriding positional assignment." [WF] — https://flatbuffers.dev/evolution/
**Q2 — absent/empty:** tables encode presence via the vtable; a field not written is absent and reads as its default. Structs cannot omit anything:
"structs... are required (so no defaults either), and fields may not be added or be deprecated." [WF] — https://flatbuffers.dev/schema/
"Fields do not have to appear in the wire representation, and you can choose to omit fields when constructing an object. You have the flexibility to add fields without fear of bloating your data." [WF] — https://flatbuffers.dev/schema/
**Q3 — enums:** append-only, and FlatBuffers explicitly pushes unknown-value handling to the application:
"Typically, enum values should only ever be added, never removed (there is no deprecation for enums). This requires code to handle forwards compatibility itself, by handling unknown enum values." [WF] — https://flatbuffers.dev/schema/
**Q4/Q7 — strictness / unknown-field retention:** unknown fields are skipped and **NOT retained** ("no" in Cap'n Proto's matrix above). Confirmed by the absence of any unknown-field API in FlatBuffers' model.
**Q5 — version semantics:** none.
**Q6 — field identity:** slot order, or explicit `id`:
"New fields MUST be added to the end of the table definition." [WF] — https://flatbuffers.dev/evolution/
"You MUST not remove a field from the schema, even if you don't use it anymore." [WF] — https://flatbuffers.dev/evolution/
"do not generate accessors for this field anymore, code should stop using this data. Old data may still contain this field, but it won't be accessible anymore by newer code." (the `deprecated` attribute) [WF] — https://flatbuffers.dev/schema/
"If you use this attribute, you must use it on ALL fields of this table, and the numbers must be a contiguous range from 0 onwards... When a new field is added to the schema it must use the next available ID." (the `id` attribute) [WF] — https://flatbuffers.dev/schema/
"You can ignore this rule if you use the `id` attribute on all the fields of a table." [WF] — https://flatbuffers.dev/evolution/
Renaming is safe because names are not serialized:
"Renaming tables and fields is generally permissible since names aren't serialized." [WF] — https://flatbuffers.dev/evolution/
**Q8 — taxonomy:** none named.
**Explicit comparison to protobuf's mistakes: NOT FOUND** in FlatBuffers' own docs. The FlatBuffers schema page "contains no explicit Protocol Buffers comparison regarding evolution rules" [WF]. The comparison exists only from the outside (Cap'n Proto's 2014 matrix, above).
**FlatBuffers' `deprecated` is the cleanest `reserved` analogue in the corpus:** it keeps the slot occupied forever *and* removes the accessor, so the compiler enforces "you cannot use this and you cannot reuse this" in one attribute. Protobuf's `reserved` does the second half only; nobody else does either half well.
All OpenAPI quotes are [RAW] — extracted from the spec markdown at https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/versions/3.0.3.md.
**Answer: JSON Schema's `oneOf`/`anyOf` ARE untagged unions — matching is by full structural validation. OpenAPI's `discriminator` bolts a tag on top, but only as an optimisation hint, not as the semantics.** This is the crucial architectural difference from JTD and the reason the OpenAPI discriminator is a persistent source of trouble.
"When request bodies or response payloads may be one of a number of different schemas, a `discriminator` object can be used to aid in serialization, deserialization, and validation. The discriminator is a specific object in a schema which is used to inform the consumer of the specification of an alternative schema based on the value associated with it." [RAW]
"The discriminator object is legal only when using one of the composite keywords `oneOf`, `anyOf`, `allOf`." [RAW]
**The "hint" sentence — this is the single most consequential sentence in the OpenAPI discriminator design:**
"which means the payload _MUST_, by validation, match exactly one of the schemas described by `Cat`, `Dog`, or `Lizard`. In this case, a discriminator MAY act as a \"hint\" to shortcut validation and selection of the matching schema which may be a costly operation, depending on the complexity of the schema." [RAW]
Because the discriminator is a *hint* over an underlying structural `oneOf`, two tools can legitimately disagree: one dispatches on the tag, the other validates all branches. Where the two disagree — a payload whose tag says `Dog` but whose shape matches `Cat` — behaviour is unspecified.
The required-ness of the tag property, and the exclusion of inline schemas:
"When used, the `discriminator` will be the name of the property that decides which schema definition validates the structure of the model. As such, the `discriminator` field MUST be a required field." [RAW]
"As such, inline schema definitions, which do not have a given id, *cannot* be used in polymorphism." [RAW]
"When using the discriminator, _inline_ schemas will not be considered." [RAW]
"propertyName | `string` | **REQUIRED**. The name of the property in the payload that will hold the discriminator value." [RAW]
"mapping | Map[`string`, `string`] | An object to hold mappings between payload values and schema names or references." [RAW]
**Unknown tag value:**
"If the discriminator _value_ does not match an implicit or explicit mapping, no schema can be determined and validation SHOULD fail. Mapping keys MUST be string values, but tooling MAY convert response values to strings for comparison." [RAW]
Note **SHOULD**, not MUST, and **"tooling MAY convert response values to strings"** — two more places where conforming tools may diverge.
"When used in conjunction with the `anyOf` construct, the use of the discriminator can avoid ambiguity where multiple schemas may satisfy a single payload." [RAW]
"In both the `oneOf` and `anyOf` use cases, all possible schemas MUST be listed explicitly. To avoid redundancy, the discriminator MAY be added to a parent schema definition, and all schemas comprising the parent schema in an `allOf` construct may be used as an alternate schema." [RAW]
**"all possible schemas MUST be listed explicitly"** is a closed-world requirement: an OpenAPI union cannot be open for extension by a third party. Compare JTD (also closed) and protobuf `oneof` (also closed). **No format in this corpus supports an extensible/open union.** That is itself a finding.
**Answer: YES — JSON Schema distinguishes all three (`required` for presence, `type: "null"` for null, `[]` for empty), but OpenAPI 3.0 famously did NOT have `type: null` and used a bespoke `nullable: true` instead; 3.1 realigned with JSON Schema.** NOT FOUND: an authoritative verbatim quote on the 3.0→3.1 `nullable` change — I did not fetch a 3.1 section covering it (the 3.1 HTML fetch truncated before reaching the relevant sections). Flagging as a gap.
JSON Schema `enum` is closed and rejecting, like JTD. NOT FOUND: a verbatim JSON Schema statement on enum evolution, which does not exist because JSON Schema has no evolution model.
**Answer: JSON Schema is TOLERANT by default — the opposite of JTD — and the mechanism for tightening it is famously broken under composition.**
"By default any additional properties are allowed." [WF] — https://json-schema.org/understanding-json-schema/reference/object
"`additionalProperties` only recognizes properties declared in the same subschema as itself. So, `additionalProperties` can restrict you from 'extending' a schema using combining keywords such as allOf." [WF] — https://json-schema.org/understanding-json-schema/reference/object
"Because `additionalProperties` only recognizes properties declared in the same subschema, it considers anything other than 'street_address', 'city', and 'state' to be additional. Combining the schemas with allOf doesn't change that." [WF] — https://json-schema.org/understanding-json-schema/reference/object
`unevaluatedProperties` is "similar to `additionalProperties` except that it can recognize properties declared in subschemas." [WF] — https://json-schema.org/understanding-json-schema/reference/object
**This is a design reversal of a sort (§3, R7): `additionalProperties` proved unusable in the presence of `allOf`, and rather than change its semantics, JSON Schema added a second, differently-scoped keyword alongside it.**
JSON Schema versions the *schema language* via `$schema`, not the *document*. Documents carry no version. NOT FOUND: any JSON Schema statement on document/instance evolution — none exists; JSON Schema is explicitly a validation vocabulary.
Names only. No reserved mechanism, no aliases, no reuse policy. NOT FOUND.
N/A (validator, not codec). With `additionalProperties` unconstrained (the default), unknown members validate; whether they survive depends entirely on the consuming code, which is exactly the ambiguity JTD closed.
NOT FOUND in JSON Schema / OpenAPI. Supplied externally by Confluent (§1.7).
Beyond the "hint" ambiguity and the `additionalProperties`/`allOf` scoping failure above, the practitioner literature documents: the spec's ambiguity requiring trial-and-error to produce correct schemas; the interaction requiring both `type` and `required` to be present; code-generator divergence on `propertyName` handling in `oneOf` with `mapping`; and the redundancy argument — a `oneOf` over variants with distinct `required` sets already discriminates, making the keyword unnecessary in many cases. [WF, secondary sources] — https://github.com/OAI/OpenAPI-Specification/issues/2376 ; https://bump.sh/blog/the-discriminator-in-openapi-is-generally-redundant-and-confusing/ ; https://github.com/OpenAPITools/openapi-generator/issues/20954
"The Confluent Schema Registry default compatibility type is `BACKWARD`." [WF] — https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html
Backward: "consumers using the new schema can read data produced with the last schema" [WF] Forward: "data produced with a new schema can be read by consumers using the last schema" [WF] Full: "Schemas are both backward and forward compatible." [WF]
Per-mode [WF, same URL]:
| Type | Changes allowed | Checked against | Upgrade first |
| BACKWARD (default) | "add optional fields, remove fields" | last version | "upgrade all consumers before you start producing new events" |
| BACKWARD_TRANSITIVE | same | "all previously registered schemas" | consumers |
| FORWARD | "remove optional fields, add fields" | last version | "first upgrade all producers to using the new schema...then upgrade the consumers" |
| FORWARD_TRANSITIVE | same | "all registered schemas" | producers |
| FULL | "both backward and forward compatible (add/remove optional fields only)" | last version | "you can upgrade the producers and consumers independently" |
| FULL_TRANSITIVE | same | all previous | independent |
| NONE | — | — | "schema compatibility checks are disabled" |
**The TRANSITIVE distinction is the item most worth stealing.** Non-transitive modes check only against the immediately preceding version. That means a chain V1→V2→V3 can be pairwise-compatible at every step and yet V1 and V3 be mutually unreadable. For a repository format where old files persist indefinitely — which is exactly the data-at-rest case — **non-transitive compatibility is nearly worthless and TRANSITIVE is the only meaningful setting.** This is arguably the single most transferable operational lesson in §1.7.
Legend: **✓** yes / **✗** no / **~** partial or conditional / **NF** not found.
| **Protobuf (proto3/editions)** | **Avro** | **Thrift** | **JSON Typedef (RFC 8927)** | **Cap'n Proto** | **FlatBuffers** | **JSON Schema / OpenAPI** | **Confluent SR** | |
| **Q1 Untagged unions allowed?** | ✗ — `oneof`, tagged by field number. Unknown variant is indistinguishable from unset. Rationale for excluding untagged: **NF** | ✗ — tagged by branch **index** (binary) / **type name** (JSON). Reordering branches corrupts binary | ✗ — `union` is a struct; variant = field id. Union spec text: **NF** | ✗ — `discriminator` only. Rationale: code-gen parity with `std::variant`; ambiguity forbidden "at the syntactic level" | ✗ — union has "a separate tag ... to track which one is currently set" | ✗ — tagged; discriminant is **positional**, so variants must be appended | **✓ — `oneOf`/`anyOf` ARE untagged (structural match).** `discriminator` is only a "hint" | n/a |
| **Q2 Empty vs absent for collections?** | ✗ **cannot distinguish.** Empty repeated == absent; "any empty repeated field" omitted from JSON. Never fixed (3.15 covers singular scalars only) | ✓ via `["null", array]` union; and Avro **always writes** the field ("encodes a field even if its value is equal to its default") | ~ via `__isset` per field (1: `optional` vs default requiredness) | ✓ **all three, orthogonally**: `optionalProperties` (absent), `nullable` (null), `elements` (empty) | ✗ — "no notion of 'optional' fields"; pointer nullness via `hasFoo()` only | ~ table fields absent via vtable; **struct** fields "are required (so no defaults either)" | ✓ (`required` / `type:null` / `[]`); OAS 3.0 used bespoke `nullable`, 3.1 realigned — quote **NF** | n/a |
| **Q3 Enum unknown value** | **OPEN** (proto3/editions): value stored, field reports *set*. **CLOSED** (proto2): goes to unknown-field set, field reports *unset*. Switched "specifically because of the unexpected behavior that _closed_ enums cause". `_UNSPECIFIED = 0` mandatory | **CLOSED, hard error** unless reader declares `default` (added 1.9.0, AVRO-1340, **5 yrs** report→fix); AVRO-3313 says default didn't work → "Not A Bug" | i32 on wire; official behaviour **NF**. Secondary: "the parser will treat it like an unknown field" | **CLOSED, rejects.** No sentinel, no fallback, **no evolution story at all** | append-only enumerants ("number larger than all previous members") | append-only; "This requires code to handle forwards compatibility itself, by handling unknown enum values" | closed, rejects. No evolution model | enum evolution is what BACKWARD/FORWARD modes gate |
| **Q4 Strict or tolerant?** | **Split by encoding.** Binary: tolerant + preserving. **JSON: "should reject unknown fields by default"** | tolerant to writer's extras ("ignored"); **strict** on reader fields lacking defaults ("an error is signalled") | tolerant — "the old server simply ignores it and processes as normal" | **STRICT by default** — "does not allow additional properties by default"; opt-in via `additionalProperties: true`; one carve-out: the "discriminator tag exemption" | tolerant (bounds-check, no parse step) | tolerant (skip) | **TOLERANT by default** — "By default any additional properties are allowed"; tightening it broken under `allOf` → `unevaluatedProperties` added | policy layer, not a codec |
| **Q5 Version semantics** | **No version in data.** Pure field-level. `edition="2024"` versions the *language* | **No version — carries the WRITER'S SCHEMA with the data.** Evolution is a *pairwise relation*, not a ladder | **No version in data**; explicit two-layer split: "protocol encoding changes are safely isolated from interface definition version changes" | **NONE — no versioning, no evolution rules.** Only inert `metadata` | none; ordinals only | none | `$schema` versions the *language*, not the document | **versions the schema registry-side**, with a compatibility mode attached to a subject |
| **Q6 Field identity** | **numeric tag.** Never reuse; `reserved` numbers **must** be used. **"Reserved field names affect only the protoc compiler behavior and not runtime behavior"** | **NAME** (+ optional `aliases`; "An implementation **may** optionally use aliases"). **No `reserved` — NF** | **(field id, type)** pair. "good programming practice to always explicitly specify field identifiers". No `reserved` — **NF** | name only. No reserved/alias/reuse policy — **NF** | ordinal `@N` + type ID. Names freely changeable "as long as the type ID / ordinal numbers stay the same" | slot order or explicit `id`; **`deprecated`** keeps the slot AND kills the accessor — best `reserved` in the corpus | name only; no reserved — **NF** | n/a |
| **Q7 Unknown-field round-trip** | **✓ binary since 3.5** (removed in early proto3, restored). **✗ JSON.** Also lost by field-by-field copying | **✗ — "the writer's value for that field is ignored"**. Mitigated instead by shipping the writer's schema with the file | **✗** (skipped). Authoritative statement **NF** | n/a (validator). Generated structs drop by construction. **NF** | **✓** — "unknown field retention: yes" | **✗** — "unknown field retention: no" | n/a; survival depends entirely on consuming code | n/a |
| **Q8 Compatibility taxonomy** | **No backward/forward vocabulary.** Three tiers: wire-**unsafe** / wire-**safe** / wire-**compatible**. Symmetric by construction | **NF in the spec** — resolution rules only. The b/f vocabulary is Confluent's, not Avro's | **NF** — the §5.3 four-case analysis *is* the taxonomy, incl. an upgrade-ordering prescription | **NF — none** | none named | none named | **NF** | **the canonical taxonomy**: BACKWARD / FORWARD / FULL × TRANSITIVE, + NONE |
Ranked by how instructive they are.
**The decision:** proto2 and Thrift both shipped a `required` keyword enforced by the parser.
**The reversal:** proto3 removed it entirely. Protobuf's own best-practices doc:
"Never add a required field, instead add `// required` to document the API contract." [WF] — https://protobuf.dev/best-practices/dos-donts/
**The stated reason** — note that it is a reason about *time*, not about correctness:
"You never know how long a message type is going to last and whether someone will be forced to fill in your required field with an empty string or zero in four years when it's no longer logically required but the proto still says it is." [WF] — https://protobuf.dev/best-practices/dos-donts/
Proto2's own guide, which cannot remove the keyword, instead brands it:
"\"Required Is Forever\" As mentioned earlier **`required` must not be used for new fields**. Semantics for required fields should be implemented at the application layer instead." [WF] — https://protobuf.dev/programming-guides/proto2/
"It is nearly impossible to safely change a field from `required` to `optional`. If there is any chance that a stale reader exists, it will consider messages without this field to be incomplete and may reject or drop them." [WF] — https://protobuf.dev/programming-guides/proto2/
"A second issue with required fields appears when someone adds a value to an enum. In this case, the unrecognized enum value is treated as if it were missing, which also causes the required value check to fail." [WF] — https://protobuf.dev/programming-guides/proto2/
**The mechanism of harm, in full** — the Cap'n Proto FAQ's message-bus story, quoted at length in §1.5, ending:
"Things like this have actually happened. At Google. Many times." [RAW] — https://capnproto.org/faq.html
"the 'required' keyword in Protocol Buffers turned out to be a horrible mistake." [RAW] — https://capnproto.org/faq.html
**The prescription:**
"The right answer is for applications to do validation as-needed in application-level code. ... Low-level infrastructure that doesn't care about message content should not validate it at all." [RAW] — https://capnproto.org/faq.html
**The same shape, independently, in Thrift** — which never removed the keyword but documents it as a trap:
"Because of this behaviour, required fields drastically limit the options with regard to soft versioning." / "Because they must be present on read, the fields cannot be deprecated." / "If a required field would be removed (or changed to optional), the data are no longer compatible between versions." [WF] — https://thrift.apache.org/docs/idl
**The re-admission (the part most people miss):** editions did not restore `required`, but it had to model proto2's existing `required` fields, and it did so as a *named legacy feature* — an explicit quarantine rather than a deletion:
"Proto2 `required` fields that have been migrated to editions will also use the `field_presence` feature, but set to `LEGACY_REQUIRED`." [WF] — https://protobuf.dev/programming-guides/editions/
And the editions design doc names `required` as unfinished business twelve years on:
"We still have `required` and `group`, `packed` is not everywhere, and string accessors in C++ still return `const std::string&`." [WF] — https://raw.githubusercontent.com/protocolbuffers/protobuf/main/docs/design/editions/what-are-protobuf-editions.md
**The decision:** proto3 removed explicit presence for singular scalars — no `optional`, no `has_` methods, default-valued fields not serialized. The intended benefit was simpler, struct-like APIs.
**The failed workaround:** Google shipped `google.protobuf.Int32Value` and friends — boxed wrapper messages — as the presence substitute.
"Users have pointed to both efficiency and usability issues with the wrapper types." [WF] — https://raw.githubusercontent.com/protocolbuffers/protobuf/main/docs/implementing_proto3_presence.md
**The reversal, with its stated cause:**
"Presence tracking was added to proto3 in response to user feedback, both from inside Google and from open-source users." [WF] — https://raw.githubusercontent.com/protocolbuffers/protobuf/main/docs/implementing_proto3_presence.md
"Presence in proto3 uses exactly the same syntax and semantics as in proto2." [WF] — same
**Timeline:** experimental behind `--experimental_allow_proto3_optional` from v3.12.0; default from v3.15.0.
"Optional fields for proto3 are enabled by default, and no longer require the --experimental_allow_proto3_optional flag." [WF] — https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0
"Presence tracking for proto3 messages is enabled by default [since v3.15.0] release, formerly up until [v3.12.0] the `--experimental_allow_proto3_optional` flag was required." [WF] — https://protobuf.dev/programming-guides/field_presence/
**The implementation is itself a lesson in reversing safely** — rather than change descriptor semantics (which old tooling would misread), they encoded presence in a construct old tooling already handled correctly:
"Every proto3 optional field is placed into a one-field `oneof`. We call this a 'synthetic' oneof, as it was not present in the source `.proto` file." [WF] — https://raw.githubusercontent.com/protocolbuffers/protobuf/main/docs/implementing_proto3_presence.md
"existing proto3 reflection-based algorithms should correctly preserve presence for proto3 optional fields with no code changes." [WF] — same
**The final position is a full inversion of proto3's original stance:**
"We recommend always adding the `optional` label for proto3 basic types. This provides a smoother path to editions, which uses explicit presence by default." [WF] — https://protobuf.dev/programming-guides/field_presence/
"`optional` is recommended over _implicit_ fields for maximum compatibility with protobuf editions and proto2." [WF] — https://protobuf.dev/programming-guides/proto3/
**Elapsed: proto3 GA 2016 → default-on 3.15 (Feb 2021). Roughly five years.** And the reversal is *incomplete*: repeated and map fields still have no presence, so §1.1/Q2's empty-vs-absent collapse is permanent.
**The decision:** early proto3 discarded unknown fields at parse time. Contemporaneous external reaction (2014-06-17):
"Apparently, version 3 of Protocol Buffers, aka 'proto3', removes this feature. I honestly don't know what they're thinking." / "This feature has been absolutely essential in many of Google's internal systems." [WF] — https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html
**The reversal took 2.5 years of public argument.** Full timeline from issue #272 (opened 2015-04-07 by `joshuarubin`, closed 2017-12-11), all [RAW] from the GitHub API — https://github.com/protocolbuffers/protobuf/issues/272:
**The release note** [WF] — https://github.com/protocolbuffers/protobuf/releases/tag/v3.5.0:
"Unknown fields are now preserved in proto3 for most of the language implementations for proto3 by default." C++: "Proto3 messages are now preserving unknown fields by default. If you rely on unknowns fields being dropped. Please use DiscardUnknownFields() explicitly." Java: "...please use the DiscardUnknownFieldsParser API." Python: "Use `message.DiscardUnknownFields()` to drop unknown fields." Ruby: "Unknown fields are now preserved by default."
**The reversal broke people in the other direction too** — `kditrj2d`, 2019-03-18 [RAW]:
"I've just upgraded a C# application that uses protobuffers from version 3.4.0 to 3.6.1. The application relies on unknown fields not being preserved. Now by default they ARE preserved and I've seen a significant and unacceptable increase in memory consumption. (The ratio of known to unknown fields is about 1:5.)" [RAW] — https://github.com/protocolbuffers/protobuf/issues/272
**Lessons, stated plainly:**
Covered in §1.1/Q3. The one-line reason:
"Proto3 and editions use _open_ enums specifically because of the unexpected behavior that _closed_ enums cause." [WF] — https://protobuf.dev/programming-guides/enum/
**This one is notable as the reversal that went the OTHER way from R2/R3 and stuck.** Proto3 changed enum semantics and *did not have to change back*; editions preserves the choice as a per-type feature (`features.enum_type = CLOSED`) purely for proto2 migration. The failure mode it fixed — silent reordering of repeated enum fields, and unknown enum values tripping `required` checks — was concrete and demonstrable, unlike the presence/unknown-field removals, whose justification was implementer convenience.
Covered in §1.2/Q3. Reported 25/May/13 (AVRO-1340), resolved 02/May/18, shipped 1.9.0. Reported broken across seven releases (AVRO-3313), resolved **Not A Bug** 27/Sep/23.
"makes it difficult to use enum's because you can never add a enum value and keep old reader's compatible" [WF] — https://issues.apache.org/jira/browse/AVRO-1340
**Lesson: an evolution escape hatch that only fires in an explicitly-two-schema resolution path is one that most users will never reach.** Ten years elapsed between "adding an enum value breaks readers" being reported and the situation being considered settled — and it was settled by reclassification, not by a fix.
The largest reversal of all: the very idea of versioning the language with a `syntax` keyword.
"The last radical change to Protobuf (`syntax = \"proto3\";`) split the ecosystem." [WF] — https://raw.githubusercontent.com/protocolbuffers/protobuf/main/docs/design/editions/what-are-protobuf-editions.md
"Protobuf is one of Google's oldest and most successful toolchain projects. However, it was designed before we learned and internalized this lesson, making modernization difficult and haphazard." [WF] — same
"Protobuf Editions replace the proto2 and proto3 designations" [WF] — https://protobuf.dev/news/2023-06-29/
"Instead of the hardcoded behaviors in older versions, editions will represent a collection of 'features'" [WF] — https://protobuf.dev/news/2023-06-29/
**The lesson is directly about versioning strategy and is the most transferable single item in this document:** a coarse, global, mutually-exclusive version marker (`syntax = "proto2"` vs `"proto3"`) *forces every behavioural change to be bundled into a schism*, splits the ecosystem, and then cannot be undone without a third version. The replacement is **fine-grained, individually-defaulted, individually-overridable feature flags**, with a *date-named edition* that merely sets the defaults. Every proto3 decision that was later reversed (R2, R3, R4) had to be reversed *globally* because it was bundled into `syntax`.
A softer reversal: the keyword for "no extra members" turned out not to compose with `allOf`, which is JSON Schema's primary extension mechanism. The fix was an additional, differently-scoped keyword rather than a semantic change to the original.
"`additionalProperties` only recognizes properties declared in the same subschema as itself. So, `additionalProperties` can restrict you from 'extending' a schema using combining keywords such as allOf." [WF] — https://json-schema.org/understanding-json-schema/reference/object
"Because `additionalProperties` only recognizes properties declared in the same subschema, it considers anything other than 'street_address', 'city', and 'state' to be additional. Combining the schemas with allOf doesn't change that." [WF] — same
**Lesson: strictness keywords must be defined in terms of what the WHOLE schema evaluated, not what the LEXICALLY ENCLOSING subschema declared. Getting this scope wrong is not fixable in place once tools depend on it.**
The largest reversal in the field, and the one that reframes all the others. RFC 1122's "be liberal in what you accept" was IETF orthodoxy for thirty years; RFC 9413 (2023) is the IAB's retraction.
"Be strict when sending and tolerant when receiving. Implementations must follow specifications precisely when sending to the network, and tolerate faulty input from the network." [WF, §2] — https://www.rfc-editor.org/rfc/rfc9413.html
"An implementation that reacts to variations in the manner recommended in the robustness principle enters a pathological feedback cycle." [WF, §4.1] — same
"A flaw can become entrenched as a de facto standard. Any implementation of the protocol is required to replicate the aberrant behavior, or it is not interoperable." [WF, §4.1] — same
"Choosing to generate fatal errors for unspecified conditions instead of attempting error recovery can ensure that faults receive attention." [WF, §5.1] — same
**Critical reading — and this is where most citations of RFC 9413 go wrong.** RFC 9413 attacks tolerance of *non-conformant* input. It does **not** attack tolerance of *declared extension points*. Protobuf's unknown-field preservation, Avro's "ignore the writer's extra fields", and JTD's `additionalProperties: true` are all *specified* behaviours at *declared* extension points, and RFC 9413's remedy — §5's insistence on responsiveness and active exercise of extension points — is compatible with all of them. **The synthesis: be strict about the grammar, be tolerant at the extension points you declared, and exercise those extension points continuously so they do not ossify.**
People get these backwards because the words describe *what the SCHEMA can read*, not *what direction data travels*. Two rules make it unconfusable:
**Fix the reader. Ask what it can read.** - **BACKWARD compatible** = the **new reader** can read **old data**. (You can read *backwards in time*.) - **FORWARD compatible** = the **old reader** can read **new data**. (Old code can read data from the *future*.)
The Confluent definitions, which are the canonical ones [WF] — https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html:
Backward: "consumers using the new schema can read data produced with the last schema" Forward: "data produced with a new schema can be read by consumers using the last schema" Full: "Schemas are both backward and forward compatible."
Start from V1: `{"id": "x", "name": "n"}`.
**BACKWARD-compatible change — ADD an optional field.** V2 adds `"nickname"` (optional, defaulted).
**FORWARD-compatible change — REMOVE an optional field.** V2 deletes `"name"`.
**FULL — both.** Only add-optional-with-default and remove-optional-with-default, and only if readers on both sides tolerate unknown members. "you can upgrade the producers and consumers independently".
**Note the symmetry that makes the whole thing click:** *adding* a field is backward-compatible-and-forward-hostile; *removing* a field is forward-compatible-and-backward-hostile. **Tolerant readers are exactly what converts each of those into FULL.** That is the entire mechanism by which unknown-field tolerance buys independent deployability — and it is why protobuf, whose readers are unconditionally tolerant in binary, does not need the backward/forward vocabulary at all and instead uses the symmetric wire-safe/wire-unsafe tiers (§1.1/Q8).
BACKWARD_TRANSITIVE / FORWARD_TRANSITIVE / FULL_TRANSITIVE check against "all previously registered schemas"; the non-transitive variants check only the immediately preceding version. [WF] — same URL
**Non-transitive compatibility does not compose.** V1→V2 compatible and V2→V3 compatible does not imply V1→V3 compatible. Classic counterexample: V2 adds field `x` with default; V3 removes `x`. Each step is BACKWARD-clean against its predecessor; V3's reader against V1's data is fine, but a V2 reader against V3 data, or any accumulated-history replay, is not guaranteed. **For data at rest — where V1 files never go away — only TRANSITIVE has meaning.**
These formats are overwhelmingly RPC/streaming designs. The transfer is not uniform. Being explicit about the failures matters more than the successes.
**1. Numeric-tag identity is the single strongest idea — and JSON cannot have it.** Every binary format here (protobuf, Thrift, Cap'n Proto, FlatBuffers) made field identity numeric and made names free to change. In a JSON file the name *is* the identity, so you inherit Avro's position — name-based identity, renaming is breaking — without Avro's saving grace (see #2). **Practical consequence: a JSON-in-a-repo format should treat key names as immutable-forever from first publication, and needs a `reserved`/`deprecated` register maintained by convention and tooling, because the format gives you nothing.** Note protobuf's own admission that even *its* reserved-names half is toothless at runtime ("Reserved field names affect only the protoc compiler behavior and not runtime behavior") — you would be building the enforcement that protobuf declined to build.
**2. Ship the schema (or a schema pointer) WITH the data — Avro's central idea, and it fits files better than it fits RPC.** *"a reader of Avro data ... can always parse that data because the original schema must be provided along with the data."* In RPC this is expensive per message, which is why Confluent invented the 5-byte schema-id prefix. **In a git repository it is nearly free**: the file can carry a `$schema`-style pointer or a version marker, and — uniquely for the repo case — *the schema's own history is in the same git history as the data*. This is a structural advantage over every format studied. Avro's model (writer's schema travels with the data, reader resolves against it) is the closest analogue and the one to copy.
**3. TRANSITIVE compatibility is the only meaningful setting.** Files written in 2019 are still sitting in the repo in 2026. Non-transitive checking, Confluent's *default*, is designed for a streaming world where old messages age out of retention. **That assumption is exactly false for a repository.** Adopt FULL_TRANSITIVE as the mental model even if nothing enforces it.
**4. Tagged unions only — and the tag must be unambiguous by construction.** Every format here rejects untagged unions except JSON Schema, and JSON Schema's `discriminator` is a *hint* over untagged semantics, which is precisely where its known problems come from. JTD's method — forbid the ambiguous schema *at the syntactic level* rather than defining a tie-break — is the right one, and it is cheap to adopt: one reserved tag key, present in every variant, never redefined by a variant.
**5. Open enums with a documented unknown-handling rule.** Protobuf switched *to* open enums and never switched back; the failure modes it fixed (silent reordering, interaction with required checks) are concrete. Avro spent ten years failing to retrofit the equivalent. **For a file format read by third-party tools you cannot upgrade, closed enums mean every vocabulary addition is a breaking change for every stale reader.** Decide the unknown-value rule up front — pass through / map to sentinel / reject — because retrofitting it is what AVRO-1340 and AVRO-3313 document as a decade of pain.
**6. The `required`-is-forever lesson transfers with full force, and arguably harder.** The Cap'n Proto message-bus story is about *validation baked into the parser propagating failure to parties that do not care about the content*. In a repo read by third-party tools this is worse than in RPC, because you cannot deploy a fix to the third-party tools at all. **Validation belongs in the application layer, downstream of parsing, and never in the load path of a generic tool.**
**7. Thrift's two-layer split — version the envelope, evolve the content.** *"protocol encoding changes are safely isolated from interface definition version changes."* For a file format: a small, explicit, coarse version marker for the *container/encoding* (which you will change rarely and can gate on), plus field-level compatibility rules for the *content* (which changes constantly and must never gate). Conflating these is exactly the `syntax = "proto3"` mistake (R6).
**8. R6's lesson about versioning strategy.** Do not create a global version marker whose value changes *behaviour* in bundles. Protobuf spent a decade unwinding it. If behaviour must be switchable, switch it per-field/per-type with individual defaults.
**9. Unknown-field preservation.** The *motivation* transfers perfectly and is arguably stronger for files: the read-modify-write cycle (tool reads file → edits one field → writes file back) is *the* dominant access pattern for config-in-a-repo, and it is precisely the pattern that destroys unknown fields. `matthewrj`'s formulation is exactly the repo case: *"C can't tell if one of the new fields was set to the default value or if B is just out of date and lost data."* In a repo, "B lost data" shows up as a **spurious git diff that silently deletes keys** — visible in review, at least, which is more than the RPC case gets.
*The caveat:* the *mechanism* does not transfer for free. Protobuf preserves unknown fields because its generated code carries an `UnknownFieldSet` alongside the typed struct. A JSON consumer that deserializes into a typed struct — which is what every code generator, including JTD's, produces — **drops unknown keys by construction**, and you cannot make third-party tools carry an unknown-field bag. So: state the preservation *requirement* normatively for writers, provide a preserving reference implementation, and **design the format so that the damage from a non-preserving tool is visible** (i.e. it shows up as deleted keys in a diff) rather than silent.
**10. Strict-vs-tolerant.** RFC 9413's critique is real but it is aimed at tolerance of *malformed* input, not at *declared extension points*. The correct posture for a repo format: **strict about grammar and about the values of keys you own; tolerant at explicitly-declared extension points; and exercise the extension points continuously so they do not ossify.** JTD's default (reject unknown members unless `additionalProperties: true`) is the better *starting* default — because the opt-in is explicit and auditable — but the opt-in must actually be used for the extensible parts, and the preservation requirement (#9) must accompany it.
**11. "Just use binary."** Protobuf's answer to every JSON-fidelity problem is *"Use binary; avoid using text formats for data exchange."* That option does not exist here. **Every unknown-field, name-stability, and default-value problem protobuf solves by pointing at the binary encoding lands squarely on us.** Concretely, we inherit ProtoJSON's whole problem set: names embedded in the data, unknown fields not propagated, and a parser whose default is to *reject*.
**12. Deployment-ordering remedies.** Thrift's "roll out the new server before the new clients" and Confluent's whole "upgrade first" column assume you control the deployment of both sides. **With third-party tools reading files from a git repo, you control neither side and there is no ordering to prescribe.** Every FORWARD-compatibility problem that RPC solves by deploying producers first is, for us, unsolvable operationally and must be solved in the format. This makes forward compatibility (old tools reading new files) *structurally more important* for us than for any of the studied formats, and it is the one they all treat as the weaker requirement.
**13. Central schema registry with enforcement.** Confluent's compatibility modes are *enforced at registration time by a server that can reject your schema*. There is no such chokepoint for files in a repo — the "registry" is a code review at best, a merge at worst. **The taxonomy is valuable as vocabulary and as CI-check design; the enforcement model is not available.** (Partial mitigation unique to the repo case: CI *can* replay every historical version of the schema against the current one, which is a genuinely feasible TRANSITIVE check, and is more than most streaming shops manage.)
**14. Positional / index-based anything.** Avro's binary union branch index, FlatBuffers' positional union discriminants and slot ordering, Cap'n Proto's ordinals — all depend on a canonical field ordering that a JSON document does not have and should not acquire. **Do not build ordering-dependent semantics into a JSON format.** (Note Avro's own JSON encoding abandons the branch index for a type-name tag, for exactly this reason. Where Avro's binary and JSON encodings disagree, follow the JSON one.)
**15. Message-size and parse-cost trade-offs.** Cap'n Proto's zero-copy arena design, FlatBuffers' random access, protobuf's varints — all of these shaped evolution rules (e.g. Cap'n Proto's "you can't resize a list", its refusal of optional fields to avoid per-field overhead). **None of these pressures apply to files in a repo, and any rule justified by them should be discarded rather than copied.** In particular: Cap'n Proto's "give the field a bogus default value and interpret that value to mean 'not present'" is a wire-efficiency hack. In JSON, absence is free and unambiguous — use it, and do not invent sentinel values.
**16. `oneof`'s unknown-variant ambiguity is avoidable for us — do not inherit it.** Protobuf cannot distinguish "oneof unset" from "oneof set to a variant I don't know" because the unknown variant's tag goes to the unknown-field set. **A JSON tagged union does not have this problem**: the tag key is present and its value is a string the reader can see and report, even when unrecognised. This is a case where the JSON representation is *strictly better* than the binary one, and copying protobuf's `oneof` semantics wholesale would import a limitation that does not apply.
All fetched **2026-08-09**. Version/date column records what the document itself claims.
| URL | Doc version / date | Notes |
| https://www.rfc-editor.org/rfc/rfc8927.txt | RFC 8927, November 2020 | 2333 lines. §1 (goals), §2.2.8 (discriminator constraints), §3.1 (additionalProperties), §3.3.6 (properties), §3.3.8, Appendix A (omitted features) |
| https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/versions/3.0.3.md | OAS 3.0.3 | 3454 lines. Discriminator Object §4.7.25 at lines 2693–2790; composition/polymorphism note at 2350–2360 |
| https://thrift.apache.org/static/files/thrift-20070401.pdf | Thrift whitepaper, 2007-04-01, Facebook | 8 pages. §5 Versioning (5.1 field identifiers, 5.3 case analysis, 5.4 protocol versioning); `__isset` design |
| https://capnproto.org/faq.html | undated, current | "How do I make a field 'required'…" and "How do I make a field optional?" — the message-bus outage narrative in full |
| https://api.github.com/repos/protocolbuffers/protobuf/issues/272/comments | issue opened 2015-04-07, closed 2017-12-11, 69 comments | The unknown-fields reversal thread. Key comments: liujisi 2016-04-20, xfxyjwf 2016-06-12, liujisi 2016-11-29 / 2017-03-13 / 2017-09-14 / 2017-12-11, acozzette 2018-07-17 |
| https://diwakergupta.github.io/thrift-missing-guide/thrift.pdf | *Thrift: The Missing Guide*, Diwaker Gupta, undated | **Secondary source.** §1.3 Enums |
| URL | Doc version / date | Notes |
| https://protobuf.dev/programming-guides/proto3/ | current | Updating A Message Type; reserved; unknown fields; optional; enums; JSON pointer |
| https://protobuf.dev/programming-guides/proto2/ | current | "Required Is Forever"; required↔optional; required×enum interaction; full rule list |
| https://protobuf.dev/programming-guides/field_presence/ | current | presence definitions; disciplines; 3.15/3.12 timeline; JSON/null discussion |
| https://protobuf.dev/programming-guides/enum/ | current | open vs closed; the "specifically because of the unexpected behavior" rationale; repeated-enum reordering; maps |
| https://protobuf.dev/best-practices/dos-donts/ | current | **note: NOT `/programming-guides/dos-donts/`, which 404s** |
| https://protobuf.dev/programming-guides/json/ | current | ProtoJSON: default-value omission; null parsing; **"should reject unknown fields by default"**; enums |
| https://protobuf.dev/programming-guides/editions/ | current | `field_presence`, `LEGACY_REQUIRED`, `enum_type = CLOSED` |
| https://protobuf.dev/news/2023-06-29/ | 2023-06-29 | editions announcement |
| https://raw.githubusercontent.com/protocolbuffers/protobuf/main/docs/design/editions/what-are-protobuf-editions.md | main branch | "split the ecosystem"; "We still have `required` and `group`" |
| https://raw.githubusercontent.com/protocolbuffers/protobuf/main/docs/implementing_proto3_presence.md | main branch | presence restoration rationale; synthetic oneof |
| https://github.com/protocolbuffers/protobuf/releases/tag/v3.5.0 | v3.5.0, Nov 2017 | unknown-field restoration, per language |
| https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0 | v3.15.0, Feb 2021 | proto3 optional default-on |
| https://avro.apache.org/docs/1.11.1/specification/ | Avro 1.11.1 | Schema Resolution; union binary+JSON encoding; field `default`; enum `default`; aliases |
| https://avro.apache.org/docs/1.12.0/specification/ | Avro 1.12.0 | "schema must be provided along with the data"; JSON encoding notes |
| https://issues.apache.org/jira/browse/AVRO-1340 | reported 25/May/13, resolved 02/May/18, fix 1.9.0 | enum default proposal |
| https://issues.apache.org/jira/browse/AVRO-3313 | affects 1.9.0–1.11.0, resolved **Not A Bug** 27/Sep/23 | enum default reportedly non-functional |
| https://thrift.apache.org/docs/idl | current | field id grammar; three requiredness levels; the required/soft-versioning critique |
| https://thrift.apache.org/docs/types | current | **returned NOT FOUND for enums and unions** |
| https://capnproto.org/language.html | current | Evolving Your Protocol: allowed/forbidden lists; union definition |
| https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html | 2014-06-17 | feature matrix incl. "Unknown field retention"; the proto3 remark |
| https://flatbuffers.dev/schema/ | current | `id`, `deprecated`, struct limitations, enum forward-compat |
| https://flatbuffers.dev/evolution/ | current | add-to-end, no-removal, union variant ordering, renaming |
| https://json-schema.org/understanding-json-schema/reference/object | current draft docs | additionalProperties default; the allOf scoping failure; unevaluatedProperties |
| https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html | current | the seven compatibility types; default BACKWARD; upgrade ordering |
| https://www.rfc-editor.org/rfc/rfc9413.html | RFC 9413, June 2023 (IAB) | robustness principle critique |
| https://yokota.blog/2021/08/26/understanding-protobuf-compatibility/ | 2021-08-26 | **secondary.** oneof compatibility; the NOT_SET ambiguity |
| https://ajv.js.org/json-type-definition.html | current | **secondary.** JTD strictness; what JTD cannot express |
| URL | Status |
| `https://docs.google.com/document/d/1KMRX-G91Aa-Y2FkEaHeeviLRRNblgIahbsk4wA14gRk/edit` | **The protobuf unknown-fields restoration design doc**, linked by maintainer `liujisi` 2017-03-13 in issue #272. Not publicly fetchable. Its contents are known only through the quoted excerpt in `jbolla`'s 2017-09-13 comment: *"3.4 release (ETA: Q3 2017): Google protobuf implementation for each language will provide APIs to explicitly drop or preserve unknowns for proto3. A temporary flag will be introduced for the default parsing behavior - default to drop unknowns."* **This is the highest-value document I could not obtain.** |
| `https://jsontypedef.com/` , `https://jsontypedef.com/docs/` , `https://jsontypedef.com/docs/jtd-in-5-minutes/` | **Domain lost.** `/docs/*` 404s; the root now serves unrelated casino-affiliate content. JTD's authored documentation site no longer exists. Use RFC 8927 + `github.com/jsontypedef`. |
| `https://spec.openapis.org/oas/v3.1.0.html` , `https://spec.openapis.org/oas/v3.0.3.html` | Fetched but **truncated before the Discriminator Object section**. Use the GitHub raw markdown instead (listed above). |
| `https://protobuf.dev/programming-guides/dos-donts/` | 404 — the correct path is `/best-practices/dos-donts/`. |