Programming Squirrel

Sharing the learning of a squirrel

Protocol Buffer aka (Protobuf)

Protocol Buffers is a data serialization system developed by Google for efficiently exchanging structured information between different systems. It uses a message definition language (.proto file) to describe the structure of the data and generates code in various programming languages to facilitate the serialization and deserialization of data based on that structure.

A little of History

Protocol Buffers (Protobuf) was developed by Google in the year 2001. To be more specific by a Google team led by Kenton Varda. It was created as an efficient way to exchange and store structured data, especially in distributed and high-performance environments.

The initial version of Protocol Buffers was internally used at Google for various applications. Subsequently, in July 2008, Google publicly released Protocol Buffers as an open-source technology and made it available to the general public.

Since then, Protocol Buffers has gained popularity in the software development community due to its efficiency, flexibility, and ability to generate code in multiple programming languages.

Protbuf vs JSON

Protobuf and JSON are both popular formas for data serialization, and each one has its own advantages and disadvantages, which one is better, well depends on the scenario of your specific project requirements.

ProtoBuf Vs JSON

Advantages

Data Size Efficiency

Faster Serialization and Deserialization

Schema Validation

Disadvantages

Human Readability

Interoperability with Other Systems

Flexibility in Data Structure

Let’s review every point.

Advantages

  • Data Size Efficiency: Protocol Buffers tends to generate more compact messages compared to JSON. This reduces the size of transmitted or stored data, resulting in better bandwidth utilization and lower storage consumption.
  • Faster Serialization and Deserialization: Due to its binary format and predefined structure, Protocol Buffers can serialize and deserialize data faster than JSON. This is especially beneficial in applications that require high-performance data transmission or processing.
  • Schema Validation: Protocol Buffers uses a schema defined in the .proto file, allowing for stricter verification of data structure and types. This helps prevent errors and ensures higher data integrity during communication.

Disadvantages

  • Human Readability: JSON is a human-readable format, making it easier to understand and debug data. On the other hand, Protocol Buffers messages are encoded in binary, making it less directly interpretable by humans.
  • Interoperability with Other Systems: JSON is widely supported across a variety of programming languages and platforms. While Protocol Buffers also has support in multiple languages, some systems or tools may not have native implementations for working with Protocol Buffers.
  • Flexibility in Data Structure: JSON allows for greater flexibility in data structure, as it can contain nested objects, dynamic objects, and additional properties. In contrast, Protocol Buffers requires a strict definition of the data schema in the .proto file, which can limit flexibility in certain cases.

Overall, Protocol Buffers excels in terms of size and speed efficiency, as well as schema validation capabilities. However, JSON is more human-readable and has broader compatibility. The choice between Protocol Buffers and JSON again depends on specific project requirements, such as performance, data size, interoperability, and ease of use.

Download and Install

If you want to use and try Protobuf you can download from https://github.com/protocolbuffers/protobuf and their documentation can be found on protpbuf.dev

How works Protobuf

I will not explain at detail, but the idea of protobuf as is mentioned above, is to serialize structured data using the .proto files.

Using these files protobuf generates a generated class (in the case of java) to persist data.

For example you have the next person.proto file:

package com.programingsquirrel;
message Person {
required string name = 1;
optional int32 age = 2;
}
view raw person.proto hosted with ❤ by GitHub

Then when we execute the below command in the terminal, (of cpourse after you have installed protobuf):

protoc --proto_path=src --java_out=build/gen src/person.proto

The option --java_out=build/gen src/person.proto means that the generated classes will be on the directory build/gen, and the src/person.proto means that is the .proto file will be processed. After execute the command the generated class will be like:

// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: person.proto
public final class PersonOuterClass {
private PersonOuterClass() {}
public static void registerAllExtensions(
com.google.protobuf.ExtensionRegistryLite registry) {
}
public static void registerAllExtensions(
com.google.protobuf.ExtensionRegistry registry) {
registerAllExtensions(
(com.google.protobuf.ExtensionRegistryLite) registry);
}
public interface PersonOrBuilder extends
// @@protoc_insertion_point(interface_extends:com.programingsquirrel.Person)
com.google.protobuf.MessageOrBuilder {
/**
* <code>required string name = 1;</code>
* @return Whether the name field is set.
*/
boolean hasName();
/**
* <code>required string name = 1;</code>
* @return The name.
*/
String getName();
/**
* <code>required string name = 1;</code>
* @return The bytes for name.
*/
com.google.protobuf.ByteString
getNameBytes();
/**
* <code>optional int64 age = 2;</code>
* @return Whether the age field is set.
*/
boolean hasAge();
/**
* <code>optional int64 age = 2;</code>
* @return The age.
*/
long getAge();
}
/**
* Protobuf type {@code com.programingsquirrel.Person}
*/
public static final class Person extends
com.google.protobuf.GeneratedMessageV3 implements
// @@protoc_insertion_point(message_implements:com.programingsquirrel.Person)
PersonOrBuilder {
private static final long serialVersionUID = 0L;
// Use Person.newBuilder() to construct.
private Person(com.google.protobuf.GeneratedMessageV3.Builder<?> builder) {
super(builder);
}
private Person() {
name_ = "";
}
@Override
@SuppressWarnings({"unused"})
protected Object newInstance(
UnusedPrivateParameter unused) {
return new Person();
}
public static final com.google.protobuf.Descriptors.Descriptor
getDescriptor() {
return PersonOuterClass.internal_static_com_programingsquirrel_Person_descriptor;
}
@Override
protected FieldAccessorTable
internalGetFieldAccessorTable() {
return PersonOuterClass.internal_static_com_programingsquirrel_Person_fieldAccessorTable
.ensureFieldAccessorsInitialized(
Person.class, Builder.class);
}
private int bitField0_;
public static final int NAME_FIELD_NUMBER = 1;
@SuppressWarnings("serial")
private volatile Object name_ = "";
/**
* <code>required string name = 1;</code>
* @return Whether the name field is set.
*/
@Override
public boolean hasName() {
return ((bitField0_ & 0x00000001) != 0);
}
/**
* <code>required string name = 1;</code>
* @return The name.
*/
@Override
public String getName() {
Object ref = name_;
if (ref instanceof String) {
return (String) ref;
} else {
com.google.protobuf.ByteString bs =
(com.google.protobuf.ByteString) ref;
String s = bs.toStringUtf8();
if (bs.isValidUtf8()) {
name_ = s;
}
return s;
}
}
/**
* <code>required string name = 1;</code>
* @return The bytes for name.
*/
@Override
public com.google.protobuf.ByteString
getNameBytes() {
Object ref = name_;
if (ref instanceof String) {
com.google.protobuf.ByteString b =
com.google.protobuf.ByteString.copyFromUtf8(
(String) ref);
name_ = b;
return b;
} else {
return (com.google.protobuf.ByteString) ref;
}
}
public static final int AGE_FIELD_NUMBER = 2;
private long age_ = 0L;
/**
* <code>optional int64 age = 2;</code>
* @return Whether the age field is set.
*/
@Override
public boolean hasAge() {
return ((bitField0_ & 0x00000002) != 0);
}
/**
* <code>optional int64 age = 2;</code>
* @return The age.
*/
@Override
public long getAge() {
return age_;
}
private byte memoizedIsInitialized = -1;
@Override
public final boolean isInitialized() {
byte isInitialized = memoizedIsInitialized;
if (isInitialized == 1) return true;
if (isInitialized == 0) return false;
if (!hasName()) {
memoizedIsInitialized = 0;
return false;
}
memoizedIsInitialized = 1;
return true;
}
@Override
public void writeTo(com.google.protobuf.CodedOutputStream output)
throws java.io.IOException {
if (((bitField0_ & 0x00000001) != 0)) {
com.google.protobuf.GeneratedMessageV3.writeString(output, 1, name_);
}
if (((bitField0_ & 0x00000002) != 0)) {
output.writeInt64(2, age_);
}
getUnknownFields().writeTo(output);
}
@Override
public int getSerializedSize() {
int size = memoizedSize;
if (size != -1) return size;
size = 0;
if (((bitField0_ & 0x00000001) != 0)) {
size += com.google.protobuf.GeneratedMessageV3.computeStringSize(1, name_);
}
if (((bitField0_ & 0x00000002) != 0)) {
size += com.google.protobuf.CodedOutputStream
.computeInt64Size(2, age_);
}
size += getUnknownFields().getSerializedSize();
memoizedSize = size;
return size;
}
@Override
public boolean equals(final Object obj) {
if (obj == this) {
return true;
}
if (!(obj instanceof Person)) {
return super.equals(obj);
}
Person other = (Person) obj;
if (hasName() != other.hasName()) return false;
if (hasName()) {
if (!getName()
.equals(other.getName())) return false;
}
if (hasAge() != other.hasAge()) return false;
if (hasAge()) {
if (getAge()
!= other.getAge()) return false;
}
if (!getUnknownFields().equals(other.getUnknownFields())) return false;
return true;
}
@Override
public int hashCode() {
if (memoizedHashCode != 0) {
return memoizedHashCode;
}
int hash = 41;
hash = (19 * hash) + getDescriptor().hashCode();
if (hasName()) {
hash = (37 * hash) + NAME_FIELD_NUMBER;
hash = (53 * hash) + getName().hashCode();
}
if (hasAge()) {
hash = (37 * hash) + AGE_FIELD_NUMBER;
hash = (53 * hash) + com.google.protobuf.Internal.hashLong(
getAge());
}
hash = (29 * hash) + getUnknownFields().hashCode();
memoizedHashCode = hash;
return hash;
}
public static Person parseFrom(
java.nio.ByteBuffer data)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data);
}
public static Person parseFrom(
java.nio.ByteBuffer data,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data, extensionRegistry);
}
public static Person parseFrom(
com.google.protobuf.ByteString data)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data);
}
public static Person parseFrom(
com.google.protobuf.ByteString data,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data, extensionRegistry);
}
public static Person parseFrom(byte[] data)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data);
}
public static Person parseFrom(
byte[] data,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
return PARSER.parseFrom(data, extensionRegistry);
}
public static Person parseFrom(java.io.InputStream input)
throws java.io.IOException {
return com.google.protobuf.GeneratedMessageV3
.parseWithIOException(PARSER, input);
}
public static Person parseFrom(
java.io.InputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws java.io.IOException {
return com.google.protobuf.GeneratedMessageV3
.parseWithIOException(PARSER, input, extensionRegistry);
}
public static Person parseDelimitedFrom(java.io.InputStream input)
throws java.io.IOException {
return com.google.protobuf.GeneratedMessageV3
.parseDelimitedWithIOException(PARSER, input);
}
public static Person parseDelimitedFrom(
java.io.InputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws java.io.IOException {
return com.google.protobuf.GeneratedMessageV3
.parseDelimitedWithIOException(PARSER, input, extensionRegistry);
}
public static Person parseFrom(
com.google.protobuf.CodedInputStream input)
throws java.io.IOException {
return com.google.protobuf.GeneratedMessageV3
.parseWithIOException(PARSER, input);
}
public static Person parseFrom(
com.google.protobuf.CodedInputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws java.io.IOException {
return com.google.protobuf.GeneratedMessageV3
.parseWithIOException(PARSER, input, extensionRegistry);
}
@Override
public Builder newBuilderForType() { return newBuilder(); }
public static Builder newBuilder() {
return DEFAULT_INSTANCE.toBuilder();
}
public static Builder newBuilder(Person prototype) {
return DEFAULT_INSTANCE.toBuilder().mergeFrom(prototype);
}
@Override
public Builder toBuilder() {
return this == DEFAULT_INSTANCE
? new Builder() : new Builder().mergeFrom(this);
}
@Override
protected Builder newBuilderForType(
BuilderParent parent) {
Builder builder = new Builder(parent);
return builder;
}
/**
* Protobuf type {@code com.programingsquirrel.Person}
*/
public static final class Builder extends
com.google.protobuf.GeneratedMessageV3.Builder<Builder> implements
// @@protoc_insertion_point(builder_implements:com.programingsquirrel.Person)
PersonOrBuilder {
public static final com.google.protobuf.Descriptors.Descriptor
getDescriptor() {
return PersonOuterClass.internal_static_com_programingsquirrel_Person_descriptor;
}
@Override
protected FieldAccessorTable
internalGetFieldAccessorTable() {
return PersonOuterClass.internal_static_com_programingsquirrel_Person_fieldAccessorTable
.ensureFieldAccessorsInitialized(
Person.class, Builder.class);
}
// Construct using com.programingsquirrel.PersonOuterClass.Person.newBuilder()
private Builder() {
}
private Builder(
BuilderParent parent) {
super(parent);
}
@Override
public Builder clear() {
super.clear();
bitField0_ = 0;
name_ = "";
age_ = 0L;
return this;
}
@Override
public com.google.protobuf.Descriptors.Descriptor
getDescriptorForType() {
return PersonOuterClass.internal_static_com_programingsquirrel_Person_descriptor;
}
@Override
public Person getDefaultInstanceForType() {
return Person.getDefaultInstance();
}
@Override
public Person build() {
Person result = buildPartial();
if (!result.isInitialized()) {
throw newUninitializedMessageException(result);
}
return result;
}
@Override
public Person buildPartial() {
Person result = new Person(this);
if (bitField0_ != 0) { buildPartial0(result); }
onBuilt();
return result;
}
private void buildPartial0(Person result) {
int from_bitField0_ = bitField0_;
int to_bitField0_ = 0;
if (((from_bitField0_ & 0x00000001) != 0)) {
result.name_ = name_;
to_bitField0_ |= 0x00000001;
}
if (((from_bitField0_ & 0x00000002) != 0)) {
result.age_ = age_;
to_bitField0_ |= 0x00000002;
}
result.bitField0_ |= to_bitField0_;
}
@Override
public Builder clone() {
return super.clone();
}
@Override
public Builder setField(
com.google.protobuf.Descriptors.FieldDescriptor field,
Object value) {
return super.setField(field, value);
}
@Override
public Builder clearField(
com.google.protobuf.Descriptors.FieldDescriptor field) {
return super.clearField(field);
}
@Override
public Builder clearOneof(
com.google.protobuf.Descriptors.OneofDescriptor oneof) {
return super.clearOneof(oneof);
}
@Override
public Builder setRepeatedField(
com.google.protobuf.Descriptors.FieldDescriptor field,
int index, Object value) {
return super.setRepeatedField(field, index, value);
}
@Override
public Builder addRepeatedField(
com.google.protobuf.Descriptors.FieldDescriptor field,
Object value) {
return super.addRepeatedField(field, value);
}
@Override
public Builder mergeFrom(com.google.protobuf.Message other) {
if (other instanceof Person) {
return mergeFrom((Person)other);
} else {
super.mergeFrom(other);
return this;
}
}
public Builder mergeFrom(Person other) {
if (other == Person.getDefaultInstance()) return this;
if (other.hasName()) {
name_ = other.name_;
bitField0_ |= 0x00000001;
onChanged();
}
if (other.hasAge()) {
setAge(other.getAge());
}
this.mergeUnknownFields(other.getUnknownFields());
onChanged();
return this;
}
@Override
public final boolean isInitialized() {
if (!hasName()) {
return false;
}
return true;
}
@Override
public Builder mergeFrom(
com.google.protobuf.CodedInputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws java.io.IOException {
if (extensionRegistry == null) {
throw new NullPointerException();
}
try {
boolean done = false;
while (!done) {
int tag = input.readTag();
switch (tag) {
case 0:
done = true;
break;
case 10: {
name_ = input.readBytes();
bitField0_ |= 0x00000001;
break;
} // case 10
case 16: {
age_ = input.readInt64();
bitField0_ |= 0x00000002;
break;
} // case 16
default: {
if (!super.parseUnknownField(input, extensionRegistry, tag)) {
done = true; // was an endgroup tag
}
break;
} // default:
} // switch (tag)
} // while (!done)
} catch (com.google.protobuf.InvalidProtocolBufferException e) {
throw e.unwrapIOException();
} finally {
onChanged();
} // finally
return this;
}
private int bitField0_;
private Object name_ = "";
/**
* <code>required string name = 1;</code>
* @return Whether the name field is set.
*/
public boolean hasName() {
return ((bitField0_ & 0x00000001) != 0);
}
/**
* <code>required string name = 1;</code>
* @return The name.
*/
public String getName() {
Object ref = name_;
if (!(ref instanceof String)) {
com.google.protobuf.ByteString bs =
(com.google.protobuf.ByteString) ref;
String s = bs.toStringUtf8();
if (bs.isValidUtf8()) {
name_ = s;
}
return s;
} else {
return (String) ref;
}
}
/**
* <code>required string name = 1;</code>
* @return The bytes for name.
*/
public com.google.protobuf.ByteString
getNameBytes() {
Object ref = name_;
if (ref instanceof String) {
com.google.protobuf.ByteString b =
com.google.protobuf.ByteString.copyFromUtf8(
(String) ref);
name_ = b;
return b;
} else {
return (com.google.protobuf.ByteString) ref;
}
}
/**
* <code>required string name = 1;</code>
* @param value The name to set.
* @return This builder for chaining.
*/
public Builder setName(
String value) {
if (value == null) { throw new NullPointerException(); }
name_ = value;
bitField0_ |= 0x00000001;
onChanged();
return this;
}
/**
* <code>required string name = 1;</code>
* @return This builder for chaining.
*/
public Builder clearName() {
name_ = getDefaultInstance().getName();
bitField0_ = (bitField0_ & ~0x00000001);
onChanged();
return this;
}
/**
* <code>required string name = 1;</code>
* @param value The bytes for name to set.
* @return This builder for chaining.
*/
public Builder setNameBytes(
com.google.protobuf.ByteString value) {
if (value == null) { throw new NullPointerException(); }
name_ = value;
bitField0_ |= 0x00000001;
onChanged();
return this;
}
private long age_ ;
/**
* <code>optional int64 age = 2;</code>
* @return Whether the age field is set.
*/
@Override
public boolean hasAge() {
return ((bitField0_ & 0x00000002) != 0);
}
/**
* <code>optional int64 age = 2;</code>
* @return The age.
*/
@Override
public long getAge() {
return age_;
}
/**
* <code>optional int64 age = 2;</code>
* @param value The age to set.
* @return This builder for chaining.
*/
public Builder setAge(long value) {
age_ = value;
bitField0_ |= 0x00000002;
onChanged();
return this;
}
/**
* <code>optional int64 age = 2;</code>
* @return This builder for chaining.
*/
public Builder clearAge() {
bitField0_ = (bitField0_ & ~0x00000002);
age_ = 0L;
onChanged();
return this;
}
@Override
public final Builder setUnknownFields(
final com.google.protobuf.UnknownFieldSet unknownFields) {
return super.setUnknownFields(unknownFields);
}
@Override
public final Builder mergeUnknownFields(
final com.google.protobuf.UnknownFieldSet unknownFields) {
return super.mergeUnknownFields(unknownFields);
}
// @@protoc_insertion_point(builder_scope:com.programingsquirrel.Person)
}
// @@protoc_insertion_point(class_scope:com.programingsquirrel.Person)
private static final Person DEFAULT_INSTANCE;
static {
DEFAULT_INSTANCE = new Person();
}
public static Person getDefaultInstance() {
return DEFAULT_INSTANCE;
}
@Deprecated public static final com.google.protobuf.Parser<Person>
PARSER = new com.google.protobuf.AbstractParser<Person>() {
@Override
public Person parsePartialFrom(
com.google.protobuf.CodedInputStream input,
com.google.protobuf.ExtensionRegistryLite extensionRegistry)
throws com.google.protobuf.InvalidProtocolBufferException {
Builder builder = newBuilder();
try {
builder.mergeFrom(input, extensionRegistry);
} catch (com.google.protobuf.InvalidProtocolBufferException e) {
throw e.setUnfinishedMessage(builder.buildPartial());
} catch (com.google.protobuf.UninitializedMessageException e) {
throw e.asInvalidProtocolBufferException().setUnfinishedMessage(builder.buildPartial());
} catch (java.io.IOException e) {
throw new com.google.protobuf.InvalidProtocolBufferException(e)
.setUnfinishedMessage(builder.buildPartial());
}
return builder.buildPartial();
}
};
public static com.google.protobuf.Parser<Person> parser() {
return PARSER;
}
@Override
public com.google.protobuf.Parser<Person> getParserForType() {
return PARSER;
}
@Override
public Person getDefaultInstanceForType() {
return DEFAULT_INSTANCE;
}
}
private static final com.google.protobuf.Descriptors.Descriptor
internal_static_com_programingsquirrel_Person_descriptor;
private static final
com.google.protobuf.GeneratedMessageV3.FieldAccessorTable
internal_static_com_programingsquirrel_Person_fieldAccessorTable;
public static com.google.protobuf.Descriptors.FileDescriptor
getDescriptor() {
return descriptor;
}
private static com.google.protobuf.Descriptors.FileDescriptor
descriptor;
static {
String[] descriptorData = {
"\n\014person.proto\022\026com.programingsquirrel\"#" +
"\n\006Person\022\014\n\004name\030\001 \002(\t\022\013\n\003age\030\002 \001(\003"
};
descriptor = com.google.protobuf.Descriptors.FileDescriptor
.internalBuildGeneratedFileFrom(descriptorData,
new com.google.protobuf.Descriptors.FileDescriptor[] {
});
internal_static_com_programingsquirrel_Person_descriptor =
getDescriptor().getMessageTypes().get(0);
internal_static_com_programingsquirrel_Person_fieldAccessorTable = new
com.google.protobuf.GeneratedMessageV3.FieldAccessorTable(
internal_static_com_programingsquirrel_Person_descriptor,
new String[] { "Name", "Age", });
}
// @@protoc_insertion_point(outer_class_scope)
}
view raw PersonOuterClass.java hosted with ❤ by GitHub

Conclusion

In conclusion it looks a great way to serialize Strucutred Data, and of course they have some best practices that allows to give a better format in the generated java classes, and this is only for demostration purposes, if you can try for your self you can download our project from Github protobuf-example.

What do you think about this serialization option?, let’s us know your thoughs, comments and opinion.

Thanks for reading and Happy Learning!!!

←
→

Leave a comment