Protobuf - An alternative approach to JSON or XML serialization in C#
In a recent project at Lambda3, we developed a sync data tool in C# that uses a lot of serialization/deserialization processes.
Optimize those serializations/deserializations was one of the biggest challenges to increase overall performance and we decided to use Protocol Buffers or
protobuf to handle with it.
Protobuf is a protocol developed by Google to help than increase data traffic performance in an extensible and optimized way, overriding JSON and XML solutions.
How it works?
.protofile is created with a data structure definition to be used.
- This file is compiled to generate a class that represents this structure in your application, in this case, in
- Instantiate the class in the application and assign the data.
- A binary encoding is performed, using generated
- Byte array is then trafficked to a file or a service.
Let’s discuss it in more details.
.proto file is created, see an example below:
The structure itself is quite simple to create and understand, much like a
C# class, it contains:
- data types
- property names
- data positioning
For more details, see the .Proto Language Guide.
.proto the next step is to compile this structure to generate a file to be used by your application, in this case you’ll generate a
C# class using
protoc, the compiler that will read the definition and generate
Where to download protoc compiler?
Until that moment,
3.6.1 is the latest stable version.
You can download it by many sources, these are some of them:
dotnet global tool - protoc
dotnet tool install --global protoc --version 3.6.1
Chocolatey - protoc
choco install protoc --version 3.6.1
If your prefer, download
protoc.exe from oficial repository project at releases page.
protoc-3.6.1-win32.zip file and extract
bin/protoc.exe in a local folder, don’t forget to include
protoc.exe path on user’s environment variables, to make it accessible by terminal, as well as installers above.
Generating C# classes from .proto files
Now, just open a terminal and run the compilation command:
protoc --proto_path=<protos_path> --csharp_out=<out_folder> name.proto
protoc --proto_path=protos/ --csharp_out=Sync/Models/ Product.proto
The generated file is something like this sample at gist.
C# file can not be changed manually, only using
How to use generated C# classes
Add the generated class on your project, it depends on
Google.Protobuf package, so, add a referente to it on your project.
Install-Package Google.Protobuf -Version 3.6.1
The serialization process creates a byte array, using its own encoding format.
Here is an example of how the serialization would look with the following definition.
test1 file will contain only 3 bytes:
Why not use only XML or JSON?
JSON are excelent formats, they’re platform and language agnostics and, in most cases, easy to read and undersantand.
However, I listed a few points below, also taken from official documentation, so you can evaluate if
protoc fits to your case.
.protostructures are simple.
- are 3 to 10 times smaller in size.
- are 20 to 100 times faster
- generated classes are easy to integrate
Instead of passing strings in
XML and etc., which will have a
Parse process that can be costly in terms of processing and memory consumption,
protoc, in turn, optimizes enconding to store only properties content, already with their pre-defined positions specified at
.proto file, as well as other optimizations as
128 Varints, Signed Integers and etc.
JSON and PROTOC comparison
I’ve created a simple project to compare
JSON serialization process. You can download and run it locally.
The results showed that the performance of
protoc is exponentially greater than
JSON serialization, even though the two formats are serializing to a byte array. Look at the chart with the compiled results.
I hope it was useful, if you have already used some implementation of
protobuf comment on the blog.