【Protobuf专题】（四）proto开发神器——Jprotobuf

时间 2021-02-15

标签 java git github 数组 maven ide 工具性能测试 gradle 栏目微服务繁體版

原文原文链接

0 前言

在JAVA团队内部交互数据时，有几种经常使用的方法，好比JAVA原生序列化方法、Json序列化方法、Protobuf序列化方法等，对比实验详见：几种序列化方式的性能比较。java

从对比实验可知protobuf性能最佳，由于其高效的序列化性能以及优秀的跨平台能力，被普遍应用于跨平台之间的数据交互。但在不须要考虑跨平台的数据交互时，protobuf的繁琐流程成了烦人的“污点”。git

基于ptoro传输数据的通常流程是，先编写proto文件，再手动编译proto文件生成java类，而后实例化java类并填充数据，最后编译成二进制数据进行传输。其中编写及手动编译proto文件是个较繁琐的过程（好比复杂项目中proto之间的相互引用），并且由proto生成的java类使用的是建立者模式（即Bulider模式），其实例化后的赋值过程与项目历史代码风格不一致，为带来较大的改动。github

为解决这个问题，本文将介绍百度团队开发的Jprotobuf工具，并总结使用过程当中的注意事项，以及经过实验验证其与谷歌团队开发的Protobuf功能的一致性。数组

1 Jprotobuf介绍

1.1 定义

jprotobuf是针对JAVA程序开发一套简易类库，目的是简化JAVA语言对protobuf类库的使用。使用jprotobuf能够无需再去了解proto文件操做与语法，直接使用JAVA注解定义字段类型便可。maven

github地址： https://github.com/jhunters/j...

1.2 原理

jprotobuf工做原理以下：ide

扫描类上的注解的信息，进行分析(与protobuf读取proto文件进行分析过程类似)
根据注解分析的结果，动态生成java代码进行protobuf序列化与反序列化的功能实现
使用JDK6及以上的 code compile API进行编译后加载到classloader

1.3 性能

jprotobuf 主要性能消耗在扫描类上注解，动态生成代码编译的过程。在执行序列化与反序列化的过程当中，几乎与protobuf生成的代码效率等同。若是使用预编译插件，则无需在运行中进行代码生成与编译，效率更高。工具

1.4 特色

无需编写proto文件及繁琐的手工编译过程，支持基于POJO对象的注解方式，方便快捷。
支持protobuf全部类型，包括对象嵌套，数组，枚举类型
提供根据proto文件，动态生成代理对象，可省去POJO对象的编写工做。
完整支持proto文件全部功能，包括内联对象，匿名对象，枚举类型
提供从POJO对象的注解方式自动生成proto文件的功能，方便proto描述文件的管理与维护
提供预编译Maven插件，进一步提高运行性能
新增预编译gradle插件
2.x版本。支持TimeStamp类型, 与原生protobuf保持一致。支持Date类型，使用long类型传递 docs

2 Jprotobuf安装

maven代码以下：性能

<!-- jprotobuf -->
<dependency>
    <groupId>com.baidu</groupId>
    <artifactId>jprotobuf</artifactId>
    <version>2.4.5</version>
</dependency>
<dependency>
    <groupId>com.baidu</groupId>
    <artifactId>jprotobuf-precompile-plugin</artifactId>
    <version>2.2.2</version>
</dependency>

3 案例数据

测试思路：经过各自的方法构建相同的测试数据，进行序列化和反序列比较二者之间的差别。
注意事项：设计的案例数据尽可能包含全部的数据类型。测试

3.1 Protobuf数据定义

组织结构如图，其中data依赖student，student依赖person，teacher做为第三方扩展包，用于测试扩展功能。
gradle

data.proto代码以下：

// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.messages;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "SchoolModel";
option java_multiple_files = true;
// import packages
import "google/protobuf/any.proto";
import "practiceProto/categories/student.proto";
message School {
  message Location{
    string name=1;
    uint32 id=2;
  }
  Location schoolLocation = 1;
  bool isOpen =2;
  repeated categories.Student allStudents = 3;
  google.protobuf.Any extend =4;
}

student.proto代码以下：

// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.categories;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "StudentModel";
option java_multiple_files = true;
// import packages
import "practiceProto/base/person.proto";
message Student {
  base.Person baseInfo = 1;
  fixed32 calssId = 2;
  sint32 score = 3;
}

person.proto代码以下：

// Google Protocol Buffers Version 3.
syntax = "proto3";
// Package name.
package prcticeProto.base;
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "PersonModel";
option java_multiple_files = true;
message Person{
  message Location{
    string placeName=1;
    uint64 placeId=2;
  }
  enum Gender{
    man=0;
    woman=1;
  }
  string name = 1;
  int32 age=2;
  Gender gender=3;
  float height=4;
  double weight=5;
  Location location=6;
}

teacher.proto代码以下：

// Google Protocol Buffers Version 3.
syntax = "proto3";
// Options for code generation.
option java_package = "learnProto.practiceTest.protoModel";
option java_outer_classname = "TeacherModel";
option java_multiple_files = true;
message Teacher{
  string name = 1;
  int32 age=2;
}

3.2 Jprotobuf数据定义

组织结构如图所示。

Jprotobuf的定义语法详见： https://github.com/jhunters/j...

school代码以下：

package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.Any;
import com.baidu.bjf.remoting.protobuf.FieldType;
import com.baidu.bjf.remoting.protobuf.annotation.Protobuf;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;
import java.util.List;

@ProtobufClass
public class School {
 @ProtobufClass
 public static class Location{
        @Protobuf(fieldType= FieldType.STRING, order=1)
        private String name;
        @Protobuf(fieldType= FieldType.UINT32, order=2)
        private Integer id;
        public String getName() {
            return name;
        }
        public Location setName(String name) {
            this.name = name;
            return this;
        }
        public Integer getId() {
            return id;
        }
        public Location setId(Integer id) {
            this.id = id;
            return this;
        }
    }
    @Protobuf(fieldType= FieldType.OBJECT, order=1)
    private Location schoolLocation;
    @Protobuf(fieldType= FieldType.BOOL, order=2)
    private Boolean isOpen;
    @Protobuf(fieldType= FieldType.OBJECT, order=3)
    private List<Student> allStudents;
    @Protobuf(fieldType = FieldType.OBJECT,order = 4)
    private Any extend;
    public Any getExtend() {
        return extend;
    }
    public School setExtend(Any extend) {
        this.extend = extend;
        return this;
    }
    public Location getSchoolLocation() {
        return schoolLocation;
    }
    public School setSchoolLocation(Location schoolLocation) {
        this.schoolLocation = schoolLocation;
        return this;
    }
    public Boolean getOpen() {
        return isOpen;
    }
    public School setOpen(Boolean open) {
        isOpen = open;
        return this;
    }
    public List<Student> getAllStudents() {
        return allStudents;
    }
    public School setAllStudents(List<Student> allStudents) {
        this.allStudents = allStudents;
        return this;
    }
}

student代码以下：

package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.FieldType;
import com.baidu.bjf.remoting.protobuf.annotation.Protobuf;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;

@ProtobufClass
public class Student{
    @Protobuf(fieldType=FieldType.OBJECT, order=1)
    private Person baseInfo;
    @Protobuf(fieldType=FieldType.FIXED32, order=2)
    private Integer calssId;
    @Protobuf(fieldType=FieldType.SINT32, order=3)
    private Integer score;
    public Person getBaseInfo() {
        return baseInfo;
    }
    public Student setBaseInfo(Person baseInfo) {
        this.baseInfo = baseInfo;
        return this;
    }
    public Integer getCalssId() {
        return calssId;
    }
    public Student setCalssId(Integer calssId) {
        this.calssId = calssId;
        return this;
    }
    public Integer getScore() {
        return score;
    }
    public Student setScore(Integer score) {
        this.score = score;
        return this;
    }
}

person代码以下：

package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.FieldType;
import com.baidu.bjf.remoting.protobuf.annotation.Protobuf;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;

@ProtobufClass
public class Person {
    @ProtobufClass
 public enum Gender {
        MAN(0),WOMAN(1);
        @Protobuf(fieldType= FieldType.INT32, order=1)
        private final Integer value;
        Gender(Integer value) { this.value = value; }
        public Integer toValue() { return this.value; }
        public Integer value() {
            return toValue();
        }
    }
 @ProtobufClass
 public static class Location{
        @Protobuf(fieldType= FieldType.STRING, order=1)
        private String placeName;
        @Protobuf(fieldType= FieldType.UINT64, order=2)
        private Long placeId;
        public String getPlaceName() {
            return placeName;
        }
        public Location setPlaceName(String placeName) {
            this.placeName = placeName;
            return this;
        }
        public Long getPlaceId() {
            return placeId;
        }
        public Location setPlaceId(Long placeId) {
            this.placeId = placeId;
            return this;
        }
    }
    @Protobuf(fieldType= FieldType.STRING, order=1)
    private String name;
    @Protobuf(fieldType= FieldType.INT32, order=2)
    private Integer age;
    @Protobuf(fieldType= FieldType.ENUM, order=3)
    private Gender gender;
    @Protobuf(fieldType= FieldType.FLOAT, order=4)
    private Float height;
    @Protobuf(fieldType= FieldType.DOUBLE, order=5)
    private Double weight;
    @Protobuf(fieldType= FieldType.OBJECT, order=6)
    private Location personLocation;
    public String getName() {
        return name;
    }
    public Person setName(String name) {
        this.name = name;
        return this;
    }
    public Integer getAge() {
        return age;
    }
    public Person setAge(Integer age) {
        this.age = age;
        return this;
    }
    public Gender getGender() {
        return gender;
    }
    public Person setGender(Gender gender) {
        this.gender = gender;
        return this;
    }
    public Float getHeight() {
        return height;
    }
    public Person setHeight(Float height) {
        this.height = height;
        return this;
    }
    public Double getWeight() {
        return weight;
    }
    public Person setWeight(Double weight) {
        this.weight = weight;
        return this;
    }
    public Location getPersonLocation() {
        return personLocation;
    }
    public Person setPersonLocation(Location personLocation) {
        this.personLocation = personLocation;
        return this;
    }
}

teacher代码以下：

package learnProto.jprotobuf.model;
import com.baidu.bjf.remoting.protobuf.annotation.ProtobufClass;

@ProtobufClass
public class Teacher {
    private Integer id;
    private String name;
    public Integer getId() {
        return id;
    }
    public Teacher setId(Integer id) {
        this.id = id;
        return this;
    }
    public String getName() {
        return name;
    }
    public Teacher setName(String name) {
        this.name = name;
        return this;
    }
}

4 案例测试

4.1 基本数据类型测试

public class jprotobufTest {
    public static void writeFileByte(byte[] b,String path) {
        File file = new File(path);
        long size = file.length();
        try (FileOutputStream fos = new FileOutputStream(file)) {
            fos.write(b);
            fos.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    public static byte[] readFileByte(String path) {
        File file = new File(path);
        long size = file.length();
        byte[] b = new byte[(int) size];
        try (InputStream fis = new FileInputStream(file)) {
            if (fis.read(b) < 0) {
                return null;
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
        return b;
    }
    public void protobuff(String path){
        // 实例化
 learnProto.practiceTest.protoModel.Person.Builder builderPerson = learnProto.practiceTest.protoModel.Person.newBuilder().setAge(10).setGender(learnProto.practiceTest.protoModel.Person.Gender.woman).setName("Tom").setHeight(100.00f).setWeight(100.00d).setLocation(
                learnProto.practiceTest.protoModel.Person.Location.newBuilder().setPlaceId(123l).setPlaceName("hubei")
        );
        learnProto.practiceTest.protoModel.School school = learnProto.practiceTest.protoModel.School.newBuilder()
                .setIsOpen(true)
                .setSchoolLocation(learnProto.practiceTest.protoModel.School.Location.newBuilder().setId(123).setName("hubei"))
                .addAllStudents(Student.newBuilder().setBaseInfo(builderPerson).setCalssId(10).setScore(-1))
                .addAllStudents(Student.newBuilder().setBaseInfo(builderPerson).setCalssId(10).setScore(1))
                .build();
        try {
            // 序列化
 byte[] bytes = school.toByteArray();
            writeFileByte(bytes,path);
            System.out.println("protobuf序列化后的数据：" + Arrays.toString(bytes)+",字节个数："+bytes.length);
            //反序列化
 byte[] bytes1 = readFileByte(path);
            learnProto.practiceTest.protoModel.School parseFrom = learnProto.practiceTest.protoModel.School.parseFrom(bytes1);
        } catch (InvalidProtocolBufferException e) {
            e.printStackTrace();
        }
    }
    public void jprotobuf(String path){
        // 实例化
 Person person = new Person().setAge(10).setGender(Person.Gender.MAN).setName("Tom").setHeight(100.00f).setWeight(100.00d).setPersonLocation(
                new Person.Location().setPlaceId(123l).setPlaceName("hubei")
        );
        ArrayList<learnProto.jprotobuf.model.Student> studentArrayList = new ArrayList<>();
        studentArrayList.add(new learnProto.jprotobuf.model.Student().setCalssId(10).setScore(-1).setBaseInfo(person));
        studentArrayList.add(new learnProto.jprotobuf.model.Student().setCalssId(10).setScore(1).setBaseInfo(person));
        School sch = new School().setOpen(true).setAllStudents(studentArrayList).setSchoolLocation(new School.Location().setId(123).setName("hubei"));
        try {
            // 序列化
 Codec<School> simpleTypeCodec = ProtobufProxy.create(School.class);
            byte[] b = simpleTypeCodec.encode(sch);
            writeFileByte(b,path);
            System.out.println("Jprotobuf序列化后的数据：" + Arrays.toString(b)+",字节个数："+b.length);
            // 反序列化
 byte[] bytes =readFileByte(path);
            School newStt = simpleTypeCodec.decode(bytes);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    @Test
 public void test(){
        protobuff("C:UsersadminDesktopProtodemo.pack");
        jprotobuf("C:UsersadminDesktopJprotodemo.pack");
    }
}

运行结果：

protobuf序列化后的数据：[10, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 16, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 1, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 1, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 2],字节个数：103
Jprotobuf序列化后的数据：[10, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 16, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 0, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 1, 26, 43, 10, 34, 10, 3, 84, 111, 109, 16, 10, 24, 0, 37, 0, 0, -56, 66, 41, 0, 0, 0, 0, 0, 0, 89, 64, 50, 9, 10, 5, 104, 117, 98, 101, 105, 16, 123, 21, 10, 0, 0, 0, 24, 2],字节个数：103

实验代表，Jprotobuf的序列化方法与Protobuf一致，能够放心使用。

4.2 第三方扩展测试

@Test
public void schooltest() throws IOException {
    // jproto
 Any any = Any.pack(new Teacher());
    School person = new School().setExtend(any);
    Codec<School> simpleTypeCodec = ProtobufProxy.create(School.class);
    try {
        // 序列化
 byte[] b = simpleTypeCodec.encode(person);
        writeFileByte(b,"C:UsersadminDesktopjproto.pack");
        System.out.println("jproto序列化后的数据：" + Arrays.toString(b)+",字节个数："+b.length);
        // 反序列化
 byte[] bytes =readFileByte("C:UsersadminDesktopjproto.pack");
        School newStt = simpleTypeCodec.decode(bytes);
    } catch (IOException e) {
        e.printStackTrace();
    }
    // proto
 com.google.protobuf.Any any1 = com.google.protobuf.Any.pack(learnProto.practiceTest.protoModel.Teacher.newBuilder().build());
    learnProto.practiceTest.protoModel.School builderPerson = learnProto.practiceTest.protoModel.School.newBuilder()
            .setExtend(any1)
            .build();
    try {
        // 序列化
 String path="C:UsersadminDesktopProtodemo.pack";
        byte[] bytes = builderPerson.toByteArray();
        writeFileByte(bytes,path);
        System.out.println("protobuf序列化后的数据：" + Arrays.toString(bytes)+",字节个数："+bytes.length);
        //反序列化
 byte[] bytes1 = readFileByte(path);
        learnProto.practiceTest.protoModel.School parseFrom = learnProto.practiceTest.protoModel.School.parseFrom(bytes1);
    } catch (InvalidProtocolBufferException e) {
        e.printStackTrace();
    }
}

运行结果：

jproto序列化后的数据：[34, 58, 10, 54, 116, 121, 112, 101, 46, 103, 111, 111, 103, 108, 101, 97, 112, 105, 115, 46, 99, 111, 109, 47, 108, 101, 97, 114, 110, 80, 114, 111, 116, 111, 46, 106, 112, 114, 111, 116, 111, 98, 117, 102, 46, 109, 111, 100, 101, 108, 46, 84, 101, 97, 99, 104, 101, 114, 18, 0],字节个数：60
protobuf序列化后的数据：[34, 29, 10, 27, 116, 121, 112, 101, 46, 103, 111, 111, 103, 108, 101, 97, 112, 105, 115, 46, 99, 111, 109, 47, 84, 101, 97, 99, 104, 101, 114],字节个数：31

实验代表，Jprotobuf的扩展功能能够正常使用，但其机制可能与protobuf有差别。

5 使用总结

一、字段的注解@Protobuf，能够不写，不写的状况下按照默认类型，order按照书写的顺序；也能够写，自定义类型和order。但必定不要有的字段写，有的字段不写，order会乱！

二、构建的@ProtobufClass不要带有有参构造器，实例化时也不能带参数构造。只能无参构造后，经过set或add赋值（与proto的实例化规范一致）。

三、能够调用_getIDL_得到IDL，即proto文件，但全部的嵌套结构都被拆开成了单独的结构。

四、 setter的返回类型能够设定为类自己，从而实现链式。如图：

五、 定义基本数据类型的数据时，要用包装类。好比对于整型，不要写int，而应该写INTGER，如上图中的id，若是设置为int，系统默认初始化了id=0，即便没有赋值，该数值在序列化时会被写入数据；若是设置为INTGER，没有赋值，不会被写入序列化的字节序中。（此处与proto不一样，proto中字段设定值等于默认值时，序列化时数据不会被写入，反序列化后全部没被赋值的字段都被赋对应类型的默认值；而jprotobuf彻底按照用户输入的意愿，序列化以前赋值什么，序列化时就有什么，反序列化时也对应有什么，若是字段没被赋值，反序列化后为null，enum为第一个枚举值）。

六、反序列化使用时注意：没被赋值的字段反序列化后值为null，并不是默认值。

七、设定了多个字段，有些没有赋值，并不会影响数据序列化后的大小。

6 参考文献

[1] 几种序列化方式的性能比较
[2] https://github.com/jhunters/j...
[3] https://github.com/jhunters/j...