avro序列化实例

左手的ㄟ右手 2022-04-03 11:06 318阅读 0赞

hadoop中，就是用的avro作为序列化的，现在按照官方文档，我们按照如下步骤来做这个实例。

1.  构建maven工程，加入apache-avro依赖库，以及插件依赖。
2.  编写avro文件
3.  生成java实体类
4.  编写测试程序

pom.xml配置文件

<dependencies>
        <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>4.9</version>
          <scope>test</scope>
        </dependency>
        <dependency>
          <groupId>org.apache.avro</groupId>
          <artifactId>avro</artifactId>
          <version>1.8.2</version>
        </dependency>
        <dependency>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
          <version>1.7.21</version>
        </dependency>
    </dependencies>
    
    <build>
        <plugins>
            <plugin>   
                <groupId>org.apache.avro</groupId>
                <artifactId>avro-maven-plugin</artifactId>
                <version>1.8.2</version>
                <executions>
                <execution>
                <phase>generate-sources</phase>
                <goals>
                    <goal>schema</goal>
                </goals>
                <configuration>	
                    <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>   
                    <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
                </configuration>
                </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

src/main/avro目录下编写user.avsc,注意后缀是avsc

{
      "namespace":"com.xxx.avro.entity",
      "type":"record",
      "name":"User",
      "fields":[
            {"name":"id","type":"int"},
            {"name":"name","type":"string"},
            {"name":"age","type":["int","null"]},
            {"name":"mobile","type":["string","null"]}
      ]
    }

这是一个用json格式表示的avro对象

因为我们的pom.xml配置文件中配置了avro生成java的插件，所以这里直接运行Run As ->Maven install，构建成功之后，会在com.xxx.avro.entity目录下生成User.java的文件。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70][]

User.java文件部分内容截图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70 1][]

接下来，编写测试验证：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70 2][]

运行结果：

![20181221221237603.png][]

eclipse下，使用avro-maven-plugin插件，工程中的pom.xml会报错： Plugin execution not covered by lifecycle configuration...这个错误提示上有解决办法，要么在pom.xml配置文件中增加配置，忽略phase，要么在lifecycle-mapping-metadata.xml配置文件中增加配置忽略phase，然后重新加载lifecycle-mapping-metadata.xml文件。这里两种解决办法都贴出来。

1、在pom.xml中增加配置：

<pluginManagement>
       <plugins>
          <plugin>
             <groupId>org.eclipse.m2e</groupId>
             <artifactId>lifecycle-mapping</artifactId>
             <version>1.0.0</version>
             <configuration>
                <lifecycleMappingMetadata>
                <pluginExecutions>
                   <pluginExecution>
                      <pluginExecutionFilter>
                      <groupId>org.apache.avro</groupId>
                      <artifactId>avro-maven-plugin</artifactId>
                      <versionRange>[1.8.2,)</versionRange>
                      <goals>
                      	<goal>schema</goal>
                      </goals>
                      </pluginExecutionFilter>
                      <action>
                      	<ignore></ignore>
                      </action>
                   </pluginExecution>
                </pluginExecutions>
                </lifecycleMappingMetadata>
             </configuration>
          </plugin>
       </plugins>
    </pluginManagement>

2、在lifecycle-mapping-metadata.xml中增加配置：

<lifecycleMappingMetadata>
      <pluginExecutions>
        <pluginExecution>
          <pluginExecutionFilter>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-maven-plugin</artifactId>
            <versionRange>1.8.2</versionRange>
            <goals>
              <goal>schema</goal>
            </goals>
          </pluginExecutionFilter>
          <action>
            <ignore />
          </action>
        </pluginExecution>
      </pluginExecutions>
    </lifecycleMappingMetadata>

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70]: /images/20220403/318255cbf9c549ac9ac8a7b66570c6a9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70 1]: /images/20220403/cfa0528bd0214904a05c91507255cb3e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70 2]: /images/20220403/46917e9fb1c04bfdb2afbf5301e0180f.png
[20181221221237603.png]: /images/20220403/3e9cbea7b49d4d9ca97e7c54a4c04af4.png