8.Flume Log4J Appender

以你之姓@ 2022-06-14 01:41 145阅读 0赞

## Log4J Appender ##

Appends Log4j events to a flume agent’s avro source. A client using this appender must have the flume-ng-sdk in the classpath (eg, flume-ng-sdk-1.8.0-SNAPSHOT.jar). Required properties are in **bold**.

<table style="border:0px; border-collapse:collapse">      
 <thead> 
  <tr> 
   <th style="text-align:left; padding:1px 8px 1px 5px; background-color:rgb(238,221,238); border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Property Name</th> 
   <th style="text-align:left; padding:1px 8px 1px 5px; background-color:rgb(238,221,238); border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Default</th> 
   <th style="text-align:left; padding:1px 8px 1px 5px; background-color:rgb(238,221,238); border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> <strong>Hostname</strong></td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> –</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> The hostname on which a remote Flume agent is running with an avro source.</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> <strong>Port</strong></td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> –</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> The port at which the remote Flume agent’s avro source is listening.</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> UnsafeMode</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> false</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> If true, the appender will not throw exceptions on failure to send the events.</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> AvroReflectionEnabled</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> false</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Use Avro Reflection to serialize Log4j events. (Do not use when users log strings)</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> AvroSchemaUrl</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> –</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> A URL from which the Avro schema can be retrieved.</td> 
  </tr> 
 </tbody> 
</table>

Sample log4j.properties file:

#...
    log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
    log4j.appender.flume.Hostname = example.com
    log4j.appender.flume.Port = 41414
    log4j.appender.flume.UnsafeMode = true
    
    # configure a class's logger to output to the flume appender
    log4j.logger.org.example.MyClass = DEBUG,flume
    #...

By default each event is converted to a string by calling toString(), or by using the Log4j layout, if specified.

If the event is an instance of org.apache.avro.generic.GenericRecord, org.apache.avro.specific.SpecificRecord, or if the property AvroReflectionEnabled is set to truethen the event will be serialized using Avro serialization.

Serializing every event with its Avro schema is inefficient, so it is good practice to provide a schema URL from which the schema can be retrieved by the downstream sink, typically the HDFS sink. If AvroSchemaUrl is not specified, then the schema will be included as a Flume header.

Sample log4j.properties file configured to use Avro serialization:

#...
    log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
    log4j.appender.flume.Hostname = example.com
    log4j.appender.flume.Port = 41414
    log4j.appender.flume.AvroReflectionEnabled = true
    log4j.appender.flume.AvroSchemaUrl = hdfs://namenode/path/to/schema.avsc
    
    # configure a class's logger to output to the flume appender
    log4j.logger.org.example.MyClass = DEBUG,flume
    #...

## Load Balancing Log4J Appender ##

Appends Log4j events to a list of flume agent’s avro source. A client using this appender must have the flume-ng-sdk in the classpath (eg, flume-ng-sdk-1.8.0-SNAPSHOT.jar). This appender supports a round-robin and random scheme for performing the load balancing. It also supports a configurable backoff timeout so that down agents are removed temporarily from the set of hosts Required properties are in **bold**.

<table style="border:0px; border-collapse:collapse">      
 <thead> 
  <tr> 
   <th style="text-align:left; padding:1px 8px 1px 5px; background-color:rgb(238,221,238); border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Property Name</th> 
   <th style="text-align:left; padding:1px 8px 1px 5px; background-color:rgb(238,221,238); border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Default</th> 
   <th style="text-align:left; padding:1px 8px 1px 5px; background-color:rgb(238,221,238); border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> <strong>Hosts</strong></td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> –</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> A space-separated list of host:port at which Flume (through an AvroSource) is listening for events</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Selector</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> ROUND_ROBIN</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Selection mechanism. Must be either ROUND_ROBIN, RANDOM or custom FQDN to class that inherits from LoadBalancingSelector.</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> MaxBackoff</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> –</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> A long value representing the maximum amount of time in milliseconds the Load balancing client will backoff from a node that has failed to consume an event. Defaults to no backoff</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> UnsafeMode</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> false</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> If true, the appender will not throw exceptions on failure to send the events.</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> AvroReflectionEnabled</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> false</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> Use Avro Reflection to serialize Log4j events.</td> 
  </tr> 
  <tr> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> AvroSchemaUrl</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> –</td> 
   <td style="padding:1px 8px 1px 5px; border-width:0px 0px 1px; border-top-style:initial; border-top-color:initial; border-left-style:initial; border-left-color:initial; border-right-style:initial; border-right-color:initial; border-bottom-style:solid; border-bottom-color:rgb(170,170,170)"> A URL from which the Avro schema can be retrieved.</td> 
  </tr> 
 </tbody> 
</table>

Sample log4j.properties file configured using defaults:

#...
    log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
    log4j.appender.out2.Hosts = localhost:25430 localhost:25431
    
    # configure a class's logger to output to the flume appender
    log4j.logger.org.example.MyClass = DEBUG,flume
    #...

Sample log4j.properties file configured using RANDOM load balancing:

#...
    log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
    log4j.appender.out2.Hosts = localhost:25430 localhost:25431
    log4j.appender.out2.Selector = RANDOM
    
    # configure a class's logger to output to the flume appender
    log4j.logger.org.example.MyClass = DEBUG,flume
    #...

Sample log4j.properties file configured using backoff:

#...
    log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
    log4j.appender.out2.Hosts = localhost:25430 localhost:25431 localhost:25432
    log4j.appender.out2.Selector = ROUND_ROBIN
    log4j.appender.out2.MaxBackoff = 30000
    
    # configure a class's logger to output to the flume appender
    log4j.logger.org.example.MyClass = DEBUG,flume