Flume-ng-extends的应用

墨蓝 2022-08-04 09:21 121阅读 0赞

Flume-ng-extends的应用

目的:  这个可以tail dir 文件，监听文件目录写入事件，只要往这个目录写入文件就能触发

首先需要flume-ng-extends：从获取源代码   https://github.com/jinoos/flume-ng-extends，maven编译打包

将flume-ng-extends-0.0.1-SNAPSHOT.jar,另外还要下载commons-vfs2-2.0.jar这个jar包（这个不然可能会报错）一起放到flume的lib下

2，agent配置

**\[html\]**  [view plain][] [copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  hadoop@stormspark:~/bigdata/flume-1.4.0-bin/dirTailTest$ cat ../conf/flume-ng-extend.properties   
2.  a.sources = sources  
3.  a.sinks = sinks  
4.  a.channels = c  
5.    
6.  \#configure sources  
7.  a.sources.sources.type = com.jinoos.flume.DirectoryTailSource  
8.  a.sources.sources.dirs = s0 s1  
9.  a.sources.sources.dirs.s0.path = /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s0in  
10. a.sources.sources.dirs.s1.path = /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in  
11. \#congfigure sinks  
12. a.sinks.sinks.type = logger  
13. \#configure channals  
14. a.channels.c.type = memory  
15. \#bind channel  
16. a.sources.sources.channels = c

启动这个agent

**\[html\]**  [view plain][] [copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  bin/flume-ng agent --conf conf  --conf-file conf/flume-ng-extend.properties --name a \-Dflume.root.logger=INFO,console

然后往s0in,s1in目录丢进去一些文件

**\[html\]**  [view plain][] [copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  cp ../conf/\*.properties  ./s1in/

可以看到已经触发了

**\[html\]**  [view plain][] [copy][view plain] [![在CODE上查看代码片][CODE]][CODE_CODE] [![派生到我的代码片][ico_fork.svg]][ico_fork.svg 1]

1.  2014-01-17 20:29:35,774 (lifecycleSupervisor-1-4) \[INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)\] Monitoried counter group for type: SOURCE, name: sources, registered successfully.  
2.  2014-01-17 20:29:35,774 (lifecycleSupervisor-1-4) \[INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)\] Component type: SOURCE, name: sources started  
3.  2014-01-17 20:29:53,314 (pool-3-thread-2) \[WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileChanged(DirectoryTailSource.java:417)\] /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in/flume-ng-extend.propertiesis not in monitoring list.  
4.  2014-01-17 20:29:53,812 (pool-3-thread-5) \[WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)\] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in  
5.  2014-01-17 20:29:53,812 (pool-3-thread-4) \[WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)\] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in  
6.  2014-01-17 20:29:53,812 (pool-3-thread-6) \[WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)\] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in  
7.  2014-01-17 20:29:53,812 (pool-3-thread-3) \[WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)\] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in  
8.  2014-01-17 20:29:53,812 (pool-3-thread-7) \[WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)\] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in

[view plain]: http://blog.csdn.net/luyee2010/article/details/18417283#
[CODE]: https://code.csdn.net/assets/CODE_ico.png
[CODE_CODE]: https://code.csdn.net/snippets/161258
[ico_fork.svg]: https://code.csdn.net/assets/ico_fork.svg
[ico_fork.svg 1]: https://code.csdn.net/snippets/161258/fork