Getting Started with various Tools

33 / 44

Flume

Purpose: Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows.

Home Page: https://flume.apache.org/

Documentation: https://flume.apache.org/documentation.html

Related resources to learn: https://cloudxlab.com/assessment/playlist-intro/14/flume?course_id=1&playlist_id=14

  1. In the web console tab on the right side of the screen, to get a copy of sample flume conf from common data:

    hadoop fs -copyToLocal /data/flume/conf
    
  2. Change the port if needed and location in HDFS

    nano conf/flume.properties
    
  3. Launch the flume agent

     flume-ng agent --conf conf --conf-file conf/flume.properties --name a1 Dflume.root.logger=INFO,console
    
  4. Open a new console and Connect to the same port that you defined in config

     nc localhost 44443
    
  5. Generate some data by typing something in the console

  6. Open a new console and Check in hdfs using

     hadoop fs -ls flume_webdata
     hadoop fs -cat 'flume_webdata/FlumeData*'
    

No hints are availble for this assesment

Answer is not availble for this assesment

Loading comments...