unstick r1816163

Revision 1816163

Date:
2017/11/23 15:39:02
Author:
hao
Revision Log:
Update v0.5.0
Files:

Legend:

 
Added
 
Removed
 
Modified
  • eagle/site/docs/ambari-plugin-install.html

     
    168 168 <li>
    169 169 <p>Create a Kafka<sup id="fnref:KAFKA"><a href="#fn:KAFKA" class="footnote">1</a></sup> topic if you have not. Here is an example command.</p>
    170 170
    171 <pre><code>$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hdfs_audit_log
    171 <div class="highlighter-rouge"><pre class="highlight"><code>$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hdfs_audit_log
    172 172 </code></pre>
    173 </div>
    173 174 </li>
    174 175 <li>
    175 176 <p>Stream HDFS log data to Kafka, and refer to <a href="/docs/import-hdfs-auditLog.html">here</a> on how to do it .</p>
     
    185 186 <li>
    186 187 <p>Install Eagle Ambari plugin</p>
    187 188
    188 <pre><code>$ /usr/hdp/current/eagle/bin/eagle-ambari.sh install
    189 <div class="highlighter-rouge"><pre class="highlight"><code>$ /usr/hdp/current/eagle/bin/eagle-ambari.sh install
    189 190 </code></pre>
    191 </div>
    190 192 </li>
    191 193 <li>
    192 194 <p>Restart <a href="http://127.0.0.1:8000/">Ambari</a> click on disable and enable Ambari back.</p>
     
    200 202 <li>
    201 203 <p>Add Policies and meta data required by running the below script.</p>
    202 204
    203 <pre><code>$ cd &lt;eagle-home&gt;
    205 <div class="highlighter-rouge"><pre class="highlight"><code>$ cd &lt;eagle-home&gt;
    204 206 $ examples/sample-sensitivity-resource-create.sh
    205 207 $ examples/sample-policy-create.sh
    206 208 </code></pre>
    209 </div>
    207 210 </li>
    208 211 </ol>
    209 212
     
    214 217 <div class="footnotes">
    215 218 <ol>
    216 219 <li id="fn:KAFKA">
    217 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    220 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    218 221 </li>
    219 222 <li id="fn:STORM">
    220 <p><em>Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    223 <p><em>Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    221 224 </li>
    222 225 <li id="fn:SPARK">
    223 <p><em>Apache Spark.</em> <a href="#fnref:SPARK" class="reversefootnote">&#8617;</a></p>
    226 <p><em>Apache Spark.</em>&nbsp;<a href="#fnref:SPARK" class="reversefootnote">&#8617;</a></p>
    224 227 </li>
    225 228 <li id="fn:HBASE">
    226 <p><em>Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    229 <p><em>Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    227 230 </li>
    228 231 <li id="fn:AMBARI">
    229 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    232 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em>&nbsp;<a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    230 233 </li>
    231 234 </ol>
    232 235 </div>
  • eagle/site/docs/cloudera-integration.html

     
    168 168 <ul>
    169 169 <li>Zookeeper (installed through Cloudera Manager)</li>
    170 170 <li>Kafka (installed through Cloudera Manager)</li>
    171 <li>Storm (<code>0.9.x</code> or <code>0.10.x</code>, installed manually)</li>
    172 <li>Logstash (<code>2.X</code>, installed manually on NameNode)</li>
    171 <li>Storm (<code class="highlighter-rouge">0.9.x</code> or <code class="highlighter-rouge">0.10.x</code>, installed manually)</li>
    172 <li>Logstash (<code class="highlighter-rouge">2.X</code>, installed manually on NameNode)</li>
    173 173 </ul>
    174 174
    175 175 <h3 id="kafka">Kafka</h3>
     
    180 180
    181 181 <ul>
    182 182 <li>
    183 <p>Open Cloudera Manager and open “kafka” configuration, then set <code>“zookeeper Root”</code> to <code>“/”</code>.</p>
    183 <p>Open Cloudera Manager and open “kafka” configuration, then set <code class="highlighter-rouge">“zookeeper Root”</code> to <code class="highlighter-rouge">“/”</code>.</p>
    184 184 </li>
    185 185 <li>
    186 <p>If Kafka cannot be started successfully, check kafka’s log. If stack trace shows: <code>“java.lang.OutOfMemoryError: Java heap space”</code>. Increase heap size by setting <code>"KAFKA_HEAP_OPTS"</code>in <code>/bin/kafka-server-start.sh</code>.</p>
    186 <p>If Kafka cannot be started successfully, check kafka’s log. If stack trace shows: <code class="highlighter-rouge">“java.lang.OutOfMemoryError: Java heap space”</code>. Increase heap size by setting <code class="highlighter-rouge">"KAFKA_HEAP_OPTS"</code>in <code class="highlighter-rouge">/bin/kafka-server-start.sh</code>.</p>
    187 187 </li>
    188 188 </ul>
    189 189
    190 190 <p>Example:</p>
    191 191
    192 <pre><code> export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
    192 <div class="highlighter-rouge"><pre class="highlight"><code> export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
    193 193 </code></pre>
    194 </div>
    194 195
    195 196 <h4 id="verification">Verification</h4>
    196 197
     
    198 199 <li>Step1: create a kafka topic (here I created a topic called “test”, which will be used in logstash configuration file to receive hdfsAudit log messages from Cloudera.</li>
    199 200 </ul>
    200 201
    201 <pre><code>bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test
    202 <div class="highlighter-rouge"><pre class="highlight"><code>bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic test
    202 203 </code></pre>
    204 </div>
    203 205
    204 206 <ul>
    205 207 <li>Step2: check if topic has been created successfully.</li>
    206 208 </ul>
    207 209
    208 <pre><code>bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181
    210 <div class="highlighter-rouge"><pre class="highlight"><code>bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181
    209 211 </code></pre>
    212 </div>
    210 213
    211 214 <p>this command will show all created topics.</p>
    212 215
     
    214 217 <li>Step3: open two terminals, start “producer” and “consumer” separately.</li>
    215 218 </ul>
    216 219
    217 <pre><code>/usr/bin/kafka-console-producer --broker-list hostname:9092 --topic test
    220 <div class="highlighter-rouge"><pre class="highlight"><code>/usr/bin/kafka-console-producer --broker-list hostname:9092 --topic test
    218 221 /usr/bin/kafka-console-consumer --zookeeper hostname:2181 --topic test
    219 222 </code></pre>
    223 </div>
    220 224
    221 225 <ul>
    222 226 <li>Step4: type in some message in producer. If consumer can receive the messages sent from producer, then kafka is working fine. Otherwise please check the configuration and logs to identify the root cause of issues.</li>
     
    228 232
    229 233 <p>You can follow <a href="https://www.elastic.co/downloads/logstash">logstash online doc</a> to download and install logstash on your machine:</p>
    230 234
    231 <p>Or you can install it through <code>yum</code> if you are using centos:</p>
    235 <p>Or you can install it through <code class="highlighter-rouge">yum</code> if you are using centos:</p>
    232 236
    233 237 <ul>
    234 238 <li>download and install the public signing key:</li>
    235 239 </ul>
    236 240
    237 <pre><code>rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
    241 <div class="highlighter-rouge"><pre class="highlight"><code>rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
    238 242 </code></pre>
    243 </div>
    239 244
    240 245 <ul>
    241 <li>Add the following lines in <code>/etc/yum.repos.d/</code> directory in a file with a <code>.repo</code> suffix, for example <code>logstash.repo</code>.</li>
    246 <li>Add the following lines in <code class="highlighter-rouge">/etc/yum.repos.d/</code> directory in a file with a <code class="highlighter-rouge">.repo</code> suffix, for example <code class="highlighter-rouge">logstash.repo</code>.</li>
    242 247 </ul>
    243 248
    244 <pre><code>[logstash-2.3]
    249 <div class="highlighter-rouge"><pre class="highlight"><code>[logstash-2.3]
    245 250 name=Logstash repository for 2.3.x packages
    246 251 baseurl=https://packages.elastic.co/logstash/2.3/centos
    247 252 gpgcheck=1
    248 253 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
    249 254 enabled=1
    250 255 </code></pre>
    256 </div>
    251 257
    252 258 <ul>
    253 <li>Then install it using <code>yum</code>:</li>
    259 <li>Then install it using <code class="highlighter-rouge">yum</code>:</li>
    254 260 </ul>
    255 261
    256 <pre><code>yum install logstash
    262 <div class="highlighter-rouge"><pre class="highlight"><code>yum install logstash
    257 263 </code></pre>
    264 </div>
    258 265
    259 266 <h4 id="create-conf-file">Create conf file</h4>
    260 267
     
    262 269
    263 270 <h4 id="start-logstash">Start logstash</h4>
    264 271
    265 <pre><code>bin/logstash -f conf/first-pipeline.conf
    272 <div class="highlighter-rouge"><pre class="highlight"><code>bin/logstash -f conf/first-pipeline.conf
    266 273 </code></pre>
    274 </div>
    267 275
    268 276 <h4 id="verification-1">Verification</h4>
    269 277
     
    274 282
    275 283 <h4 id="installation-1">Installation</h4>
    276 284
    277 <p>Download Apache Storm from <a href="http://storm.apache.org/downloads.html">here</a>, the version you choose should be <code>0.10.x</code> or <code>0.9.x</code> release.
    285 <p>Download Apache Storm from <a href="http://storm.apache.org/downloads.html">here</a>, the version you choose should be <code class="highlighter-rouge">0.10.x</code> or <code class="highlighter-rouge">0.9.x</code> release.
    278 286 Then follow <a href="http://storm.apache.org/releases/0.10.0/Setting-up-a-Storm-cluster.html">Apache Storm online doc</a>) to install Apache Storm on your cluster.</p>
    279 287
    280 <p>In <code>/etc/profile</code>, add this:</p>
    288 <p>In <code class="highlighter-rouge">/etc/profile</code>, add this:</p>
    281 289
    282 <pre><code>export PATH=$PATH:/opt/apache-storm-0.10.1/bin/
    290 <div class="highlighter-rouge"><pre class="highlight"><code>export PATH=$PATH:/opt/apache-storm-0.10.1/bin/
    283 291 </code></pre>
    292 </div>
    284 293
    285 294 <p>save the profile and then type:</p>
    286 295
    287 <pre><code>source /etc/profile
    296 <div class="highlighter-rouge"><pre class="highlight"><code>source /etc/profile
    288 297 </code></pre>
    298 </div>
    289 299
    290 300 <p>to make it work.</p>
    291 301
    292 302 <h4 id="configuration-1">Configuration</h4>
    293 303
    294 <p>In <code>storm/conf/storm.yaml</code>, change the hostname to your own host.</p>
    304 <p>In <code class="highlighter-rouge">storm/conf/storm.yaml</code>, change the hostname to your own host.</p>
    295 305
    296 306 <h4 id="start-apache-storm">Start Apache Storm</h4>
    297 307
    298 308 <p>In Termial, type:</p>
    299 309
    300 <pre><code>$: storm nimbus
    310 <div class="highlighter-rouge"><pre class="highlight"><code>$: storm nimbus
    301 311 $: storm supervisor
    302 312 $: storm UI
    303 313 </code></pre>
    314 </div>
    304 315
    305 316 <h4 id="verification-2">Verification</h4>
    306 317
    307 <p>Open storm UI in your browser, default URL is : <code>http://hostname:8080/index.html</code>.</p>
    318 <p>Open storm UI in your browser, default URL is : <code class="highlighter-rouge">http://hostname:8080/index.html</code>.</p>
    308 319
    309 320 <h3 id="apache-eagle">Apache Eagle</h3>
    310 321
    311 322 <p>To download and install Apache Eagle, please refer to <a href="http://eagle.incubator.apache.org/docs/quick-start.html">Get Started with Sandbox.</a> .</p>
    312 323
    313 <p>One thing need to mention is: in <code>“/bin/eagle-topology.sh”</code>, line 102:</p>
    324 <p>One thing need to mention is: in <code class="highlighter-rouge">“/bin/eagle-topology.sh”</code>, line 102:</p>
    314 325
    315 <pre><code> storm_ui=http://localhost:8080
    326 <div class="highlighter-rouge"><pre class="highlight"><code> storm_ui=http://localhost:8080
    316 327 </code></pre>
    328 </div>
    317 329
    318 330 <p>If you are not using the default port number, change this to your own Storm UI url.</p>
    319 331
    320 332 <p>I know it takes time to finish these configuration, but now it is time to have fun!
    321 Just try <code>HDFS Data Activity Monitoring</code> with <code>Demo</code> listed in <a href="http://eagle.incubator.apache.org/docs/hdfs-data-activity-monitoring.html">HDFS Data Activity Monitoring.</a></p>
    333 Just try <code class="highlighter-rouge">HDFS Data Activity Monitoring</code> with <code class="highlighter-rouge">Demo</code> listed in <a href="http://eagle.incubator.apache.org/docs/hdfs-data-activity-monitoring.html">HDFS Data Activity Monitoring.</a></p>
    322 334
    323 335
    324 336 </div><!--end of loadcontent-->
  • eagle/site/docs/community.html

     
    183 183 <td> </td>
    184 184 <td> </td>
    185 185 <td> </td>
    186 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#117;&#115;&#101;&#114;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></td>
    186 <td><a href="mailto:user@eagle.apache.org">user@eagle.apache.org</a></td>
    187 187 <td> </td>
    188 188 <td> </td>
    189 189 <td> </td>
    190 190 <td> </td>
    191 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#045;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">subscribe</a></td>
    192 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#117;&#115;&#101;&#114;&#045;&#117;&#110;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">unsubscribe</a></td>
    191 <td><a href="mailto:user-subscribe@eagle.apache.org">subscribe</a></td>
    192 <td><a href="mailto:user-unsubscribe@eagle.apache.org">unsubscribe</a></td>
    193 193 <td><a href="http://mail-archives.apache.org/mod_mbox/eagle-user/">eagle-user</a></td>
    194 194 </tr>
    195 195 <tr>
     
    198 198 <td> </td>
    199 199 <td> </td>
    200 200 <td> </td>
    201 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></td>
    201 <td><a href="mailto:dev@eagle.apache.org">dev@eagle.apache.org</a></td>
    202 202 <td> </td>
    203 203 <td> </td>
    204 204 <td> </td>
    205 205 <td> </td>
    206 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#045;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">subscribe</a></td>
    207 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#045;&#117;&#110;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">unsubscribe</a></td>
    206 <td><a href="mailto:dev-subscribe@eagle.apache.org">subscribe</a></td>
    207 <td><a href="mailto:dev-unsubscribe@eagle.apache.org">unsubscribe</a></td>
    208 208 <td><a href="http://mail-archives.apache.org/mod_mbox/eagle-dev/">eagle-dev</a></td>
    209 209 </tr>
    210 210 <tr>
     
    213 213 <td> </td>
    214 214 <td> </td>
    215 215 <td> </td>
    216 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#105;&#115;&#115;&#117;&#101;&#115;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#105;&#115;&#115;&#117;&#101;&#115;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></td>
    216 <td><a href="mailto:issues@eagle.apache.org">issues@eagle.apache.org</a></td>
    217 217 <td> </td>
    218 218 <td> </td>
    219 219 <td> </td>
    220 220 <td> </td>
    221 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#105;&#115;&#115;&#117;&#101;&#115;&#045;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">subscribe</a></td>
    222 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#105;&#115;&#115;&#117;&#101;&#115;&#045;&#117;&#110;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">unsubscribe</a></td>
    221 <td><a href="mailto:issues-subscribe@eagle.apache.org">subscribe</a></td>
    222 <td><a href="mailto:issues-unsubscribe@eagle.apache.org">unsubscribe</a></td>
    223 223 <td><a href="http://mail-archives.apache.org/mod_mbox/eagle-issues/">eagle-issues</a></td>
    224 224 </tr>
    225 225 <tr>
     
    228 228 <td> </td>
    229 229 <td> </td>
    230 230 <td> </td>
    231 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#099;&#111;&#109;&#109;&#105;&#116;&#115;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#099;&#111;&#109;&#109;&#105;&#116;&#115;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></td>
    231 <td><a href="mailto:commits@eagle.apache.org">commits@eagle.apache.org</a></td>
    232 232 <td> </td>
    233 233 <td> </td>
    234 234 <td> </td>
    235 235 <td> </td>
    236 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#099;&#111;&#109;&#109;&#105;&#116;&#115;&#045;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">subscribe</a></td>
    237 <td><a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#099;&#111;&#109;&#109;&#105;&#116;&#115;&#045;&#117;&#110;&#115;&#117;&#098;&#115;&#099;&#114;&#105;&#098;&#101;&#064;&#101;&#097;&#103;&#108;&#101;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">unsubscribe</a></td>
    236 <td><a href="mailto:commits-subscribe@eagle.apache.org">subscribe</a></td>
    237 <td><a href="mailto:commits-unsubscribe@eagle.apache.org">unsubscribe</a></td>
    238 238 <td><a href="http://mail-archives.apache.org/mod_mbox/eagle-commits/">eagle-commits</a></td>
    239 239 </tr>
    240 240 </tbody>
     
    317 317 <li><strong>Wechat</strong>: Apache_Eagle</li>
    318 318 </ul>
    319 319
    320 <h3 id="events-and-meetupshadoop">Events and Meetups<sup id="fnref:HADOOP"><a href="#fn:HADOOP" class="footnote">1</a></sup></h3>
    320 <h3 id="events-and-meetups">Events and Meetups<sup id="fnref:HADOOP"><a href="#fn:HADOOP" class="footnote">1</a></sup></h3>
    321 321
    322 322 <p><strong>Conferences</strong></p>
    323 323
     
    335 335 <div class="footnotes">
    336 336 <ol>
    337 337 <li id="fn:HADOOP">
    338 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    338 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    339 339 </li>
    340 340 </ol>
    341 341 </div>
  • eagle/site/docs/configuration.html

     
    156 156 </div>
    157 157 <div class="col-xs-6 col-sm-9 page-main-content" style="margin-left: -15px" id="loadcontent">
    158 158 <h1 class="page-header" style="margin-top: 0px">Application Configuration</h1>
    159 <p>Apache Eagle (called Eagle in the following) requires you to create a configuration file under <code>$EAGLE_HOME/conf/</code> for each application. Basically, there are some common properties shared, e.g., envContextConfig, eagleProps, and dynamicConfigSource. While dataSourceConfig differs from application to application.</p>
    159 <p>Apache Eagle (called Eagle in the following) requires you to create a configuration file under <code class="highlighter-rouge">$EAGLE_HOME/conf/</code> for each application. Basically, there are some common properties shared, e.g., envContextConfig, eagleProps, and dynamicConfigSource. While dataSourceConfig differs from application to application.</p>
    160 160
    161 161 <p>In this page we take the following two application as examples</p>
    162 162
     
    610 610 <div class="footnotes">
    611 611 <ol>
    612 612 <li id="fn:HIVE">
    613 <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    613 <p><em>All mentions of “hive” on this page represent Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    614 614 </li>
    615 615 <li id="fn:STORM">
    616 <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    616 <p><em>All mentions of “storm” on this page represent Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    617 617 </li>
    618 618 <li id="fn:KAFKA">
    619 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    619 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    620 620 </li>
    621 621 <li id="fn:ZOOKEEPER">
    622 <p><em>All mentions of “zookeeper” on this page represent Apache ZooKeeper.</em> <a href="#fnref:ZOOKEEPER" class="reversefootnote">&#8617;</a></p>
    622 <p><em>All mentions of “zookeeper” on this page represent Apache ZooKeeper.</em>&nbsp;<a href="#fnref:ZOOKEEPER" class="reversefootnote">&#8617;</a></p>
    623 623 </li>
    624 624 <li id="fn:HBASE">
    625 <p><em>Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    625 <p><em>Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    626 626 </li>
    627 627 <li id="fn:TOMCAT">
    628 <p><em>Apache Tomcat.</em> <a href="#fnref:TOMCAT" class="reversefootnote">&#8617;</a></p>
    628 <p><em>Apache Tomcat.</em>&nbsp;<a href="#fnref:TOMCAT" class="reversefootnote">&#8617;</a></p>
    629 629 </li>
    630 630 </ol>
    631 631 </div>
  • eagle/site/docs/deployment-env.html

     
    158 158 <h1 class="page-header" style="margin-top: 0px">Deploy Environment</h1>
    159 159 <h3 id="setup-environment">Setup Environment</h3>
    160 160
    161 <p>Apache Eagle (called Eagle in the following) as an analytics solution for identifying security and performance issues instantly, relies on streaming platform <code>Storm</code><sup id="fnref:STORM"><a href="#fn:STORM" class="footnote">1</a></sup> + <code>Kafka</code><sup id="fnref:KAFKA"><a href="#fn:KAFKA" class="footnote">2</a></sup> to meet the realtime criteria, and persistence storage to store metadata and some metrics. As for the persistence storage, it supports three types of database: <code>HBase</code><sup id="fnref:HBASE"><a href="#fn:HBASE" class="footnote">3</a></sup>, <code>Derby</code><sup id="fnref:DERBY"><a href="#fn:DERBY" class="footnote">4</a></sup>, and <code>Mysql</code></p>
    161 <p>Apache Eagle (called Eagle in the following) as an analytics solution for identifying security and performance issues instantly, relies on streaming platform <code class="highlighter-rouge">Storm</code><sup id="fnref:STORM"><a href="#fn:STORM" class="footnote">1</a></sup> + <code class="highlighter-rouge">Kafka</code><sup id="fnref:KAFKA"><a href="#fn:KAFKA" class="footnote">2</a></sup> to meet the realtime criteria, and persistence storage to store metadata and some metrics. As for the persistence storage, it supports three types of database: <code class="highlighter-rouge">HBase</code><sup id="fnref:HBASE"><a href="#fn:HBASE" class="footnote">3</a></sup>, <code class="highlighter-rouge">Derby</code><sup id="fnref:DERBY"><a href="#fn:DERBY" class="footnote">4</a></sup>, and <code class="highlighter-rouge">Mysql</code></p>
    162 162
    163 163 <p>To run monitoring applications, Eagle requires the following dependencies.</p>
    164 164
     
    226 226 <div class="footnotes">
    227 227 <ol>
    228 228 <li id="fn:STORM">
    229 <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    229 <p><em>All mentions of “storm” on this page represent Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    230 230 </li>
    231 231 <li id="fn:KAFKA">
    232 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    232 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    233 233 </li>
    234 234 <li id="fn:HBASE">
    235 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    235 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    236 236 </li>
    237 237 <li id="fn:DERBY">
    238 <p><em>All mentions of “derby” on this page represent Apache Derby.</em> <a href="#fnref:DERBY" class="reversefootnote">&#8617;</a></p>
    238 <p><em>All mentions of “derby” on this page represent Apache Derby.</em>&nbsp;<a href="#fnref:DERBY" class="reversefootnote">&#8617;</a></p>
    239 239 </li>
    240 240 <li id="fn:HADOOP">
    241 <p><em>Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    241 <p><em>Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    242 242 </li>
    243 243 <li id="fn:AMBARI">
    244 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    244 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em>&nbsp;<a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    245 245 </li>
    246 246 </ol>
    247 247 </div>
  • eagle/site/docs/deployment-in-docker.html

     
    166 166 <li>
    167 167 <p>Pull latest eagle docker image from <a href="https://hub.docker.com/r/apacheeagle/sandbox/">docker hub</a> directly:</p>
    168 168
    169 <pre><code>docker pull apacheeagle/sandbox
    169 <div class="highlighter-rouge"><pre class="highlight"><code>docker pull apacheeagle/sandbox
    170 170 </code></pre>
    171 </div>
    171 172 </li>
    172 173 <li>
    173 174 <p>Then run eagle docker image:</p>
    174 175
    175 <pre><code>docker run -p 9099:9099 -p 8080:8080 -p 8744:8744 -p 2181:2181 -p 2888:2888 \
    176 <div class="highlighter-rouge"><pre class="highlight"><code>docker run -p 9099:9099 -p 8080:8080 -p 8744:8744 -p 2181:2181 -p 2888:2888 \
    176 177 -p 6667:6667 -p 60020:60020 -p 60030:60030 -p 60010:60010 -d --dns 127.0.0.1 \
    177 178 --entrypoint /usr/local/serf/bin/start-serf-agent.sh -e KEYCHAIN= \
    178 179 --env EAGLE_SERVER_HOST=sandbox.eagle.apache.org --name sandbox \
     
    182 183 --link sandbox:ambariserver --entrypoint /bin/sh apacheeagle/sandbox:latest \
    183 184 -c /tmp/install-cluster.sh
    184 185 </code></pre>
    186 </div>
    185 187 </li>
    186 188 </ul>
    187 189 </li>
     
    192 194 <li>
    193 195 <p>Get latest source code of eagle.</p>
    194 196
    195 <pre><code>git clone https://github.com/apache/eagle.git
    197 <div class="highlighter-rouge"><pre class="highlight"><code>git clone https://github.com/apache/eagle.git
    196 198 </code></pre>
    199 </div>
    197 200 </li>
    198 201 <li>
    199 202 <p>Then run eagle docker command.</p>
    200 203
    201 <pre><code>cd eagle &amp;&amp; ./eagle-docker boot
    204 <div class="highlighter-rouge"><pre class="highlight"><code>cd eagle &amp;&amp; ./eagle-docker boot
    202 205 </code></pre>
    206 </div>
    203 207 </li>
    204 208 </ul>
    205 209 </li>
  • eagle/site/docs/deployment-in-production.html

     
    184 184
    185 185 <ul>
    186 186 <li>
    187 <p>Edit <code>bin/eagle-env.sh</code></p>
    187 <p>Edit <code class="highlighter-rouge">bin/eagle-env.sh</code></p>
    188 188
    189 <pre><code> # TODO: make sure java version is 1.7.x
    189 <div class="highlighter-rouge"><pre class="highlight"><code> # TODO: make sure java version is 1.7.x
    190 190 export JAVA_HOME=
    191 191
    192 192 # TODO: Apache Storm nimbus host. Default is localhost
     
    195 195 # TODO: EAGLE_SERVICE_HOST, default is `hostname -f`
    196 196 export EAGLE_SERVICE_HOST=localhost
    197 197 </code></pre>
    198 </div>
    198 199 </li>
    199 200 <li>
    200 <p>Edit <code>conf/eagle-service.conf</code> to configure the database to use (for example: HBase<sup id="fnref:HBASE"><a href="#fn:HBASE" class="footnote">1</a></sup>)</p>
    201 <p>Edit <code class="highlighter-rouge">conf/eagle-service.conf</code> to configure the database to use (for example: HBase<sup id="fnref:HBASE"><a href="#fn:HBASE" class="footnote">1</a></sup>)</p>
    201 202
    202 <pre><code> # TODO: hbase.zookeeper.quorum in the format host1,host2,host3,...
    203 <div class="highlighter-rouge"><pre class="highlight"><code> # TODO: hbase.zookeeper.quorum in the format host1,host2,host3,...
    203 204 # default is "localhost"
    204 205 hbase-zookeeper-quorum="localhost"
    205 206
     
    211 212 # default is "/hbase"
    212 213 zookeeper-znode-parent="/hbase"
    213 214 </code></pre>
    215 </div>
    214 216 </li>
    215 217 </ul>
    216 218 </li>
    217 219 <li>
    218 220 <p>Step 2: Install metadata for policies</p>
    219 221
    220 <pre><code> $ cd &lt;eagle-home&gt;
    222 <div class="highlighter-rouge"><pre class="highlight"><code> $ cd &lt;eagle-home&gt;
    221 223
    222 224 # start Eagle web service
    223 225 $ bin/eagle-service.sh start
     
    225 227 # import metadata after Eagle service is successfully started
    226 228 $ bin/eagle-topology-init.sh
    227 229 </code></pre>
    230 </div>
    228 231 </li>
    229 232 </ul>
    230 233
     
    260 263 <li>
    261 264 <p>Stop eagle service</p>
    262 265
    263 <pre><code>$ bin/eagle-service.sh stop
    266 <div class="highlighter-rouge"><pre class="highlight"><code>$ bin/eagle-service.sh stop
    264 267 </code></pre>
    268 </div>
    265 269 </li>
    266 270 </ul>
    267 271
     
    272 276 <div class="footnotes">
    273 277 <ol>
    274 278 <li id="fn:HBASE">
    275 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    279 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    276 280 </li>
    277 281 <li id="fn:HADOOP">
    278 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    282 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    279 283 </li>
    280 284 <li id="fn:HIVE">
    281 <p><em>Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    285 <p><em>Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    282 286 </li>
    283 287 </ol>
    284 288 </div>
  • eagle/site/docs/deployment-in-sandbox.html

     
    201 201 <p><strong>Option 1</strong>: Download eagle jar from <a href="http://66.211.190.194/eagle-0.1.0.tar.gz">here</a>.</p>
    202 202 </li>
    203 203 <li>
    204 <p><strong>Option 2</strong>: Build form source code <a href="https://github.com/apache/eagle">eagle github</a>. After successful build, ‘eagle-xxx-bin.tar.gz’ will be generated under <code>./eagle-assembly/target</code></p>
    204 <p><strong>Option 2</strong>: Build form source code <a href="https://github.com/apache/eagle">eagle github</a>. After successful build, ‘eagle-xxx-bin.tar.gz’ will be generated under <code class="highlighter-rouge">./eagle-assembly/target</code></p>
    205 205
    206 <pre><code># installed npm is required before compiling
    206 <div class="highlighter-rouge"><pre class="highlight"><code># installed npm is required before compiling
    207 207 $ mvn clean install -DskipTests=true
    208 208 </code></pre>
    209 </div>
    209 210 </li>
    210 211 </ul>
    211 212 </li>
    212 213 <li>
    213 214 <p><strong>Copy and extract the package to sandbox</strong></p>
    214 215
    215 <pre><code>#extract
    216 <div class="highlighter-rouge"><pre class="highlight"><code>#extract
    216 217 $ tar -zxvf eagle-0.1.0-bin.tar.gz
    217 218 $ mv eagle-0.1.0 /usr/hdp/current/eagle
    218 219 </code></pre>
    220 </div>
    219 221 </li>
    220 222 </ul>
    221 223
     
    229 231 <li>
    230 232 <p><strong>Option 1</strong>: Install Eagle using command line</p>
    231 233
    232 <pre><code>$ cd /usr/hdp/current/eagle
    234 <div class="highlighter-rouge"><pre class="highlight"><code>$ cd /usr/hdp/current/eagle
    233 235 $ examples/eagle-sandbox-starter.sh
    234 236 </code></pre>
    237 </div>
    235 238 </li>
    236 239 <li>
    237 240 <p><strong>Option 2</strong>: Install Eagle using <a href="/docs/ambari-plugin-install.html">Eagle Ambari plugin</a></p>
     
    246 249 <li>
    247 250 <p><strong>Step 1</strong>: Configure Advanced hadoop-log4j via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a>, and add below “KAFKA_HDFS_AUDIT” log4j appender to hdfs audit logging.</p>
    248 251
    249 <pre><code>log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
    252 <div class="highlighter-rouge"><pre class="highlight"><code>log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
    250 253 log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log
    251 254 log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667
    252 255 log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer
     
    256 259 #log4j.appender.KAFKA_HDFS_AUDIT.BatchSize=1
    257 260 #log4j.appender.KAFKA_HDFS_AUDIT.QueueSize=1
    258 261 </code></pre>
    262 </div>
    259 263
    260 264 <p><img src="/images/docs/hdfs-log4j-conf.png" alt="HDFS LOG4J Configuration" title="hdfslog4jconf" /></p>
    261 265 </li>
    262 266 <li>
    263 267 <p><strong>Step 3</strong>: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a>, and add the reference to KAFKA_HDFS_AUDIT to HADOOP_NAMENODE_OPTS.</p>
    264 268
    265 <pre><code>-Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
    269 <div class="highlighter-rouge"><pre class="highlight"><code>-Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
    266 270 </code></pre>
    271 </div>
    267 272
    268 273 <p><img src="/images/docs/hdfs-env-conf.png" alt="HDFS Environment Configuration" title="hdfsenvconf" /></p>
    269 274 </li>
    270 275 <li>
    271 276 <p><strong>Step 4</strong>: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a>, and append the following command to it.</p>
    272 277
    273 <pre><code>export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
    278 <div class="highlighter-rouge"><pre class="highlight"><code>export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
    274 279 </code></pre>
    280 </div>
    275 281
    276 282 <p><img src="/images/docs/hdfs-env-conf2.png" alt="HDFS Environment Configuration" title="hdfsenvconf2" /></p>
    277 283 </li>
     
    279 285 <p><strong>Step 5</strong>: save the changes and restart the namenode.</p>
    280 286 </li>
    281 287 <li>
    282 <p><strong>Step 6</strong>: Check whether logs are flowing into topic <code>sandbox_hdfs_audit_log</code></p>
    288 <p><strong>Step 6</strong>: Check whether logs are flowing into topic <code class="highlighter-rouge">sandbox_hdfs_audit_log</code></p>
    283 289
    284 <pre><code>$ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log
    290 <div class="highlighter-rouge"><pre class="highlight"><code>$ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log
    285 291 </code></pre>
    292 </div>
    286 293 </li>
    287 294 </ul>
    288 295
    289 <p>Now please login to Eagle web http://localhost:9099/eagle-service with account <code>admin/secret</code>, and try the sample demos on
    296 <p>Now please login to Eagle web http://localhost:9099/eagle-service with account <code class="highlighter-rouge">admin/secret</code>, and try the sample demos on
    290 297 <a href="/docs/quick-start.html">Quick Starer</a></p>
    291 298
    292 299 <p>(If the NAT network is used in a virtual machine, it’s required to add port 9099 to forwarding ports)
     
    300 307 <div class="footnotes">
    301 308 <ol>
    302 309 <li id="fn:HADOOP">
    303 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    310 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    304 311 </li>
    305 312 <li id="fn:AMBARI">
    306 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    313 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em>&nbsp;<a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    307 314 </li>
    308 315 <li id="fn:HBASE">
    309 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    316 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    310 317 </li>
    311 318 <li id="fn:STORM">
    312 <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    319 <p><em>All mentions of “storm” on this page represent Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    313 320 </li>
    314 321 <li id="fn:KAFKA">
    315 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    322 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    316 323 </li>
    317 324 </ol>
    318 325 </div>
  • eagle/site/docs/development-in-intellij.html

     
    158 158 <h1 class="page-header" style="margin-top: 0px">Development in Intellij</h1>
    159 159 <p>Apache Eagle (called Eagle in the following) can be developed in popular IDE, e.g. Intellij and Eclipse. Here we focus on development in Intellij.</p>
    160 160
    161 <h3 id="prepare-hadoophadoop-environment">1. Prepare Hadoop<sup id="fnref:HADOOP"><a href="#fn:HADOOP" class="footnote">1</a></sup> environment</h3>
    161 <h3 id="1-prepare-hadoop-environment">1. Prepare Hadoop<sup id="fnref:HADOOP"><a href="#fn:HADOOP" class="footnote">1</a></sup> environment</h3>
    162 162
    163 163 <p>Normally HDP sandbox is needed for testing Hadoop monitoring. Please reference <a href="/docs/quick-start.html">Quick Start</a> for setting up HDP sandbox.</p>
    164 164
     
    185 185 </li>
    186 186 </ul>
    187 187
    188 <h3 id="start-eagle-web-service-in-intellij">2. Start Eagle web service in Intellij</h3>
    188 <h3 id="2-start-eagle-web-service-in-intellij">2. Start Eagle web service in Intellij</h3>
    189 189
    190 190 <p>Import source code into Intellij, and find eagle-webservice project. Intellij Ultimate supports launching J2EE server within Intellij. If you don’t have
    191 191 Intellij Ultimate version, Eclipse is another option.</p>
     
    208 208
    209 209 <p>Configure Intellij for running Apache Tomcat server with eagle-service artifacts</p>
    210 210
    211 <h3 id="start-topology-in-intellij">3. Start topology in Intellij</h3>
    211 <h3 id="3-start-topology-in-intellij">3. Start topology in Intellij</h3>
    212 212
    213 213 <ul>
    214 214 <li><strong>Check topology configuration</strong></li>
     
    237 237 <div class="footnotes">
    238 238 <ol>
    239 239 <li id="fn:HADOOP">
    240 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    240 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    241 241 </li>
    242 242 <li id="fn:HBASE">
    243 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    243 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    244 244 </li>
    245 245 <li id="fn:ZOOKEEPER">
    246 <p><em>Apache ZooKeeper.</em> <a href="#fnref:ZOOKEEPER" class="reversefootnote">&#8617;</a></p>
    246 <p><em>Apache ZooKeeper.</em>&nbsp;<a href="#fnref:ZOOKEEPER" class="reversefootnote">&#8617;</a></p>
    247 247 </li>
    248 248 <li id="fn:KAFKA">
    249 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    249 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    250 250 </li>
    251 251 <li id="fn:STORM">
    252 <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    252 <p><em>All mentions of “storm” on this page represent Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    253 253 </li>
    254 254 </ol>
    255 255 </div>
  • eagle/site/docs/development-in-macosx.html

     
    159 159 <h2 id="how-to-setup-apache-eagle-development-environment-on-mac-osx">How to Setup Apache Eagle Development Environment on Mac OSX</h2>
    160 160
    161 161 <p><em>Apache Eagle will be called Eagle in the following.</em><br />
    162 This tutorial is based <code>Mac OS X</code>. It can be used as a reference guide for other OS like Linux or Windows as well. To save your time of jumping back and forth between different web pages, all necessary references will be pointed out.</p>
    162 This tutorial is based <code class="highlighter-rouge">Mac OS X</code>. It can be used as a reference guide for other OS like Linux or Windows as well. To save your time of jumping back and forth between different web pages, all necessary references will be pointed out.</p>
    163 163
    164 164 <h3 id="prerequisite">Prerequisite</h3>
    165 165
     
    169 169
    170 170 <p>Make sure you have HomeBrew installed on your mac. If not, please run:</p>
    171 171
    172 <pre><code>$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    172 <div class="highlighter-rouge"><pre class="highlight"><code>$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    173 173 </code></pre>
    174 </div>
    174 175
    175 176 <p>you can find more information about HomeBrew at http://brew.sh/ .</p>
    176 177
     
    180 181
    181 182 <p>Some core eagle modules are written with scala. To install Scala and SBT, just run:</p>
    182 183
    183 <pre><code> $ brew install scala
    184 <div class="highlighter-rouge"><pre class="highlight"><code> $ brew install scala
    184 185 $ brew install sbt
    185 186 </code></pre>
    187 </div>
    186 188
    187 189 <ul>
    188 190 <li><strong>NPM</strong></li>
     
    190 192
    191 193 <p>Eagle-webservice module uses npm. To install it, run:</p>
    192 194
    193 <pre><code> $ brew install npm
    195 <div class="highlighter-rouge"><pre class="highlight"><code> $ brew install npm
    194 196 </code></pre>
    197 </div>
    195 198
    196 199 <ul>
    197 200 <li><strong>Apache Maven</strong></li>
     
    199 202
    200 203 <p>Eagle is built with maven:</p>
    201 204
    202 <pre><code> $ brew install maven
    205 <div class="highlighter-rouge"><pre class="highlight"><code> $ brew install maven
    203 206 </code></pre>
    207 </div>
    204 208
    205 209 <ul>
    206 210 <li>
     
    210 214 <li>
    211 215 <p>Install HomeBrew Cask:</p>
    212 216
    213 <pre><code>$ brew install caskroom/cask/brew-cask
    217 <div class="highlighter-rouge"><pre class="highlight"><code>$ brew install caskroom/cask/brew-cask
    214 218 </code></pre>
    219 </div>
    215 220 </li>
    216 221 <li>
    217 222 <p>Next, install JDK via HomeBrew:</p>
     
    224 229
    225 230 <p>you will see all available JDK versions and you can install multiple JDK versions in this way. For eagle please choose java7 to install:</p>
    226 231
    227 <pre><code> $ brew cask install java7
    232 <div class="highlighter-rouge"><pre class="highlight"><code> $ brew cask install java7
    228 233 </code></pre>
    234 </div>
    229 235
    230 <p><strong>Note:</strong>
    231 - During this writing SBT has issue with JDK 8. This issue has been tested confirmed by using:
    232 - Java 1.8.0_66
    233 - Maven 3.3.9
    234 - Scala 2.11.7
    235 - Sbt 0.13.9</p>
    236 <p><strong>Note:</strong></p>
    237 <ul>
    238 <li>During this writing SBT has issue with JDK 8. This issue has been tested confirmed by using:</li>
    239 <li>Java 1.8.0_66</li>
    240 <li>Maven 3.3.9</li>
    241 <li>Scala 2.11.7</li>
    242 <li>Sbt 0.13.9</li>
    243 </ul>
    236 244
    237 245 <p>you can find more information about HomeBrew Cask at <a href="http://caskroom.io">http://caskroom.io</a></p>
    238 246
     
    242 250
    243 251 <p>you can use Jenv to manage installed multiple Java versions. To install it:</p>
    244 252
    245 <pre><code>$ brew install https://raw.githubusercontent.com/entrypass/jenv/homebrew/homebrew/jenv.rb
    253 <div class="highlighter-rouge"><pre class="highlight"><code>$ brew install https://raw.githubusercontent.com/entrypass/jenv/homebrew/homebrew/jenv.rb
    246 254 </code></pre>
    255 </div>
    247 256
    248 257 <p>and make sure activate it automatically:</p>
    249 258
    250 <pre><code>$ echo 'eval "$(jenv init -)"' &gt;&gt; ~/.bash_profile
    259 <div class="highlighter-rouge"><pre class="highlight"><code>$ echo 'eval "$(jenv init -)"' &gt;&gt; ~/.bash_profile
    251 260 </code></pre>
    261 </div>
    252 262
    253 <p><strong>Note:</strong>
    254 - There is a known issue at this writing: https://github.com/gcuisinier/jenv/wiki/Trouble-Shooting
    255 - Please make sure JENV_ROOT has been set before jenv init:
    256 - $ export JENV_ROOT=/usr/local/opt/jenv</p>
    263 <p><strong>Note:</strong></p>
    264 <ul>
    265 <li>There is a known issue at this writing: https://github.com/gcuisinier/jenv/wiki/Trouble-Shooting</li>
    266 <li>Please make sure JENV_ROOT has been set before jenv init:</li>
    267 <li>$ export JENV_ROOT=/usr/local/opt/jenv</li>
    268 </ul>
    257 269
    258 270 <p>Now let Jenv manage JDK versions (remember In OSX all JVMs are located at /Library/Java/JavaVirtualMachines):</p>
    259 271
    260 <pre><code>$ jenv add /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/
    272 <div class="highlighter-rouge"><pre class="highlight"><code>$ jenv add /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/
    261 273 $ jenv add /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/
    262 274 </code></pre>
    275 </div>
    263 276
    264 277 <p>and</p>
    265 278
    266 <pre><code>$ jenv rehash
    279 <div class="highlighter-rouge"><pre class="highlight"><code>$ jenv rehash
    267 280 </code></pre>
    281 </div>
    268 282
    269 283 <p>You can see all managed JDK versions:</p>
    270 284
    271 <pre><code>$ jenv versions
    285 <div class="highlighter-rouge"><pre class="highlight"><code>$ jenv versions
    272 286 </code></pre>
    287 </div>
    273 288
    274 289 <p>set global java version:</p>
    275 290
    276 <pre><code>$ jenv global oracle64-1.8.0.66
    291 <div class="highlighter-rouge"><pre class="highlight"><code>$ jenv global oracle64-1.8.0.66
    277 292 </code></pre>
    293 </div>
    278 294
    279 295 <p>switch to your eagle home directory and set the local JDK version for eagle:</p>
    280 296
    281 <pre><code>$ jenv local oracle64-1.7.0.80
    297 <div class="highlighter-rouge"><pre class="highlight"><code>$ jenv local oracle64-1.7.0.80
    282 298 </code></pre>
    299 </div>
    283 300
    284 301 <p>you can find more information about Jenv at https://github.com/rbenv/rbenv and http://hanxue-it.blogspot.com/2014/05/installing-java-8-managing-multiple.html.</p>
    285 302
     
    287 304
    288 305 <p>Go to Eagle home directory and run:</p>
    289 306
    290 <pre><code>mvn -DskipTests clean package
    307 <div class="highlighter-rouge"><pre class="highlight"><code>mvn -DskipTests clean package
    291 308 </code></pre>
    309 </div>
    292 310
    293 311 <p>That’s all. Now you have runnable eagle on your Mac. Have fun. :-)</p>
    294 312
  • eagle/site/docs/download-latest.html

     
    157 157 <div class="col-xs-6 col-sm-9 page-main-content" style="margin-left: -15px" id="loadcontent">
    158 158 <h1 class="page-header" style="margin-top: 0px">Apache Eagle Latest Download</h1>
    159 159 <blockquote>
    160 <p>Version <code>0.4.0-incubating</code> is the latest release and <code>0.5.0-SNAPSHOT</code> is under active development on <a href="https://github.com/apache/eagle/tree/master">master</a> branch.</p>
    160 <p>Version <strong>0.5.0</strong> is the latest release and 0.5.0-SNAPSHOT is under active development on <a href="https://github.com/apache/eagle/tree/master">master</a> branch.</p>
    161 161
    162 162 <p>You can verify your download by following these <a href="https://www.apache.org/info/verification.html">procedures</a> and using these <a href="https://dist.apache.org/repos/dist/release/eagle/KEYS">KEYS</a>.</p>
    163 163 </blockquote>
    164 164
    165 <h1 id="snapshot">0.5.0-SNAPSHOT</h1>
    166
    167 <blockquote>
    168 <p>The first GA version <code>v0.5.0</code> with fantastic improvements and features is coming soon!</p>
    169 </blockquote>
    170
    165 <h1 id="050">0.5.0</h1>
    171 166 <ul>
    172 <li>
    173 <p>Build from source code:</p>
    174
    175 <pre><code> git clone https://github.com/apache/eagle.git
    176 </code></pre>
    177 </li>
    178 <li>Release notes for preview:
    179 <ul>
    180 <li><a href="https://cwiki.apache.org/confluence/display/EAG/Eagle+Version+0.5.0">Eagle 0.5.0 Release Notes</a></li>
    181 </ul>
    182 </li>
    183 <li>Documentation:
    184 <ul>
    185 <li><a href="/docs/latest/">Eagle 0.5.0 Documentations</a></li>
    186 </ul>
    187 </li>
    188 </ul>
    189
    190 <h1 id="incubating">0.4.0-incubating</h1>
    191
    192 <p><a href="http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.eagle%22%20AND%20a%3A%22eagle-parent%22"><img src="https://maven-badges.herokuapp.com/maven-central/org.apache.eagle/eagle-parent/badge.svg" alt="Eagle Latest Maven Release" /></a></p>
    193
    194 <ul>
    195 167 <li>Release notes:
    196 168 <ul>
    197 <li><a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=blob_plain;f=CHANGELOG.txt;hb=refs/tags/v0.4.0-incubating">Eagle 0.4.0 Release Notes</a></li>
    169 <li><a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=blob_plain;f=CHANGELOG.txt;hb=refs/tags/v0.5.0">Eagle 0.5.0 Release Notes</a></li>
    198 170 </ul>
    199 171 </li>
    200 172 <li>Source download:
    201 173 <ul>
    202 <li><a href="http://www.apache.org/dyn/closer.cgi?path=/eagle/apache-eagle-0.4.0-incubating">apache-eagle-0.4.0-incubating-src.tar.gz</a></li>
    203 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.4.0-incubating/apache-eagle-0.4.0-incubating-src.tar.gz.md5">apache-eagle-0.4.0-incubating-src.tar.gz.md5</a></li>
    204 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.4.0-incubating/apache-eagle-0.4.0-incubating-src.tar.gz.sha1">apache-eagle-0.4.0-incubating-src.tar.gz.sha1</a></li>
    205 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.4.0-incubating/apache-eagle-0.4.0-incubating-src.tar.gz.asc">apache-eagle-0.4.0-incubating-src.tar.gz.asc</a></li>
    174 <li><a href="http://www.apache.org/dyn/closer.cgi?path=/incubator/eagle/apache-eagle-0.5.0-incubating">apache-eagle-0.5.0-src.tar.gz</a></li>
    175 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.5.0/apache-eagle-0.5.0-src.tar.gz.md5">apache-eagle-0.5.0-src.tar.gz.md5</a></li>
    176 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.5.0/apache-eagle-0.5.0-src.tar.gz.sha1">apache-eagle-0.5.0-src.tar.gz.sha1</a></li>
    177 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.5.0/apache-eagle-0.5.0-src.tar.gz.asc">apache-eagle-0.5.0-src.tar.gz.asc</a></li>
    206 178 </ul>
    207 179 </li>
    208 180 <li>Git revision:
    209 181 <ul>
    210 <li>tag: <a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=commit;h=refs/tags/v0.4.0-incubating">v0.4.0-incubating</a></li>
    211 <li>commit: <a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=commit;h=eac0f27958f2ed8c6842938dad0a995a87fd0715">eac0f27958f2ed8c6842938dad0a995a87fd0715</a></li>
    182 <li>tag: <a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=commit;h=refs/tags/v0.5.0">v0.5.0</a></li>
    212 183 </ul>
    213 184 </li>
    214 185 </ul>
     
    216 187 <h1 id="previous-releases">Previous Releases</h1>
    217 188
    218 189 <ul>
    190 <li><a href="/docs/download.html#0.4.0-incubating">Eagle 0.4.0-incubating</a></li>
    219 191 <li><a href="/docs/download.html#0.3.0-incubating">Eagle 0.3.0-incubating</a></li>
    220 192 </ul>
    221 193
  • eagle/site/docs/download.html

     
    160 160 <p>You can verify your download by following these <a href="https://www.apache.org/info/verification.html">procedures</a> and using these <a href="https://dist.apache.org/repos/dist/release/eagle/KEYS">KEYS</a>.</p>
    161 161 </blockquote>
    162 162
    163 <h1 id="incubating">0.4.0-incubating</h1>
    163 <h1 id="050">0.5.0</h1>
    164 164 <ul>
    165 165 <li>Release notes:
    166 166 <ul>
    167 <li><a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=blob_plain;f=CHANGELOG.txt;hb=refs/tags/v0.5.0">Eagle 0.5.0 Release Notes</a></li>
    168 </ul>
    169 </li>
    170 <li>Source download:
    171 <ul>
    172 <li><a href="http://www.apache.org/dyn/closer.cgi?path=/incubator/eagle/apache-eagle-0.5.0-incubating">apache-eagle-0.5.0-src.tar.gz</a></li>
    173 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.5.0/apache-eagle-0.5.0-src.tar.gz.md5">apache-eagle-0.5.0-src.tar.gz.md5</a></li>
    174 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.5.0/apache-eagle-0.5.0-src.tar.gz.sha1">apache-eagle-0.5.0-src.tar.gz.sha1</a></li>
    175 <li><a href="https://dist.apache.org/repos/dist/release/eagle/apache-eagle-0.5.0/apache-eagle-0.5.0-src.tar.gz.asc">apache-eagle-0.5.0-src.tar.gz.asc</a></li>
    176 </ul>
    177 </li>
    178 <li>Git revision:
    179 <ul>
    180 <li>tag: <a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=commit;h=refs/tags/v0.5.0">v0.5.0</a></li>
    181 <li>commit: <a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=commit;h=c930a7ab4a4fb78a1cbacd8f8419de3b5dbf1bd7">c930a7ab4a4fb78a1cbacd8f8419de3b5dbf1bd7</a></li>
    182 </ul>
    183 </li>
    184 </ul>
    185
    186 <h1 id="040-incubating">0.4.0-incubating</h1>
    187 <ul>
    188 <li>Release notes:
    189 <ul>
    167 190 <li><a href="https://git-wip-us.apache.org/repos/asf?p=eagle.git;a=blob_plain;f=CHANGELOG.txt;hb=refs/tags/v0.4.0-incubating">Eagle 0.4.0 Release Notes</a></li>
    168 191 </ul>
    169 192 </li>
     
    183 206 </li>
    184 207 </ul>
    185 208
    186 <h1 id="incubating-1">0.3.0-incubating</h1>
    209 <h1 id="030-incubating">0.3.0-incubating</h1>
    187 210
    188 211 <ul>
    189 212 <li>Release notes:
  • eagle/site/docs/FAQ.html

     
    164 164 <p>Add the following line in host machine’s hosts file</p>
    165 165 </blockquote>
    166 166
    167 <pre><code>127.0.0.1 sandbox.hortonworks.com
    167 <div class="highlighter-rouge"><pre class="highlight"><code>127.0.0.1 sandbox.hortonworks.com
    168 168 </code></pre>
    169 </div>
    169 170 </li>
    170 171 <li>
    171 172 <p><strong>Q2. Not able to send data into kafka using kafka console producer</strong>:</p>
    172 173
    173 <pre><code>/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic sandbox_hdfs_audit_log
    174 <div class="highlighter-rouge"><pre class="highlight"><code>/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic sandbox_hdfs_audit_log
    174 175 </code></pre>
    176 </div>
    175 177
    176 178 <blockquote>
    177 179 <p>Apache Kafka broker are binding to host sandbox.hortonworks.com</p>
  • eagle/site/docs/hbase-auth-activity-monitoring.html

     
    160 160
    161 161 <p>Please follow below steps to enable HBase authorization auditing in HDP sandbox and Cloudera</p>
    162 162
    163 <h4 id="in-hbase-sitexml">1. in hbase-site.xml</h4>
    163 <h4 id="1-in-hbase-sitexml">1. in hbase-site.xml</h4>
    164 164
    165 165 <p>Note: when testing in HDP sandbox, sometimes Apache Ranger will take over access controll for HBase, so maybe you need change that back to native hbase access controller, i.e. change com.xasecure.authorization.hbase.XaSecureAuthorizationCoprocessor to org.apache.hadoop.hbase.security.access.AccessController</p>
    166 166
    167 <pre><code>&lt;property&gt;
    167 <div class="highlighter-rouge"><pre class="highlight"><code>&lt;property&gt;
    168 168 &lt;name&gt;hbase.security.authorization&lt;/name&gt;
    169 169 &lt;value&gt;true&lt;/value&gt;
    170 170 &lt;/property&gt;
     
    177 177 &lt;value&gt;org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController&lt;/value&gt;
    178 178 &lt;/property&gt;
    179 179 </code></pre>
    180 </div>
    180 181
    181 <h4 id="log4jproperties">2. log4j.properties</h4>
    182 <h4 id="2-log4jproperties">2. log4j.properties</h4>
    182 183
    183 <pre><code>#
    184 <div class="highlighter-rouge"><pre class="highlight"><code>#
    184 185 # Security audit appender
    185 186 #
    186 187 hbase.security.log.file=SecurityAuth.audit
     
    196 197 log4j.additivity.SecurityLogger=false
    197 198 log4j.logger.SecurityLogger.org.apache.hadoop.hbase.security.access.AccessController=TRACE
    198 199 </code></pre>
    200 </div>
    199 201
    200 202 <hr />
    201 203
     
    204 206 <div class="footnotes">
    205 207 <ol>
    206 208 <li id="fn:HBASE">
    207 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    209 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    208 210 </li>
    209 211 </ol>
    210 212 </div>
  • eagle/site/docs/hbase-data-activity-monitoring.html

     
    174 174
    175 175 <ol>
    176 176 <li>
    177 <p>edit Advanced hbase-log4j via Ambari<sup id="fnref:AMBARI"><a href="#fn:AMBARI" class="footnote">3</a></sup> UI, and append below sentence to <code>Security audit appender</code></p>
    177 <p>edit Advanced hbase-log4j via Ambari<sup id="fnref:AMBARI"><a href="#fn:AMBARI" class="footnote">3</a></sup> UI, and append below sentence to <code class="highlighter-rouge">Security audit appender</code></p>
    178 178
    179 <pre><code> log4j.logger.SecurityLogger.org.apache.hadoop.hbase.security.access.AccessController=TRACE,RFAS
    179 <div class="highlighter-rouge"><pre class="highlight"><code> log4j.logger.SecurityLogger.org.apache.hadoop.hbase.security.access.AccessController=TRACE,RFAS
    180 180 </code></pre>
    181 </div>
    181 182 </li>
    182 183 <li>
    183 184 <p>edit Advanced hbase-site.xml</p>
    184 185
    185 <pre><code> &lt;property&gt;
    186 <div class="highlighter-rouge"><pre class="highlight"><code> &lt;property&gt;
    186 187 &lt;name&gt;hbase.security.authorization&lt;/name&gt;
    187 188 &lt;value&gt;true&lt;/value&gt;
    188 189 &lt;/property&gt;
     
    197 198 &lt;value&gt;org.apache.hadoop.hbase.security.access.AccessController&lt;/value&gt;
    198 199 &lt;/property&gt;
    199 200 </code></pre>
    201 </div>
    200 202 </li>
    201 203 <li>
    202 204 <p>Save and restart HBase</p>
     
    208 210 <h3 id="how-to-add-a-kafka-log4j-appender">How to add a Kafka log4j appender</h3>
    209 211
    210 212 <blockquote>
    211 <p>Notice: if you are willing to use sample logs under <code>eagle-security-hbase-security/test/resources/securityAuditLog</code>, please skip this part.</p>
    213 <p>Notice: if you are willing to use sample logs under <code class="highlighter-rouge">eagle-security-hbase-security/test/resources/securityAuditLog</code>, please skip this part.</p>
    212 214 </blockquote>
    213 215
    214 216 <ol>
    215 217 <li>
    216 <p>create Kafka topic <code>sandbox_hbase_security_log</code></p>
    218 <p>create Kafka topic <code class="highlighter-rouge">sandbox_hbase_security_log</code></p>
    217 219
    218 <pre><code> $ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hbase_security_log
    220 <div class="highlighter-rouge"><pre class="highlight"><code> $ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hbase_security_log
    219 221 </code></pre>
    222 </div>
    220 223 </li>
    221 224 <li>
    222 <p>add below “KAFKA_HBASE_AUDIT” log4j appender to <code>Security audit appender</code>
    225 <p>add below “KAFKA_HBASE_AUDIT” log4j appender to <code class="highlighter-rouge">Security audit appender</code>
    223 226 Please refer to http://goeagle.io/docs/import-hdfs-auditLog.html.</p>
    224 227
    225 <pre><code> log4j.appender.KAFKA_HBASE_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
    228 <div class="highlighter-rouge"><pre class="highlight"><code> log4j.appender.KAFKA_HBASE_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
    226 229 log4j.appender.KAFKA_HBASE_AUDIT.Topic=sandbox_hbase_security_log
    227 230 log4j.appender.KAFKA_HBASE_AUDIT.BrokerList=sandbox.hortonworks.com:6667
    228 231 log4j.appender.KAFKA_HBASE_AUDIT.Layout=org.apache.log4j.PatternLayout
     
    231 234 log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.GenericLogKeyer
    232 235 log4j.appender.KAFKA_HDFS_AUDIT.KeyPattern=user=(\\w+),\\s+
    233 236 </code></pre>
    237 </div>
    234 238 </li>
    235 239 <li>
    236 240 <p>add the reference to KAFKA_HBASE_AUDIT to log4j appender</p>
    237 241
    238 <pre><code> log4j.logger.SecurityLogger.org.apache.hadoop.hbase.security.access.AccessController=TRACE,RFAS,KAFKA_HBASE_AUDIT
    242 <div class="highlighter-rouge"><pre class="highlight"><code> log4j.logger.SecurityLogger.org.apache.hadoop.hbase.security.access.AccessController=TRACE,RFAS,KAFKA_HBASE_AUDIT
    239 243 </code></pre>
    244 </div>
    240 245 </li>
    241 246 <li>
    242 247 <p>add Eagle log4j appender jars into HBASE_CLASSPATH BY editing Advanced hbase-env via Ambari UI</p>
    243 248
    244 <pre><code> export HBASE_CLASSPATH=${HBASE_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
    249 <div class="highlighter-rouge"><pre class="highlight"><code> export HBASE_CLASSPATH=${HBASE_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
    245 250 </code></pre>
    251 </div>
    246 252 </li>
    247 253 <li>
    248 254 <p>Save and restart HBase</p>
     
    253 259
    254 260 <ol>
    255 261 <li>
    256 <p>create tables (<code>skip if you do not use hbase</code>)</p>
    262 <p>create tables (<code class="highlighter-rouge">skip if you do not use hbase</code>)</p>
    257 263
    258 <pre><code> bin/eagle-service-init.sh
    264 <div class="highlighter-rouge"><pre class="highlight"><code> bin/eagle-service-init.sh
    259 265 </code></pre>
    266 </div>
    260 267 </li>
    261 268 <li>
    262 269 <p>start Eagle service</p>
    263 270
    264 <pre><code> bin/eagle-service.sh start
    271 <div class="highlighter-rouge"><pre class="highlight"><code> bin/eagle-service.sh start
    265 272 </code></pre>
    273 </div>
    266 274 </li>
    267 275 <li>
    268 276 <p>import metadata</p>
    269 277
    270 <pre><code> bin/eagle-topology-init.sh
    278 <div class="highlighter-rouge"><pre class="highlight"><code> bin/eagle-topology-init.sh
    271 279 </code></pre>
    280 </div>
    272 281 </li>
    273 282 <li>
    274 283 <p>submit topology</p>
    275 284
    276 <pre><code> bin/eagle-topology.sh --main org.apache.eagle.security.hbase.HbaseAuditLogProcessorMain --config conf/sandbox-hbaseSecurityLog-application.conf start
    285 <div class="highlighter-rouge"><pre class="highlight"><code> bin/eagle-topology.sh --main org.apache.eagle.security.hbase.HbaseAuditLogProcessorMain --config conf/sandbox-hbaseSecurityLog-application.conf start
    277 286 </code></pre>
    287 </div>
    278 288 </li>
    279 289 </ol>
    280 290
    281 <p>(sample sensitivity data at <code>examples/sample-sensitivity-resource-create.sh</code>)</p>
    291 <p>(sample sensitivity data at <code class="highlighter-rouge">examples/sample-sensitivity-resource-create.sh</code>)</p>
    282 292
    283 293 <h3 id="q--a">Q &amp; A</h3>
    284 294
     
    293 303 <div class="footnotes">
    294 304 <ol>
    295 305 <li id="fn:HBASE">
    296 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    306 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    297 307 </li>
    298 308 <li id="fn:KAFKA">
    299 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    309 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    300 310 </li>
    301 311 <li id="fn:AMBARI">
    302 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    312 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em>&nbsp;<a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    303 313 </li>
    304 314 </ol>
    305 315 </div>
  • eagle/site/docs/hdfs-auth-activity-monitoring.html

     
    160 160
    161 161 <h4 id="sample-authorization-logs">Sample authorization logs</h4>
    162 162
    163 <pre><code>2016-06-08 02:55:07,742 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for hdfs (auth:SIMPLE) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol
    163 <div class="highlighter-rouge"><pre class="highlight"><code>2016-06-08 02:55:07,742 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for hdfs (auth:SIMPLE) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol
    164 164 2016-06-08 02:55:35,304 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for hdfs (auth:SIMPLE) for protocol=interface org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol
    165 165 2016-06-08 02:55:36,862 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for hive (auth:SIMPLE) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol
    166 166 </code></pre>
    167 </div>
    167 168
    168 169 <p>Steps for enabling service-level authorization activity</p>
    169 170
    170 <h4 id="enable-hdfs-authorization-security-in-core-sitexml">1. Enable HDFS Authorization Security in core-site.xml</h4>
    171 <h4 id="1-enable-hdfs-authorization-security-in-core-sitexml">1. Enable HDFS Authorization Security in core-site.xml</h4>
    171 172
    172 <pre><code> &lt;property&gt;
    173 <div class="highlighter-rouge"><pre class="highlight"><code> &lt;property&gt;
    173 174 &lt;name&gt;hadoop.security.authorization&lt;/name&gt;
    174 175 &lt;value&gt;true&lt;/value&gt;
    175 176 &lt;/property&gt;
    176 177 </code></pre>
    178 </div>
    177 179
    178 <h4 id="enable-hdfs-security-log-in-log4jproperties">2. Enable HDFS security log in log4j.properties</h4>
    179 <pre><code>#
    180 <h4 id="2-enable-hdfs-security-log-in-log4jproperties">2. Enable HDFS security log in log4j.properties</h4>
    181 <div class="highlighter-rouge"><pre class="highlight"><code>#
    180 182 #Security audit appender
    181 183 #
    182 184 hadoop.security.logger=INFO,DRFAS
     
    190 192 log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
    191 193 log4j.appender.DRFAS.DatePattern=.yyyy-MM-dd
    192 194 </code></pre>
    195 </div>
    193 196
    194 197 </div><!--end of loadcontent-->
    195 198 </div>
  • eagle/site/docs/hdfs-data-activity-monitoring.html

     
    181 181 <li>
    182 182 <p><strong>Step 1</strong>: Configure Advanced hdfs-log4j via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a><sup id="fnref:AMBARI"><a href="#fn:AMBARI" class="footnote">2</a></sup>, by adding below “KAFKA_HDFS_AUDIT” log4j appender to hdfs audit logging.</p>
    183 183
    184 <pre><code> log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
    184 <div class="highlighter-rouge"><pre class="highlight"><code> log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
    185 185 log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log
    186 186 log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667
    187 187 log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer
     
    189 189 log4j.appender.KAFKA_HDFS_AUDIT.Layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
    190 190 log4j.appender.KAFKA_HDFS_AUDIT.ProducerType=async
    191 191 </code></pre>
    192 </div>
    192 193
    193 194 <p><img src="/images/docs/hdfs-log4j-conf.png" alt="HDFS LOG4J Configuration" title="hdfslog4jconf" /></p>
    194 195 </li>
    195 196 <li>
    196 197 <p><strong>Step 2</strong>: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a>, and add the reference to KAFKA_HDFS_AUDIT to HADOOP_NAMENODE_OPTS.</p>
    197 198
    198 <pre><code>-Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
    199 <div class="highlighter-rouge"><pre class="highlight"><code>-Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
    199 200 </code></pre>
    201 </div>
    200 202
    201 203 <p><img src="/images/docs/hdfs-env-conf.png" alt="HDFS Environment Configuration" title="hdfsenvconf" /></p>
    202 204 </li>
    203 205 <li>
    204 206 <p><strong>Step 3</strong>: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs" target="_blank">Ambari UI</a>, and append the following command to it.</p>
    205 207
    206 <pre><code>export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
    208 <div class="highlighter-rouge"><pre class="highlight"><code>export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
    207 209 </code></pre>
    210 </div>
    208 211
    209 212 <p><img src="/images/docs/hdfs-env-conf2.png" alt="HDFS Environment Configuration" title="hdfsenvconf2" /></p>
    210 213 </li>
     
    223 226
    224 227 <ul>
    225 228 <li>
    226 <p><strong>Step 7</strong>: Check whether logs from “/var/log/hadoop/hdfs/hdfs-audit.log” are flowing into topic <code>sandbox_hdfs_audit_log</code></p>
    229 <p><strong>Step 7</strong>: Check whether logs from “/var/log/hadoop/hdfs/hdfs-audit.log” are flowing into topic <code class="highlighter-rouge">sandbox_hdfs_audit_log</code></p>
    227 230
    228 <pre><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic sandbox_hdfs_audit_log
    231 <div class="highlighter-rouge"><pre class="highlight"><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic sandbox_hdfs_audit_log
    229 232 </code></pre>
    233 </div>
    230 234 </li>
    231 235 </ul>
    232 236
     
    260 264 <div class="footnotes">
    261 265 <ol>
    262 266 <li id="fn:KAFKA">
    263 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    267 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    264 268 </li>
    265 269 <li id="fn:AMBARI">
    266 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    270 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em>&nbsp;<a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    267 271 </li>
    268 272 <li id="fn:STORM">
    269 <p><em>Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    273 <p><em>Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    270 274 </li>
    271 275 </ol>
    272 276 </div>
  • eagle/site/docs/hive-query-activity-monitoring.html

     
    187 187 </li>
    188 188 </ul>
    189 189
    190 <pre><code>$ su hive
    190 <div class="highlighter-rouge"><pre class="highlight"><code>$ su hive
    191 191 $ hive
    192 192 $ set hive.execution.engine=mr;
    193 193 $ use xademo;
    194 194 $ select a.phone_number from customer_details a, call_detail_records b where a.phone_number=b.phone_number;
    195 195 </code></pre>
    196 </div>
    196 197
    197 198 <p>From UI click on alert tab and you should see alert for your attempt to read restricted column.</p>
    198 199
     
    203 204 <div class="footnotes">
    204 205 <ol>
    205 206 <li id="fn:HIVE">
    206 <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    207 <p><em>All mentions of “hive” on this page represent Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    207 208 </li>
    208 209 </ol>
    209 210 </div>
  • eagle/site/docs/import-hdfs-auditLog.html

     
    169 169
    170 170 <p>Here is an sample Kafka command to create topic ‘sandbox_hdfs_audit_log’</p>
    171 171
    172 <pre><code>cd &lt;kafka-home&gt;
    172 <div class="highlighter-rouge"><pre class="highlight"><code>cd &lt;kafka-home&gt;
    173 173 bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hdfs_audit_log
    174 174 </code></pre>
    175 </div>
    175 176 </li>
    176 177 <li>
    177 178 <p><strong>Step 2</strong>: Install Logstash-kafka plugin</p>
     
    189 190 <li>
    190 191 <p><strong>Step 3</strong>: Create a Logstash configuration file under ${LOGSTASH_HOME}/conf. Here is a sample.</p>
    191 192
    192 <pre><code> input {
    193 <div class="highlighter-rouge"><pre class="highlight"><code> input {
    193 194 file {
    194 195 type =&gt; "hdp-nn-audit"
    195 196 path =&gt; "/path/to/audit.log"
     
    230 231 }
    231 232 }
    232 233 </code></pre>
    234 </div>
    233 235 </li>
    234 236 <li>
    235 237 <p><strong>Step 4</strong>: Start Logstash</p>
    236 238
    237 <pre><code>bin/logstash -f conf/sample.conf
    239 <div class="highlighter-rouge"><pre class="highlight"><code>bin/logstash -f conf/sample.conf
    238 240 </code></pre>
    241 </div>
    239 242 </li>
    240 243 <li>
    241 <p><strong>Step 5</strong>: Check whether logs are flowing into the kafka topic specified by <code>topic_id</code></p>
    244 <p><strong>Step 5</strong>: Check whether logs are flowing into the kafka topic specified by <code class="highlighter-rouge">topic_id</code></p>
    242 245 </li>
    243 246 </ul>
    244 247
     
    252 255 <li>
    253 256 <p><strong>Step 1</strong>: Create a Kafka topic. Here is a example Kafka command for creating topic “sandbox_hdfs_audit_log”</p>
    254 257
    255 <pre><code>cd &lt;kafka-home&gt;
    258 <div class="highlighter-rouge"><pre class="highlight"><code>cd &lt;kafka-home&gt;
    256 259 bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hdfs_audit_log
    257 260 </code></pre>
    261 </div>
    258 262 </li>
    259 263 <li>
    260 264 <p><strong>Step 2</strong>: Configure $HADOOP_CONF_DIR/log4j.properties, and add a log4j appender “KAFKA_HDFS_AUDIT” to hdfs audit logging</p>
    261 265
    262 <pre><code>log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
    266 <div class="highlighter-rouge"><pre class="highlight"><code>log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
    263 267 log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log
    264 268 log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667
    265 269 log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer
     
    269 273 #log4j.appender.KAFKA_HDFS_AUDIT.BatchSize=1
    270 274 #log4j.appender.KAFKA_HDFS_AUDIT.QueueSize=1
    271 275 </code></pre>
    276 </div>
    272 277
    273 278 <p><img src="/images/docs/hdfs-log4j-conf.png" alt="HDFS LOG4J Configuration" title="hdfslog4jconf" /></p>
    274 279 </li>
    275 280 <li>
    276 281 <p><strong>Step 3</strong>: Edit $HADOOP_CONF_DIR/hadoop-env.sh, and add the reference to KAFKA_HDFS_AUDIT to HADOOP_NAMENODE_OPTS.</p>
    277 282
    278 <pre><code>-Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
    283 <div class="highlighter-rouge"><pre class="highlight"><code>-Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
    279 284 </code></pre>
    285 </div>
    280 286
    281 287 <p><img src="/images/docs/hdfs-env-conf.png" alt="HDFS Environment Configuration" title="hdfsenvconf" /></p>
    282 288 </li>
    283 289 <li>
    284 290 <p><strong>Step 4</strong>: Edit $HADOOP_CONF_DIR/hadoop-env.sh, and append the following command to it.</p>
    285 291
    286 <pre><code>export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/path/to/eagle/lib/log4jkafka/lib/*
    292 <div class="highlighter-rouge"><pre class="highlight"><code>export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/path/to/eagle/lib/log4jkafka/lib/*
    287 293 </code></pre>
    294 </div>
    288 295
    289 296 <p><img src="/images/docs/hdfs-env-conf2.png" alt="HDFS Environment Configuration" title="hdfsenvconf2" /></p>
    290 297 </li>
     
    294 301 <li>
    295 302 <p><strong>Step 6</strong>: Check whether logs are flowing into Topic sandbox_hdfs_audit_log</p>
    296 303
    297 <pre><code>$ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log
    304 <div class="highlighter-rouge"><pre class="highlight"><code>$ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log
    298 305 </code></pre>
    306 </div>
    299 307 </li>
    300 308 </ul>
    301 309
     
    306 314 <div class="footnotes">
    307 315 <ol>
    308 316 <li id="fn:KAFKA">
    309 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    317 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    310 318 </li>
    311 319 <li id="fn:AMBARI">
    312 <p><em>all mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    320 <p><em>all mentions of “ambari” on this page represent Apache Ambari.</em>&nbsp;<a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    313 321 </li>
    314 322 </ol>
    315 323 </div>
  • eagle/site/docs/index.html

     
    204 204 <div class="footnotes">
    205 205 <ol>
    206 206 <li id="fn:HADOOP">
    207 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    207 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    208 208 </li>
    209 209 <li id="fn:SPARK">
    210 <p><em>All mentions of “spark” on this page represent Apache Spark.</em> <a href="#fnref:SPARK" class="reversefootnote">&#8617;</a></p>
    210 <p><em>All mentions of “spark” on this page represent Apache Spark.</em>&nbsp;<a href="#fnref:SPARK" class="reversefootnote">&#8617;</a></p>
    211 211 </li>
    212 212 <li id="fn:HIVE">
    213 <p><em>All mentions of “hive” on this page represent Apache HIVE.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    213 <p><em>All mentions of “hive” on this page represent Apache HIVE.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    214 214 </li>
    215 215 </ol>
    216 216 </div>
  • eagle/site/docs/installation.html

     
    180 180 <h4 id="install-eagle">Install Eagle</h4>
    181 181
    182 182 <ul>
    183 <li>
    184 <p><strong>Step 1</strong>: Clone stable version from <a href="https://github.com/apache/eagle/releases/tag/v0.4.0-incubating">eagle github</a>
    185 &gt; Build project mvn clean install -DskipTests=true</p>
    183 <li><strong>Step 1</strong>: Clone stable version from <a href="https://github.com/apache/eagle/releases/tag/v0.4.0-incubating">eagle github</a>
    184 <blockquote>
    185 <div class="highlighter-rouge"><pre class="highlight"><code> Build project mvn clean install -DskipTests=true
    186 </code></pre>
    187 </div>
    188 </blockquote>
    186 189 </li>
    187 190 <li>
    188 191 <p><strong>Step 2</strong>: Download eagle-bin-0.1.0.tar.gz package from successful build into your HDP sandbox.</p>
    189 192
    190 193 <ul>
    191 194 <li>
    192 <p>Option 1: <code>scp -P 2222 eagle/eagle-assembly/target/eagle-0.1.0-bin.tar.gz root@127.0.0.1:/usr/hdp/current/</code></p>
    195 <p>Option 1: <code class="highlighter-rouge">scp -P 2222 eagle/eagle-assembly/target/eagle-0.1.0-bin.tar.gz root@127.0.0.1:/usr/hdp/current/</code></p>
    193 196 </li>
    194 197 <li>
    195 198 <p>Option 2: Create shared directory between host and Sandbox, and restart Sandbox. Then you can find the shared directory under /media in Sandbox.</p>
     
    199 202 <li>
    200 203 <p><strong>Step 3</strong>: Extract eagle tarball package</p>
    201 204
    202 <pre><code>$ cd /usr/hdp/current
    205 <div class="highlighter-rouge"><pre class="highlight"><code>$ cd /usr/hdp/current
    203 206 $ tar -zxvf eagle-0.1.0-bin.tar.gz
    204 207 $ mv eagle-0.1.0 eagle
    205 208 </code></pre>
    209 </div>
    206 210 </li>
    207 211 <li>
    208 212 <p><strong>Step 4</strong>: Add root as a HBase<sup id="fnref:HBASE"><a href="#fn:HBASE" class="footnote">1</a></sup> superuser via <a href="http://127.0.0.1:8080/#/main/services/HBASE/configs">Ambari</a> (Optional, a user can operate HBase by sudo su hbase, as an alternative).</p>
    209 213 </li>
    210 <li>
    211 <p><strong>Step 5</strong>: Install Eagle Ambari<sup id="fnref:AMBARI"><a href="#fn:AMBARI" class="footnote">2</a></sup> service
    212 &gt;
    213 /usr/hdp/current/eagle/bin/eagle-ambari.sh install.</p>
    214 <li><strong>Step 5</strong>: Install Eagle Ambari<sup id="fnref:AMBARI"><a href="#fn:AMBARI" class="footnote">2</a></sup> service
    215 <blockquote>
    216
    217 <p>/usr/hdp/current/eagle/bin/eagle-ambari.sh install.</p>
    218 </blockquote>
    214 219 </li>
    215 220 <li>
    216 221 <p><strong>Step 6</strong>: Restart <a href="http://127.0.0.1:8000/">Ambari</a> click on disable and enable Ambari back.</p>
    217 222 </li>
    218 <li>
    219 <p><strong>Step 7</strong>: Start HBase &amp; Storm<sup id="fnref:STORM"><a href="#fn:STORM" class="footnote">3</a></sup> &amp; Kafka<sup id="fnref:KAFKA"><a href="#fn:KAFKA" class="footnote">4</a></sup>
    220 From Ambari UI, restart any suggested components(“Restart button on top”) &amp; Start Storm (Start “Nimbus” ,”Supervisor” &amp; “Storm UI Server”), Kafka (Start “Kafka Broker”) , HBase (Start “RegionServer” and “ HBase Master”)
    221 &gt;
    222 <img src="/images/docs/Services.png" alt="Restart Services" title="Services" /></p>
    223 <li><strong>Step 7</strong>: Start HBase &amp; Storm<sup id="fnref:STORM"><a href="#fn:STORM" class="footnote">3</a></sup> &amp; Kafka<sup id="fnref:KAFKA"><a href="#fn:KAFKA" class="footnote">4</a></sup>
    224 From Ambari UI, restart any suggested components(“Restart button on top”) &amp; Start Storm (Start “Nimbus” ,”Supervisor” &amp; “Storm UI Server”), Kafka (Start “Kafka Broker”) , HBase (Start “RegionServer” and “ HBase Master”)
    225 <blockquote>
    226
    227 <p><img src="/images/docs/Services.png" alt="Restart Services" title="Services" /></p>
    228 </blockquote>
    223 229 </li>
    224 230 <li>
    225 231 <p><strong>Step 8</strong>: Add Eagle Service To Ambari. (Click For Video)</p>
     
    241 247 <li>
    242 248 <p><strong>Step 9</strong>: Add Policies and meta data required by running below script.</p>
    243 249
    244 <pre><code>$ /usr/hdp/current/eagle/examples/sample-sensitivity-resource-create.sh
    250 <div class="highlighter-rouge"><pre class="highlight"><code>$ /usr/hdp/current/eagle/examples/sample-sensitivity-resource-create.sh
    245 251 $ /usr/hdp/current/eagle/examples/sample-policy-create.sh
    246 252 </code></pre>
    253 </div>
    247 254 </li>
    248 255 </ul>
    249 256
     
    254 261 <div class="footnotes">
    255 262 <ol>
    256 263 <li id="fn:HBASE">
    257 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    264 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    258 265 </li>
    259 266 <li id="fn:AMBARI">
    260 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em> <a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    267 <p><em>All mentions of “ambari” on this page represent Apache Ambari.</em>&nbsp;<a href="#fnref:AMBARI" class="reversefootnote">&#8617;</a></p>
    261 268 </li>
    262 269 <li id="fn:STORM">
    263 <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    270 <p><em>All mentions of “storm” on this page represent Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    264 271 </li>
    265 272 <li id="fn:KAFKA">
    266 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    273 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    267 274 </li>
    268 275 </ol>
    269 276 </div>
  • eagle/site/docs/jmx-metric-monitoring.html

     
    177 177 <h3 id="setup"><strong>Setup</strong></h3>
    178 178 <p>From Hortonworks sandbox just run below setup script to Install Pyton JMX script, Create Kafka topic, update Apache Hbase tables and deploy “hadoopjmx” Storm topology.</p>
    179 179
    180 <pre><code>$ /usr/hdp/current/eagle/examples/hadoop-metric-sandbox-starter.sh
    180 <div class="highlighter-rouge"><pre class="highlight"><code>$ /usr/hdp/current/eagle/examples/hadoop-metric-sandbox-starter.sh
    181 181 $ /usr/hdp/current/eagle/examples/hadoop-metric-policy-create.sh
    182 182 </code></pre>
    183 </div>
    183 184
    184 185 <p><br /></p>
    185 186
     
    204 205 <li>
    205 206 <p>First make sure that Kafka topic “nn_jmx_metric_sandbox” is populated with JMX metric data periodically.(To make sure that python script is running)</p>
    206 207
    207 <pre><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic nn_jmx_metric_sandbox
    208 <div class="highlighter-rouge"><pre class="highlight"><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181 --topic nn_jmx_metric_sandbox
    208 209 </code></pre>
    210 </div>
    209 211 </li>
    210 212 <li>
    211 213 <p>Genrate Alert by producing alert triggering message into Kafka topic.</p>
    212 214
    213 <pre><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic nn_jmx_metric_sandbox
    215 <div class="highlighter-rouge"><pre class="highlight"><code> $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic nn_jmx_metric_sandbox
    214 216 $ {"host": "localhost", "timestamp": 1457033916718, "metric": "hadoop.namenode.fsnamesystemstate.fsstate", "component": "namenode", "site": "sandbox", "value": 1.0}
    215 217 </code></pre>
    218 </div>
    216 219 </li>
    217 220 </ul>
    218 221
     
    223 226 <div class="footnotes">
    224 227 <ol>
    225 228 <li id="fn:KAFKA">
    226 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    229 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    227 230 </li>
    228 231 <li id="fn:STORM">
    229 <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    232 <p><em>All mentions of “storm” on this page represent Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    230 233 </li>
    231 234 </ol>
    232 235 </div>
  • eagle/site/docs/mapr-integration.html

     
    171 171 <p>Here are the steps to follow:</p>
    172 172
    173 173 <h4 id="step1-enable-audit-logs-for-filesystem-operations-and-table-operations-in-mapr">Step1: Enable audit logs for FileSystem Operations and Table Operations in MapR</h4>
    174 <p>First we need to enable data auditing at all three levels: cluster level, volume level and directory,file or table level.
    175 ##### Cluster level:</p>
    174 <p>First we need to enable data auditing at all three levels: cluster level, volume level and directory,file or table level.</p>
    175 <h5 id="cluster-level">Cluster level:</h5>
    176 176
    177 <pre><code> $ maprcli audit data -cluster &lt;cluster name&gt; -enabled true
    177 <div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli audit data -cluster &lt;cluster name&gt; -enabled true
    178 178 [ -maxsize &lt;GB, defaut value is 32. When size of audit logs exceed this number, an alarm will be sent to the dashboard in the MapR Control Service &gt; ]
    179 179 [ -retention &lt;number of Days&gt; ]
    180 180 </code></pre>
    181 </div>
    181 182 <p>Example:</p>
    182 183
    183 <pre><code> $ maprcli audit data -cluster mapr.cluster.com -enabled true -maxsize 30 -retention 30
    184 <div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli audit data -cluster mapr.cluster.com -enabled true -maxsize 30 -retention 30
    184 185 </code></pre>
    186 </div>
    185 187
    186 188 <h5 id="volume-level">Volume level:</h5>
    187 189
    188 <pre><code> $ maprcli volume audit -cluster &lt;cluster name&gt; -enabled true
    190 <div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli volume audit -cluster &lt;cluster name&gt; -enabled true
    189 191 -name &lt;volume name&gt;
    190 192 [ -coalesce &lt;interval in minutes, the interval of time during which READ, WRITE, or GETATTR operations on one file from one client IP address are logged only once, if auditing is enabled&gt; ]
    191 193 </code></pre>
    194 </div>
    192 195
    193 196 <p>Example:</p>
    194 197
    195 <pre><code> $ maprcli volume audit -cluster mapr.cluster.com -name mapr.tmp -enabled true
    198 <div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli volume audit -cluster mapr.cluster.com -name mapr.tmp -enabled true
    196 199 </code></pre>
    200 </div>
    197 201
    198 202 <p>To verify that auditing is enabled for a particular volume, use this command:</p>
    199 203
    200 <pre><code> $ maprcli volume info -name &lt;volume name&gt; -json | grep -i 'audited\|coalesce'
    204 <div class="highlighter-rouge"><pre class="highlight"><code> $ maprcli volume info -name &lt;volume name&gt; -json | grep -i 'audited\|coalesce'
    201 205 </code></pre>
    206 </div>
    202 207 <p>and you should see something like this:</p>
    203 208
    204 <pre><code> "audited":1,
    209 <div class="highlighter-rouge"><pre class="highlight"><code> "audited":1,
    205 210 "coalesceInterval":60
    206 211 </code></pre>
    212 </div>
    207 213 <p>If “audited” is ‘1’ then auditing is enabled for this volume.</p>
    208 214
    209 215 <h5 id="directory-file-or-mapr-db-table-level">Directory, file, or MapR-DB table level:</h5>
    210 216
    211 <pre><code> $ hadoop mfs -setaudit on &lt;directory|file|table&gt;
    217 <div class="highlighter-rouge"><pre class="highlight"><code> $ hadoop mfs -setaudit on &lt;directory|file|table&gt;
    212 218 </code></pre>
    219 </div>
    213 220
    214 <p>To check whether Auditing is Enabled for a Directory, File, or MapR-DB Table, use <code>$ hadoop mfs -ls</code>
    221 <p>To check whether Auditing is Enabled for a Directory, File, or MapR-DB Table, use <code class="highlighter-rouge">$ hadoop mfs -ls</code>
    215 222 Example:
    216 Before enable the audit log on file <code>/tmp/dir</code>, try <code>$ hadoop mfs -ls /tmp/dir</code>, you should see something like this:</p>
    223 Before enable the audit log on file <code class="highlighter-rouge">/tmp/dir</code>, try <code class="highlighter-rouge">$ hadoop mfs -ls /tmp/dir</code>, you should see something like this:</p>
    217 224
    218 <pre><code>drwxr-xr-x Z U U - root root 0 2016-03-02 15:02 268435456 /tmp/dir
    225 <div class="highlighter-rouge"><pre class="highlight"><code>drwxr-xr-x Z U U - root root 0 2016-03-02 15:02 268435456 /tmp/dir
    219 226 p 2050.32.131328 mapr2.da.dg:5660 mapr1.da.dg:5660
    220 227 </code></pre>
    228 </div>
    221 229
    222 <p>The second <code>U</code> means auditing on this file is not enabled.
    230 <p>The second <code class="highlighter-rouge">U</code> means auditing on this file is not enabled.
    223 231 Enable auditing with this command:</p>
    224 232
    225 <pre><code>$ hadoop mfs -setaudit on /tmp/dir
    233 <div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop mfs -setaudit on /tmp/dir
    226 234 </code></pre>
    235 </div>
    227 236
    228 237 <p>Then check the auditing bit with :</p>
    229 238
    230 <pre><code>$ hadoop mfs -ls /tmp/dir
    239 <div class="highlighter-rouge"><pre class="highlight"><code>$ hadoop mfs -ls /tmp/dir
    231 240 </code></pre>
    241 </div>
    232 242
    233 243 <p>you should see something like this:</p>
    234 244
    235 <pre><code>drwxr-xr-x Z U A - root root 0 2016-03-02 15:02 268435456 /tmp/dir
    245 <div class="highlighter-rouge"><pre class="highlight"><code>drwxr-xr-x Z U A - root root 0 2016-03-02 15:02 268435456 /tmp/dir
    236 246 p 2050.32.131328 mapr2.da.dg:5660 mapr1.da.dg:5660
    237 247 </code></pre>
    248 </div>
    238 249
    239 <p>We can see the previous <code>U</code> has been changed to <code>A</code> which indicates auditing on this file is enabled.</p>
    250 <p>We can see the previous <code class="highlighter-rouge">U</code> has been changed to <code class="highlighter-rouge">A</code> which indicates auditing on this file is enabled.</p>
    240 251
    241 <p><code>Important</code>:
    252 <p><code class="highlighter-rouge">Important</code>:
    242 253 When a directory has been enabled auditing, directories/files located in this dir won’t inherit auditing, but a newly created file/dir (after enabling the auditing on this dir) in this directory will.</p>
    243 254
    244 255 <h4 id="step2-stream-log-data-into-kafka-by-using-logstash">Step2: Stream log data into Kafka by using Logstash</h4>
    245 <p>As MapR do not have name node, instead it use CLDB service, we have to use logstash to stream log data into Kafka.
    246 - First find out the nodes that have CLDB service
    247 - Then find out the location of audit log files, eg: <code>/mapr/mapr.cluster.com/var/mapr/local/mapr1.da.dg/audit/</code>, file names should be in this format: <code>FSAudit.log-2016-05-04-001.json</code>
    248 - Created a logstash conf file and run it, following this doc<a href="https://github.com/apache/eagle/blob/master/eagle-assembly/src/main/docs/logstash-kafka-conf.md">Logstash-kafka</a></p>
    256 <p>As MapR do not have name node, instead it use CLDB service, we have to use logstash to stream log data into Kafka.</p>
    257 <ul>
    258 <li>First find out the nodes that have CLDB service</li>
    259 <li>Then find out the location of audit log files, eg: <code class="highlighter-rouge">/mapr/mapr.cluster.com/var/mapr/local/mapr1.da.dg/audit/</code>, file names should be in this format: <code class="highlighter-rouge">FSAudit.log-2016-05-04-001.json</code></li>
    260 <li>Created a logstash conf file and run it, following this doc<a href="https://github.com/apache/eagle/blob/master/eagle-assembly/src/main/docs/logstash-kafka-conf.md">Logstash-kafka</a></li>
    261 </ul>
    249 262
    250 263 <h4 id="step3-set-up-maprfsauditlog-applicaiton-in-eagle-service">Step3: Set up maprFSAuditLog applicaiton in Eagle Service</h4>
    251 <p>After Eagle Service gets started, create mapFSAuditLog application using: <code>$ ./maprFSAuditLog-init.sh</code>. By default it will create maprFSAuditLog in site “sandbox”, you may need to change it to your own site.
    264 <p>After Eagle Service gets started, create mapFSAuditLog application using: <code class="highlighter-rouge">$ ./maprFSAuditLog-init.sh</code>. By default it will create maprFSAuditLog in site “sandbox”, you may need to change it to your own site.
    252 265 After these steps you are good to go.</p>
    253 266
    254 267 <p>Have fun!!! :)</p>
     
    266 279 <div class="footnotes">
    267 280 <ol>
    268 281 <li id="fn:KAFKA">
    269 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    282 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    270 283 </li>
    271 284 </ol>
    272 285 </div>
  • eagle/site/docs/quick-start-0.3.0.html

     
    178 178 <li>
    179 179 <p>Build manually with <a href="https://maven.apache.org/">Apache Maven</a>:</p>
    180 180
    181 <pre><code>$ tar -zxvf apache-eagle-0.3.0-incubating-src.tar.gz
    181 <div class="highlighter-rouge"><pre class="highlight"><code>$ tar -zxvf apache-eagle-0.3.0-incubating-src.tar.gz
    182 182 $ cd incubator-eagle-release-0.3.0-rc3
    183 183 $ curl -O https://patch-diff.githubusercontent.com/raw/apache/eagle/pull/180.patch
    184 184 $ git apply 180.patch
    185 185 $ mvn clean package -DskipTests
    186 186 </code></pre>
    187 </div>
    187 188
    188 <p>After building successfully, you will get tarball under <code>eagle-assembly/target/</code> named as <code>eagle-0.3.0-incubating-bin.tar.gz</code>
    189 <p>After building successfully, you will get tarball under <code class="highlighter-rouge">eagle-assembly/target/</code> named as <code class="highlighter-rouge">eagle-0.3.0-incubating-bin.tar.gz</code>
    189 190 <br /></p>
    190 191 </li>
    191 192 </ul>
    192 193
    193 194 <h3 id="install-eagle"><strong>Install Eagle</strong></h3>
    194 195
    195 <pre><code> $ scp -P 2222 eagle-assembly/target/eagle-0.3.0-incubating-bin.tar.gz root@127.0.0.1:/root/
    196 <div class="highlighter-rouge"><pre class="highlight"><code> $ scp -P 2222 eagle-assembly/target/eagle-0.3.0-incubating-bin.tar.gz root@127.0.0.1:/root/
    196 197 $ ssh root@127.0.0.1 -p 2222 (password is hadoop)
    197 198 $ tar -zxvf eagle-0.3.0-incubating-bin.tar.gz
    198 199 $ mv eagle-0.3.0-incubating eagle
     
    200 201 $ cd /usr/hdp/current/eagle
    201 202 $ examples/eagle-sandbox-starter.sh
    202 203 </code></pre>
    204 </div>
    203 205
    204 206 <p><br /></p>
    205 207
     
    217 219 <div class="footnotes">
    218 220 <ol>
    219 221 <li id="fn:HADOOP">
    220 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    222 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    221 223 </li>
    222 224 </ol>
    223 225 </div>
  • eagle/site/docs/quick-start.html

     
    179 179 <li>
    180 180 <p>Build manually with <a href="https://maven.apache.org/">Apache Maven</a>:</p>
    181 181
    182 <pre><code>$ tar -zxvf apache-eagle-0.4.0-incubating-src.tar.gz
    182 <div class="highlighter-rouge"><pre class="highlight"><code>$ tar -zxvf apache-eagle-0.4.0-incubating-src.tar.gz
    183 183 $ cd apache-eagle-0.4.0-incubating-src
    184 184 $ curl -O https://patch-diff.githubusercontent.com/raw/apache/eagle/pull/268.patch
    185 185 $ git apply 268.patch
    186 186 $ mvn clean package -DskipTests
    187 187 </code></pre>
    188 </div>
    188 189
    189 <p>After building successfully, you will get a tarball under <code>eagle-assembly/target/</code> named <code>apache-eagle-0.4.0-incubating-bin.tar.gz</code>
    190 <p>After building successfully, you will get a tarball under <code class="highlighter-rouge">eagle-assembly/target/</code> named <code class="highlighter-rouge">apache-eagle-0.4.0-incubating-bin.tar.gz</code>
    190 191 <br /></p>
    191 192 </li>
    192 193 </ul>
    193 194
    194 195 <h3 id="install-eagle"><strong>Install Eagle</strong></h3>
    195 196
    196 <pre><code> $ scp -P 2222 eagle-assembly/target/apache-eagle-0.4.0-incubating-bin.tar.gz root@127.0.0.1:/root/
    197 <div class="highlighter-rouge"><pre class="highlight"><code> $ scp -P 2222 eagle-assembly/target/apache-eagle-0.4.0-incubating-bin.tar.gz root@127.0.0.1:/root/
    197 198 $ ssh root@127.0.0.1 -p 2222 (password is hadoop)
    198 199 $ tar -zxvf apache-eagle-0.4.0-incubating-bin.tar.gz
    199 200 $ mv apache-eagle-0.4.0-incubating eagle
     
    201 202 $ cd /usr/hdp/current/eagle
    202 203 $ examples/eagle-sandbox-starter.sh
    203 204 </code></pre>
    205 </div>
    204 206
    205 207 <p><br /></p>
    206 208
    207 209 <h3 id="sample-application-hive-query-activity-monitoring-in-sandbox"><strong>Sample Application: Hive query activity monitoring in sandbox</strong></h3>
    208 <p>After executing <code>examples/eagle-sandbox-starter.sh</code>, you have a sample application (topology) running on the Apache Storm (check with <a href="http://sandbox.hortonworks.com:8744/index.html">storm ui</a>), and a sample policy of Hive activity monitoring defined.</p>
    210 <p>After executing <code class="highlighter-rouge">examples/eagle-sandbox-starter.sh</code>, you have a sample application (topology) running on the Apache Storm (check with <a href="http://sandbox.hortonworks.com:8744/index.html">storm ui</a>), and a sample policy of Hive activity monitoring defined.</p>
    209 211
    210 212 <p>Next you can trigger an alert by running a Hive query.</p>
    211 213
    212 <pre><code>$ su hive
    214 <div class="highlighter-rouge"><pre class="highlight"><code>$ su hive
    213 215 $ hive
    214 216 $ set hive.execution.engine=mr;
    215 217 $ use xademo;
    216 218 $ select a.phone_number from customer_details a, call_detail_records b where a.phone_number=b.phone_number;
    217 219 </code></pre>
    220 </div>
    218 221 <p><br /></p>
    219 222
    220 223 <hr />
     
    224 227 <div class="footnotes">
    225 228 <ol>
    226 229 <li id="fn:HADOOP">
    227 <p><em>Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    230 <p><em>Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    228 231 </li>
    229 232 <li id="fn:HIVE">
    230 <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    233 <p><em>All mentions of “hive” on this page represent Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    231 234 </li>
    232 235 </ol>
    233 236 </div>
  • eagle/site/docs/security.html

     
    158 158 <h1 class="page-header" style="margin-top: 0px">Apache Eagle Security</h1>
    159 159 <p>The Apache Software Foundation takes a very active stance in eliminating security problems in its software products. Apache Eagle is also responsive to such issues around its features.</p>
    160 160
    161 <p>If you have any concern regarding to Eagle’s Security or you believe a vulnerability is discovered, don’t hesitate to get connected with Aapche Security Team by sending emails to <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#115;&#101;&#099;&#117;&#114;&#105;&#116;&#121;&#064;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#115;&#101;&#099;&#117;&#114;&#105;&#116;&#121;&#064;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a>. In the message, you can indicate the project name is Eagle, provide a description of the issue, and you are recommended to give the way of reproducing it. The security team and eagle community will get back to you after assessing the findings.</p>
    161 <p>If you have any concern regarding to Eagle’s Security or you believe a vulnerability is discovered, don’t hesitate to get connected with Aapche Security Team by sending emails to <a href="mailto:security@apache.org">security@apache.org</a>. In the message, you can indicate the project name is Eagle, provide a description of the issue, and you are recommended to give the way of reproducing it. The security team and eagle community will get back to you after assessing the findings.</p>
    162 162
    163 163 <blockquote>
    164 164 <p><strong>PLEASE PAY ATTENTION</strong> to report any security problem to the security email address before disclosing it publicly.</p>
  • eagle/site/docs/serviceconfiguration.html

     
    171 171 <li>for hbase</li>
    172 172 </ul>
    173 173
    174 <pre><code>eagle {
    174 <div class="highlighter-rouge"><pre class="highlight"><code>eagle {
    175 175 service{
    176 176 storage-type="hbase"
    177 177 hbase-zookeeper-quorum="sandbox.hortonworks.com"
     
    182 182 }
    183 183 }
    184 184 </code></pre>
    185 </div>
    185 186
    186 187 <ul>
    187 188 <li>for mysql</li>
    188 189 </ul>
    189 190
    190 <pre><code>eagle {
    191 <div class="highlighter-rouge"><pre class="highlight"><code>eagle {
    191 192 service {
    192 193 storage-type="jdbc"
    193 194 storage-adapter="mysql"
     
    201 202 }
    202 203 }
    203 204 </code></pre>
    205 </div>
    204 206
    205 207 <ul>
    206 208 <li>for derby</li>
    207 209 </ul>
    208 210
    209 <pre><code>eagle {
    211 <div class="highlighter-rouge"><pre class="highlight"><code>eagle {
    210 212 service {
    211 213 storage-type="jdbc"
    212 214 storage-adapter="derby"
     
    220 222 }
    221 223 }
    222 224 </code></pre>
    225 </div>
    223 226 <p><br /></p>
    224 227
    225 228 </div><!--end of loadcontent-->
  • eagle/site/docs/terminology.html

     
    193 193 <div class="footnotes">
    194 194 <ol>
    195 195 <li id="fn:HADOOP">
    196 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    196 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    197 197 </li>
    198 198 <li id="fn:HIVE">
    199 <p><em>Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    199 <p><em>Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    200 200 </li>
    201 201 </ol>
    202 202 </div>
  • eagle/site/docs/tutorial/classification.html

     
    182 182
    183 183 <p>You may configure the default path for Apache Hadoop clients to connect remote hdfs namenode.</p>
    184 184
    185 <pre><code> classification.fs.defaultFS=hdfs://sandbox.hortonworks.com:8020
    185 <div class="highlighter-rouge"><pre class="highlight"><code> classification.fs.defaultFS=hdfs://sandbox.hortonworks.com:8020
    186 186 </code></pre>
    187 </div>
    187 188 </li>
    188 189 <li>
    189 190 <p>HA case</p>
    190 191
    191 192 <p>Basically, you point your fs.defaultFS at your nameservice and let the client know how its configured (the backing namenodes) and how to fail over between them under the HA mode</p>
    192 193
    193 <pre><code> classification.fs.defaultFS=hdfs://nameservice1
    194 <div class="highlighter-rouge"><pre class="highlight"><code> classification.fs.defaultFS=hdfs://nameservice1
    194 195 classification.dfs.nameservices=nameservice1
    195 196 classification.dfs.ha.namenodes.nameservice1=namenode1,namenode2
    196 197 classification.dfs.namenode.rpc-address.nameservice1.namenode1=hadoopnamenode01:8020
    197 198 classification.dfs.namenode.rpc-address.nameservice1.namenode2=hadoopnamenode02:8020
    198 199 classification.dfs.client.failover.proxy.provider.nameservice1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
    199 200 </code></pre>
    201 </div>
    200 202 </li>
    201 203 <li>
    202 204 <p>Kerberos-secured cluster</p>
    203 205
    204 206 <p>For Kerberos-secured cluster, you need to get a keytab file and the principal from your admin, and configure “eagle.keytab.file” and “eagle.kerberos.principal” to authenticate its access.</p>
    205 207
    206 <pre><code> classification.eagle.keytab.file=/EAGLE-HOME/.keytab/eagle.keytab
    208 <div class="highlighter-rouge"><pre class="highlight"><code> classification.eagle.keytab.file=/EAGLE-HOME/.keytab/eagle.keytab
    207 209 classification.eagle.kerberos.principal=eagle@SOMEWHERE.COM
    208 210 </code></pre>
    211 </div>
    209 212
    210 213 <p>If there is an exception about “invalid server principal name”, you may need to check the DNS resolver, or the data transfer , such as “dfs.encrypt.data.transfer”, “dfs.encrypt.data.transfer.algorithm”, “dfs.trustedchannel.resolver.class”, “dfs.datatransfer.client.encrypt”.</p>
    211 214 </li>
     
    216 219 <li>
    217 220 <p>Basic</p>
    218 221
    219 <pre><code> classification.accessType=metastoredb_jdbc
    222 <div class="highlighter-rouge"><pre class="highlight"><code> classification.accessType=metastoredb_jdbc
    220 223 classification.password=hive
    221 224 classification.user=hive
    222 225 classification.jdbcDriverClassName=com.mysql.jdbc.Driver
    223 226 classification.jdbcUrl=jdbc:mysql://sandbox.hortonworks.com/hive?createDatabaseIfNotExist=true
    224 227 </code></pre>
    228 </div>
    225 229 </li>
    226 230 </ul>
    227 231 </li>
     
    234 238
    235 239 <p>You need to sett “hbase.zookeeper.quorum”:”localhost” property and “hbase.zookeeper.property.clientPort” property.</p>
    236 240
    237 <pre><code> classification.hbase.zookeeper.property.clientPort=2181
    241 <div class="highlighter-rouge"><pre class="highlight"><code> classification.hbase.zookeeper.property.clientPort=2181
    238 242 classification.hbase.zookeeper.quorum=localhost
    239 243 </code></pre>
    244 </div>
    240 245 </li>
    241 246 <li>
    242 247 <p>Kerberos-secured cluster</p>
    243 248
    244 249 <p>According to your environment, you can add or remove some of the following properties. Here is the reference.</p>
    245 250
    246 <pre><code> classification.hbase.zookeeper.property.clientPort=2181
    251 <div class="highlighter-rouge"><pre class="highlight"><code> classification.hbase.zookeeper.property.clientPort=2181
    247 252 classification.hbase.zookeeper.quorum=localhost
    248 253 classification.hbase.security.authentication=kerberos
    249 254 classification.hbase.master.kerberos.principal=hadoop/_HOST@EXAMPLE.COM
     
    251 256 classification.eagle.keytab.file=/EAGLE-HOME/.keytab/eagle.keytab
    252 257 classification.eagle.kerberos.principal=eagle@EXAMPLE.COM
    253 258 </code></pre>
    259 </div>
    254 260 </li>
    255 261 </ul>
    256 262 </li>
     
    321 327 <div class="footnotes">
    322 328 <ol>
    323 329 <li id="fn:HIVE">
    324 <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    330 <p><em>All mentions of “hive” on this page represent Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    325 331 </li>
    326 332 <li id="fn:HBASE">
    327 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    333 <p><em>All mentions of “hbase” on this page represent Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    328 334 </li>
    329 335 </ol>
    330 336 </div>
  • eagle/site/docs/tutorial/ldap.html

     
    160 160
    161 161 <p>Step 1: edit configuration under conf/ldap.properties.</p>
    162 162
    163 <pre><code>ldap.server=ldap://localhost:10389
    163 <div class="highlighter-rouge"><pre class="highlight"><code>ldap.server=ldap://localhost:10389
    164 164 ldap.username=uid=admin,ou=system
    165 165 ldap.password=secret
    166 166 ldap.user.searchBase=ou=Users,o=mojo
     
    169 169 acl.adminRole=
    170 170 acl.defaultRole=ROLE_USER
    171 171 </code></pre>
    172 </div>
    172 173
    173 174 <p>acl.adminRole and acl.defaultRole are two customized properties for Eagle. Eagle manages admin users with groups. If you set acl.adminRole as ROLE_{EAGLE-ADMIN-GROUP-NAME}, members in this group have the admin privilege. acl.defaultRole is ROLE_USER.</p>
    174 175
    175 176 <p>Step 2: edit conf/eagle-service.conf, and add springActiveProfile=”default”</p>
    176 177
    177 <pre><code>eagle{
    178 <div class="highlighter-rouge"><pre class="highlight"><code>eagle{
    178 179 service{
    179 180 storage-type="hbase"
    180 181 hbase-zookeeper-quorum="localhost"
     
    184 185 }
    185 186 }
    186 187 </code></pre>
    188 </div>
    187 189
    188 190
    189 191 </div><!--end of loadcontent-->
  • eagle/site/docs/tutorial/notificationplugin.html

     
    183 183 </li>
    184 184 </ul>
    185 185
    186 <p><img src="/images/notificationPlugin.png" alt="notificationPlugin" />
    187 ### Customized Notification Plugin</p>
    186 <p><img src="/images/notificationPlugin.png" alt="notificationPlugin" /></p>
    187 <h3 id="customized-notification-plugin">Customized Notification Plugin</h3>
    188 188
    189 189 <p>To integrate a customized notification plugin, we must implement an interface</p>
    190 190
    191 <pre><code>public interface NotificationPlugin {
    191 <div class="highlighter-rouge"><pre class="highlight"><code>public interface NotificationPlugin {
    192 192 /**
    193 193 * for initialization
    194 194 * @throws Exception
     
    218 218 List&lt;NotificationStatus&gt; getStatusList();
    219 219 } Examples: AlertKafkaPlugin, AlertEmailPlugin, and AlertEagleStorePlugin.
    220 220 </code></pre>
    221 </div>
    221 222
    222 223 <p>The second and crucial step is to register the configurations of the customized plugin. In other words, we need persist the configuration template into database in order to expose the configurations to users in the front end.</p>
    223 224
    224 225 <p>Examples:</p>
    225 226
    226 <pre><code>{
    227 "prefix": "alertNotifications",
    228 "tags": {
    229 "notificationType": "kafka"
    230 },
    231 "className": "org.apache.eagle.notification.plugin.AlertKafkaPlugin",
    232 "description": "send alert to kafka bus",
    233 "enabled":true,
    234 "fields": "[{\"name\":\"kafka_broker\",\"value\":\"sandbox.hortonworks.com:6667\"},{\"name\":\"topic\"}]"
    235 }
    236 </code></pre>
    227 <div class="highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
    228 </span><span class="nt">"prefix"</span><span class="p">:</span><span class="w"> </span><span class="s2">"alertNotifications"</span><span class="p">,</span><span class="w">
    229 </span><span class="nt">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    230 </span><span class="nt">"notificationType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kafka"</span><span class="w">
    231 </span><span class="p">},</span><span class="w">
    232 </span><span class="nt">"className"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.apache.eagle.notification.plugin.AlertKafkaPlugin"</span><span class="p">,</span><span class="w">
    233 </span><span class="nt">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"send alert to kafka bus"</span><span class="p">,</span><span class="w">
    234 </span><span class="nt">"enabled"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="w">
    235 </span><span class="nt">"fields"</span><span class="p">:</span><span class="w"> </span><span class="s2">"[{\"name\":\"kafka_broker\",\"value\":\"sandbox.hortonworks.com:6667\"},{\"name\":\"topic\"}]"</span><span class="w">
    236 </span><span class="p">}</span><span class="w">
    237 </span></code></pre>
    238 </div>
    237 239
    238 <p><strong>Note</strong>: <code>fields</code> is the configuration for notification type <code>kafka</code></p>
    240 <p><strong>Note</strong>: <code class="highlighter-rouge">fields</code> is the configuration for notification type <code class="highlighter-rouge">kafka</code></p>
    239 241
    240 242 <p>How can we do that? <a href="https://github.com/apache/eagle/blob/master/eagle-assembly/src/main/bin/eagle-topology-init.sh">Here</a> are Eagle other notification plugin configurations. Just append yours to it, and run this script when Eagle service is up.</p>
    241 243
     
    246 248 <div class="footnotes">
    247 249 <ol>
    248 250 <li id="fn:KAFKA">
    249 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em> <a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    251 <p><em>All mentions of “kafka” on this page represent Apache Kafka.</em>&nbsp;<a href="#fnref:KAFKA" class="reversefootnote">&#8617;</a></p>
    250 252 </li>
    251 253 </ol>
    252 254 </div>
  • eagle/site/docs/tutorial/policy.html

     
    183 183 <li>
    184 184 <p><strong>Step 2</strong>: Eagle supports a variety of properties for match critera where users can set different values. Eagle also supports window functions to extend policies with time functions.</p>
    185 185
    186 <pre><code>command = delete
    186 <div class="highlighter-rouge"><pre class="highlight"><code>command = delete
    187 187 (Eagle currently supports the following commands open, delete, copy, append, copy from local, get, move, mkdir, create, list, change permissions)
    188 188
    189 189 source = /tmp/private
    190 190 (Eagle supports wildcarding for property values for example /tmp/*)
    191 191 </code></pre>
    192 </div>
    192 193
    193 194 <p><img src="/images/docs/hdfs-policy2.png" alt="HDFS Policies" /></p>
    194 195 </li>
     
    215 216 <li>
    216 217 <p><strong>Step 2</strong>: Eagle support a variety of properties for match critera where users can set different values. Eagle also supports window functions to extend policies with time functions.</p>
    217 218
    218 <pre><code>command = Select
    219 <div class="highlighter-rouge"><pre class="highlight"><code>command = Select
    219 220 (Eagle currently supports the following commands DDL statements Create, Drop, Alter, Truncate, Show)
    220 221
    221 222 sensitivity type = PHONE_NUMBER
    222 223 (Eagle supports classifying data in Hive with different sensitivity types. Users can use these sensitivity types to create policies)
    223 224 </code></pre>
    225 </div>
    224 226
    225 227 <p><img src="/images/docs/hive-policy2.png" alt="Hive Policies" /></p>
    226 228 </li>
     
    238 240 <div class="footnotes">
    239 241 <ol>
    240 242 <li id="fn:HIVE">
    241 <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    243 <p><em>All mentions of “hive” on this page represent Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    242 244 </li>
    243 245 </ol>
    244 246 </div>
  • eagle/site/docs/tutorial/site-0.3.0.html

     
    180 180
    181 181 <p>You may configure the default path for Hadoop clients to connect remote hdfs namenode.</p>
    182 182
    183 <pre><code> {"fs.defaultFS":"hdfs://sandbox.hortonworks.com:8020"}
    184 </code></pre>
    183 <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="nt">"fs.defaultFS"</span><span class="p">:</span><span class="s2">"hdfs://sandbox.hortonworks.com:8020"</span><span class="p">}</span><span class="w">
    184 </span></code></pre>
    185 </div>
    185 186 </li>
    186 187 <li>
    187 188 <p>HA case</p>
    188 189
    189 190 <p>Basically, you point your fs.defaultFS at your nameservice and let the client know how its configured (the backing namenodes) and how to fail over between them under the HA mode</p>
    190 191
    191 <pre><code> {"fs.defaultFS":"hdfs://nameservice1",
    192 "dfs.nameservices": "nameservice1",
    193 "dfs.ha.namenodes.nameservice1":"namenode1,namenode2",
    194 "dfs.namenode.rpc-address.nameservice1.namenode1": "hadoopnamenode01:8020",
    195 "dfs.namenode.rpc-address.nameservice1.namenode2": "hadoopnamenode02:8020",
    196 "dfs.client.failover.proxy.provider.nameservice1": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
    197 }
    198 </code></pre>
    192 <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="nt">"fs.defaultFS"</span><span class="p">:</span><span class="s2">"hdfs://nameservice1"</span><span class="p">,</span><span class="w">
    193 </span><span class="nt">"dfs.nameservices"</span><span class="p">:</span><span class="w"> </span><span class="s2">"nameservice1"</span><span class="p">,</span><span class="w">
    194 </span><span class="nt">"dfs.ha.namenodes.nameservice1"</span><span class="p">:</span><span class="s2">"namenode1,namenode2"</span><span class="p">,</span><span class="w">
    195 </span><span class="nt">"dfs.namenode.rpc-address.nameservice1.namenode1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hadoopnamenode01:8020"</span><span class="p">,</span><span class="w">
    196 </span><span class="nt">"dfs.namenode.rpc-address.nameservice1.namenode2"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hadoopnamenode02:8020"</span><span class="p">,</span><span class="w">
    197 </span><span class="nt">"dfs.client.failover.proxy.provider.nameservice1"</span><span class="p">:</span><span class="w"> </span><span class="s2">"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"</span><span class="w">
    198 </span><span class="p">}</span><span class="w">
    199 </span></code></pre>
    200 </div>
    199 201 </li>
    200 202 <li>
    201 203 <p>Kerberos-secured cluster</p>
    202 204
    203 205 <p>For Kerberos-secured cluster, you need to get a keytab file and the principal from your admin, and configure “eagle.keytab.file” and “eagle.kerberos.principal” to authenticate its access.</p>
    204 206
    205 <pre><code> { "eagle.keytab.file":"/EAGLE-HOME/.keytab/eagle.keytab",
    206 "eagle.kerberos.principal":"eagle@SOMEWHERE.COM"
    207 }
    208 </code></pre>
    207 <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"eagle.keytab.file"</span><span class="p">:</span><span class="s2">"/EAGLE-HOME/.keytab/eagle.keytab"</span><span class="p">,</span><span class="w">
    208 </span><span class="nt">"eagle.kerberos.principal"</span><span class="p">:</span><span class="s2">"eagle@SOMEWHERE.COM"</span><span class="w">
    209 </span><span class="p">}</span><span class="w">
    210 </span></code></pre>
    211 </div>
    209 212
    210 213 <p>If there is an exception about “invalid server principal name”, you may need to check the DNS resolver, or the data transfer , such as “dfs.encrypt.data.transfer”, “dfs.encrypt.data.transfer.algorithm”, “dfs.trustedchannel.resolver.class”, “dfs.datatransfer.client.encrypt”.</p>
    211 214 </li>
     
    216 219 <li>
    217 220 <p>Basic</p>
    218 221
    219 <pre><code> {
    220 "accessType": "metastoredb_jdbc",
    221 "password": "hive",
    222 "user": "hive",
    223 "jdbcDriverClassName": "com.mysql.jdbc.Driver",
    224 "jdbcUrl": "jdbc:mysql://sandbox.hortonworks.com/hive?createDatabaseIfNotExist=true"
    225 }
    226 </code></pre>
    222 <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
    223 </span><span class="nt">"accessType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"metastoredb_jdbc"</span><span class="p">,</span><span class="w">
    224 </span><span class="nt">"password"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hive"</span><span class="p">,</span><span class="w">
    225 </span><span class="nt">"user"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hive"</span><span class="p">,</span><span class="w">
    226 </span><span class="nt">"jdbcDriverClassName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"com.mysql.jdbc.Driver"</span><span class="p">,</span><span class="w">
    227 </span><span class="nt">"jdbcUrl"</span><span class="p">:</span><span class="w"> </span><span class="s2">"jdbc:mysql://sandbox.hortonworks.com/hive?createDatabaseIfNotExist=true"</span><span class="w">
    228 </span><span class="p">}</span><span class="w">
    229 </span></code></pre>
    230 </div>
    227 231 </li>
    228 232 </ul>
    229 233 </li>
     
    236 240
    237 241 <p>You need to sett “hbase.zookeeper.quorum”:”localhost” property and “hbase.zookeeper.property.clientPort” property.</p>
    238 242
    239 <pre><code> {
    240 "hbase.zookeeper.property.clientPort":"2181",
    241 "hbase.zookeeper.quorum":"localhost"
    242 }
    243 </code></pre>
    243 <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
    244 </span><span class="nt">"hbase.zookeeper.property.clientPort"</span><span class="p">:</span><span class="s2">"2181"</span><span class="p">,</span><span class="w">
    245 </span><span class="nt">"hbase.zookeeper.quorum"</span><span class="p">:</span><span class="s2">"localhost"</span><span class="w">
    246 </span><span class="p">}</span><span class="w">
    247 </span></code></pre>
    248 </div>
    244 249 </li>
    245 250 <li>
    246 251 <p>Kerberos-secured cluster</p>
    247 252
    248 253 <p>According to your environment, you can add or remove some of the following properties. Here is the reference.</p>
    249 254
    250 <pre><code> {
    251 "hbase.zookeeper.property.clientPort":"2181",
    252 "hbase.zookeeper.quorum":"localhost",
    253 "hbase.security.authentication":"kerberos",
    254 "hbase.master.kerberos.principal":"hadoop/_HOST@EXAMPLE.COM",
    255 "zookeeper.znode.parent":"/hbase",
    256 "eagle.keytab.file":"/EAGLE-HOME/.keytab/eagle.keytab",
    257 "eagle.kerberos.principal":"eagle@EXAMPLE.COM"
    258 }
    259 </code></pre>
    255 <div class="highlighter-rouge"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w">
    256 </span><span class="nt">"hbase.zookeeper.property.clientPort"</span><span class="p">:</span><span class="s2">"2181"</span><span class="p">,</span><span class="w">
    257 </span><span class="nt">"hbase.zookeeper.quorum"</span><span class="p">:</span><span class="s2">"localhost"</span><span class="p">,</span><span class="w">
    258 </span><span class="nt">"hbase.security.authentication"</span><span class="p">:</span><span class="s2">"kerberos"</span><span class="p">,</span><span class="w">
    259 </span><span class="nt">"hbase.master.kerberos.principal"</span><span class="p">:</span><span class="s2">"hadoop/_HOST@EXAMPLE.COM"</span><span class="p">,</span><span class="w">
    260 </span><span class="nt">"zookeeper.znode.parent"</span><span class="p">:</span><span class="s2">"/hbase"</span><span class="p">,</span><span class="w">
    261 </span><span class="nt">"eagle.keytab.file"</span><span class="p">:</span><span class="s2">"/EAGLE-HOME/.keytab/eagle.keytab"</span><span class="p">,</span><span class="w">
    262 </span><span class="nt">"eagle.kerberos.principal"</span><span class="p">:</span><span class="s2">"eagle@EXAMPLE.COM"</span><span class="w">
    263 </span><span class="p">}</span><span class="w">
    264 </span></code></pre>
    265 </div>
    260 266 </li>
    261 267 </ul>
    262 268 </li>
     
    274 280 <div class="footnotes">
    275 281 <ol>
    276 282 <li id="fn:HADOOP">
    277 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    283 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    278 284 </li>
    279 285 <li id="fn:HIVE">
    280 <p><em>Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    286 <p><em>Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    281 287 </li>
    282 288 <li id="fn:HBASE">
    283 <p><em>Apache HBase.</em> <a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    289 <p><em>Apache HBase.</em>&nbsp;<a href="#fnref:HBASE" class="reversefootnote">&#8617;</a></p>
    284 290 </li>
    285 291 </ol>
    286 292 </div>
  • eagle/site/docs/tutorial/topologymanagement.html

     
    168 168 <p>Application manager consists of a daemon scheduler and an execution module. The scheduler periodically loads user operations(start/stop) from database, and the execution module executes these operations. For more details, please refer to <a href="https://cwiki.apache.org/confluence/display/EAG/Application+Management">here</a>.</p>
    169 169
    170 170 <h3 id="configurations">Configurations</h3>
    171 <p>The configuration file <code>eagle-scheduler.conf</code> defines scheduler parameters, execution platform settings and parts of default topology configuration.</p>
    171 <p>The configuration file <code class="highlighter-rouge">eagle-scheduler.conf</code> defines scheduler parameters, execution platform settings and parts of default topology configuration.</p>
    172 172
    173 173 <ul>
    174 174 <li>
     
    262 262 <li>
    263 263 <p>Editing eagle-scheduler.conf, and start Eagle service</p>
    264 264
    265 <pre><code> # enable application manager
    265 <div class="highlighter-rouge"><pre class="highlight"><code> # enable application manager
    266 266 appCommandLoaderEnabled = true
    267 267
    268 268 # provide jar path
     
    272 272 envContextConfig.url = "http://sandbox.hortonworks.com:8744"
    273 273 envContextConfig.nimbusHost = "sandbox.hortonworks.com"
    274 274 </code></pre>
    275 </div>
    275 276
    276 277 <p>For more configurations, please back to <a href="/docs/configuration.html">Application Configuration</a>. <br />
    277 After the configuration is ready, start Eagle service <code>bin/eagle-service.sh start</code>.</p>
    278 After the configuration is ready, start Eagle service <code class="highlighter-rouge">bin/eagle-service.sh start</code>.</p>
    278 279 </li>
    279 280 <li>
    280 281 <p>Go to admin page
     
    296 297 <li>
    297 298 <p>Go to site page, and add topology configurations.</p>
    298 299
    299 <p><strong>NOTICE</strong> topology configurations defined here are REQUIRED an extra prefix <code>.app</code></p>
    300 <p><strong>NOTICE</strong> topology configurations defined here are REQUIRED an extra prefix <code class="highlighter-rouge">.app</code></p>
    300 301
    301 302 <p>Blow are some example configurations for [site=sandbox, applicatoin=hbaseSecurityLog].</p>
    302 303
    303 <pre><code> classification.hbase.zookeeper.property.clientPort=2181
    304 <div class="highlighter-rouge"><pre class="highlight"><code> classification.hbase.zookeeper.property.clientPort=2181
    304 305 classification.hbase.zookeeper.quorum=sandbox.hortonworks.com
    305 306
    306 307 app.envContextConfig.env=storm
     
    329 330 app.eagleProps.eagleService.username=admin
    330 331 app.eagleProps.eagleService.password=secret
    331 332 </code></pre>
    333 </div>
    332 334
    333 335 <p><img src="/images/appManager/topology-configuration-1.png" alt="topology-configuration-1" />
    334 336 <img src="/images/appManager/topology-configuration-2.png" alt="topology-configuration-2" /></p>
     
    351 353 <div class="footnotes">
    352 354 <ol>
    353 355 <li id="fn:STORM">
    354 <p><em>All mentions of “storm” on this page represent Apache Storm.</em> <a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    356 <p><em>All mentions of “storm” on this page represent Apache Storm.</em>&nbsp;<a href="#fnref:STORM" class="reversefootnote">&#8617;</a></p>
    355 357 </li>
    356 358 </ol>
    357 359 </div>
  • eagle/site/docs/tutorial/userprofile.html

     
    173 173 <li>
    174 174 <p>Option 1: command line</p>
    175 175
    176 <pre><code>$ cd &lt;eagle-home&gt;/bin
    176 <div class="highlighter-rouge"><pre class="highlight"><code>$ cd &lt;eagle-home&gt;/bin
    177 177 $ bin/eagle-userprofile-scheduler.sh --site sandbox start
    178 178 </code></pre>
    179 </div>
    179 180 </li>
    180 181 <li>
    181 182 <p>Option 2: start via Apache Ambari
     
    203 204
    204 205 <p>submit userProfiles topology if it’s not on <a href="http://sandbox.hortonworks.com:8744">topology UI</a></p>
    205 206
    206 <pre><code>$ bin/eagle-topology.sh --main org.apache.eagle.security.userprofile.UserProfileDetectionMain --config conf/sandbox-userprofile-topology.conf start
    207 <div class="highlighter-rouge"><pre class="highlight"><code>$ bin/eagle-topology.sh --main org.apache.eagle.security.userprofile.UserProfileDetectionMain --config conf/sandbox-userprofile-topology.conf start
    207 208 </code></pre>
    209 </div>
    208 210 </li>
    209 211 <li>
    210 212 <p><strong>Option 2</strong>: Apache Ambari</p>
     
    219 221 <li>Prepare sample data for ML training and validation sample data
    220 222 <ul>
    221 223 <li>a. Download following sample data to be used for training</li>
    222 <li><a href="/data/user1.hdfs-audit.2015-10-11-00.txt"><code>user1.hdfs-audit.2015-10-11-00.txt</code></a></li>
    223 <li><a href="/data/user1.hdfs-audit.2015-10-11-01.txt"><code>user1.hdfs-audit.2015-10-11-01.txt</code></a></li>
    224 <li>b. Downlaod <a href="/data/userprofile-validate.txt"><code>userprofile-validate.txt</code></a>file which contains data points that you can try to test the models</li>
    225 224 </ul>
    225 <ul>
    226 <li><a href="/data/user1.hdfs-audit.2015-10-11-00.txt"><code class="highlighter-rouge">user1.hdfs-audit.2015-10-11-00.txt</code></a></li>
    227 <li><a href="/data/user1.hdfs-audit.2015-10-11-01.txt"><code class="highlighter-rouge">user1.hdfs-audit.2015-10-11-01.txt</code></a>
    228 * b. Downlaod <a href="/data/userprofile-validate.txt"><code class="highlighter-rouge">userprofile-validate.txt</code></a>file which contains data points that you can try to test the models</li>
    229 </ul>
    226 230 </li>
    227 231 <li>Copy the files (downloaded in the previous step) into a location in sandbox
    228 For example: <code>/usr/hdp/current/eagle/lib/userprofile/data/</code></li>
    229 <li>Modify <code>&lt;Eagle-home&gt;/conf/sandbox-userprofile-scheduler.conf </code>
    230 update <code>training-audit-path</code> to set to the path for training data sample (the path you used for Step 1.a)
    232 For example: <code class="highlighter-rouge">/usr/hdp/current/eagle/lib/userprofile/data/</code></li>
    233 <li>Modify <code class="highlighter-rouge">&lt;Eagle-home&gt;/conf/sandbox-userprofile-scheduler.conf </code>
    234 update <code class="highlighter-rouge">training-audit-path</code> to set to the path for training data sample (the path you used for Step 1.a)
    231 235 update detection-audit-path to set to the path for validation (the path you used for Step 1.b)</li>
    232 236 <li>Run ML training program from eagle UI</li>
    233 237 <li>
    234 238 <p>Produce Apache Kafka data using the contents from validate file (Step 1.b)
    235 Run the command (assuming the eagle configuration uses Kafka topic <code>sandbox_hdfs_audit_log</code>)</p>
    239 Run the command (assuming the eagle configuration uses Kafka topic <code class="highlighter-rouge">sandbox_hdfs_audit_log</code>)</p>
    236 240
    237 <pre><code> ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic sandbox_hdfs_audit_log
    241 <div class="highlighter-rouge"><pre class="highlight"><code> ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic sandbox_hdfs_audit_log
    238 242 </code></pre>
    243 </div>
    239 244 </li>
    240 245 <li>Paste few lines of data from file validate onto kafka-console-producer
    241 246 Check <a href="http://localhost:9099/eagle-service/#/dam/alertList">http://localhost:9099/eagle-service/#/dam/alertList</a> for generated alerts</li>
  • eagle/site/docs/usecases.html

     
    218 218 <div class="footnotes">
    219 219 <ol>
    220 220 <li id="fn:HADOOP">
    221 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em> <a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    221 <p><em>All mentions of “hadoop” on this page represent Apache Hadoop.</em>&nbsp;<a href="#fnref:HADOOP" class="reversefootnote">&#8617;</a></p>
    222 222 </li>
    223 223 <li id="fn:HIVE">
    224 <p><em>All mentions of “hive” on this page represent Apache Hive.</em> <a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    224 <p><em>All mentions of “hive” on this page represent Apache Hive.</em>&nbsp;<a href="#fnref:HIVE" class="reversefootnote">&#8617;</a></p>
    225 225 </li>
    226 226 <li id="fn:SPARK">
    227 <p><em>All mentions of “spark” on this page represent Apache Spark.</em> <a href="#fnref:SPARK" class="reversefootnote">&#8617;</a></p>
    227 <p><em>All mentions of “spark” on this page represent Apache Spark.</em>&nbsp;<a href="#fnref:SPARK" class="reversefootnote">&#8617;</a></p>
    228 228 </li>
    229 229 <li id="fn:CASSANDRA">
    230 <p><em>Apache Cassandra.</em> <a href="#fnref:CASSANDRA" class="reversefootnote">&#8617;</a></p>
    230 <p><em>Apache Cassandra.</em>&nbsp;<a href="#fnref:CASSANDRA" class="reversefootnote">&#8617;</a></p>
    231 231 </li>
    232 232 </ol>
    233 233 </div>
  • eagle/site/feed.xml

     
    5 5 <description>Eagle - Analyze Big Data Platforms for Security and Performance</description>
    6 6 <link>http://goeagle.io/</link>
    7 7 <atom:link href="http://goeagle.io/feed.xml" rel="self" type="application/rss+xml"/>
    8 <pubDate>Mon, 03 Apr 2017 19:17:59 +0800</pubDate>
    9 <lastBuildDate>Mon, 03 Apr 2017 19:17:59 +0800</lastBuildDate>
    10 <generator>Jekyll v2.5.3</generator>
    8 <pubDate>Wed, 22 Nov 2017 13:52:35 +0800</pubDate>
    9 <lastBuildDate>Wed, 22 Nov 2017 13:52:35 +0800</lastBuildDate>
    10 <generator>Jekyll v3.4.3</generator>
    11 11
    12 12 <item>
    13 13 <title>Apache Eagle 正式发布:分布式实时Hadoop数据安全方案</title>
     
    17 17
    18 18 &lt;p&gt;日前,eBay公司隆重宣布正式向开源业界推出分布式实时安全监控方案 - Apache Eagle (http://goeagle.io),该项目已于2015年10月26日正式加入Apache 成为孵化器项目。Apache Eagle提供一套高效分布式的流式策略引擎,具有高实时、可伸缩、易扩展、交互友好等特点,同时集成机器学习对用户行为建立Profile以实现智能实时地保护Hadoop生态系统中大数据的安全。&lt;/p&gt;
    19 19
    20 &lt;h2 id=&quot;section&quot;&gt;背景&lt;/h2&gt;
    20 &lt;h2 id=&quot;背景&quot;&gt;背景&lt;/h2&gt;
    21 21 &lt;p&gt;随着大数据的发展,越来越多的成功企业或者组织开始采取数据驱动商业的运作模式。在eBay,我们拥有数万名工程师、分析师和数据科学家,他们每天访问分析数PB级的数据,以为我们的用户带来无与伦比的体验。在全球业务中,我们也广泛地利用海量大数据来连接我们数以亿计的用户。&lt;/p&gt;
    22 22
    23 23 &lt;p&gt;近年来,Hadoop已经逐渐成为大数据分析领域最受欢迎的解决方案,eBay也一直在使用Hadoop技术从数据中挖掘价值,例如,我们通过大数据提高用户的搜索体验,识别和优化精准广告投放,充实我们的产品目录,以及通过点击流分析以理解用户如何使用我们的在线市场平台等。&lt;/p&gt;
     
    54 54 &lt;li&gt;&lt;strong&gt;开源&lt;/strong&gt;:Eagle一直根据开源的标准开发,并构建于诸多大数据领域的开源产品之上,因此我们决定以Apache许可证开源Eagle,以回馈社区,同时也期待获得社区的反馈、协作与支持。&lt;/li&gt;
    55 55 &lt;/ul&gt;
    56 56
    57 &lt;h2 id=&quot;eagle&quot;&gt;Eagle概览&lt;/h2&gt;
    57 &lt;h2 id=&quot;eagle概览&quot;&gt;Eagle概览&lt;/h2&gt;
    58 58
    59 59 &lt;p&gt;&lt;img src=&quot;/images/posts/eagle-group.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
    60 60
    61 &lt;h4 id=&quot;data-collection-and-storage&quot;&gt;数据流接入和存储(Data Collection and Storage)&lt;/h4&gt;
    61 &lt;h4 id=&quot;数据流接入和存储data-collection-and-storage&quot;&gt;数据流接入和存储(Data Collection and Storage)&lt;/h4&gt;
    62 62 &lt;p&gt;Eagle提供高度可扩展的编程API,可以支持将任何类型的数据源集成到Eagle的策略执行引擎中。例如,在Eagle HDFS 审计事件(Audit)监控模块中,通过Kafka来实时接收来自Namenode Log4j Appender 或者 Logstash Agent 收集的数据;在Eagle Hive 监控模块中,通过YARN API 收集正在运行Job的Hive 查询日志,并保证比较高的可伸缩性和容错性。&lt;/p&gt;
    63 63
    64 &lt;h4 id=&quot;data-processing&quot;&gt;数据实时处理(Data Processing)&lt;/h4&gt;
    64 &lt;h4 id=&quot;数据实时处理data-processing&quot;&gt;数据实时处理(Data Processing)&lt;/h4&gt;
    65 65
    66 66 &lt;p&gt;&lt;strong&gt;流处理API(Stream Processing API)Eagle&lt;/strong&gt; 提供独立于物理平台而高度抽象的流处理API,目前默认支持Apache Storm,但是也允许扩展到其他任意流处理引擎,比如Flink 或者 Samza等。该层抽象允许开发者在定义监控数据处理逻辑时,无需在物理执行层绑定任何特定流处理平台,而只需通过复用、拼接和组装例如数据转换、过滤、外部数据Join等组件,以实现满足需求的DAG(有向无环图),同时,开发者也可以很容易地以编程地方式将业务逻辑流程和Eagle 策略引擎框架集成起来。Eagle框架内部会将描述业务逻辑的DAG编译成底层流处理架构的原生应用,例如Apache Storm Topology 等,从事实现平台的独立。&lt;/p&gt;
    67 67
    68 68 &lt;p&gt;&lt;strong&gt;以下是一个Eagle如何处理事件和告警的示例:&lt;/strong&gt;&lt;/p&gt;
    69 69
    70 &lt;pre&gt;&lt;code&gt;StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); // storm env
    70 &lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); // storm env
    71 71 StreamProducer producer = env.newSource(new KafkaSourcedSpoutProvider().getSpout(config)).renameOutputFields(1) // declare kafka source
    72 72 .flatMap(new AuditLogTransformer()) // transform event
    73 73 .groupBy(Arrays.asList(0)) // group by 1st field
     
    75 75 .alertWithConsumer(“userActivity“,”userProfileExecutor“) // ML policy evaluation
    76 76 env.execute(); // execute stream processing and alert
    77 77 &lt;/code&gt;&lt;/pre&gt;
    78 &lt;/div&gt;
    78 79
    79 80 &lt;p&gt;&lt;strong&gt;告警框架(Alerting Framework)Eagle&lt;/strong&gt;告警框架由流元数据API、策略引擎服务提供API、策略Partitioner API 以及预警去重框架等组成:&lt;/p&gt;
    80 81
     
    84 85 &lt;li&gt;
    85 86 &lt;p&gt;&lt;strong&gt;扩展性&lt;/strong&gt; Eagle的策略引擎服务提供API允许你插入新的策略引擎&lt;/p&gt;
    86 87
    87 &lt;pre&gt;&lt;code&gt; public interface PolicyEvaluatorServiceProvider {
    88 &lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt; public interface PolicyEvaluatorServiceProvider {
    88 89 public String getPolicyType(); // literal string to identify one type of policy
    89 90 public Class&amp;lt;? extends PolicyEvaluator&amp;gt; getPolicyEvaluator(); // get policy evaluator implementation
    90 91 public List&amp;lt;Module&amp;gt; getBindingModules(); // policy text with json format to object mapping
     
    95 96 public void onPolicyDelete(); // invoked when policy is deleted
    96 97 }
    97 98 &lt;/code&gt;&lt;/pre&gt;
    99 &lt;/div&gt;
    98 100 &lt;/li&gt;
    99 101 &lt;li&gt;&lt;strong&gt;策略Partitioner API&lt;/strong&gt; 允许策略在不同的物理节点上并行执行。也允许你自定义策略Partitioner类。这些功能使得策略和事件完全以分布式的方式执行。&lt;/li&gt;
    100 102 &lt;li&gt;
    101 103 &lt;p&gt;&lt;strong&gt;可伸缩性&lt;/strong&gt; Eagle 通过支持策略的分区接口来实现大量的策略可伸缩并发地运行&lt;/p&gt;
    102 104
    103 &lt;pre&gt;&lt;code&gt; public interface PolicyPartitioner extends Serializable {
    105 &lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt; public interface PolicyPartitioner extends Serializable {
    104 106 int partition(int numTotalPartitions, String policyType, String policyId); // method to distribute policies
    105 107 }
    106 108 &lt;/code&gt;&lt;/pre&gt;
    109 &lt;/div&gt;
    107 110
    108 111 &lt;p&gt;&lt;img src=&quot;/images/posts/policy-partition.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
    109 112
     
    160 163 &lt;li&gt;
    161 164 &lt;p&gt;单一事件执行策略(用户访问Hive中的敏感数据列)&lt;/p&gt;
    162 165
    163 &lt;pre&gt;&lt;code&gt; from hiveAccessLogStream[sensitivityType==&#39;PHONE_NUMBER&#39;] select * insert into outputStream;
    166 &lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt; from hiveAccessLogStream[sensitivityType=='PHONE_NUMBER'] select * insert into outputStream;
    164 167 &lt;/code&gt;&lt;/pre&gt;
    168 &lt;/div&gt;
    165 169 &lt;/li&gt;
    166 170 &lt;li&gt;
    167 171 &lt;p&gt;基于窗口的策略(用户在10分钟内访问目录 /tmp/private 多余 5次)&lt;/p&gt;
    168 172
    169 &lt;pre&gt;&lt;code&gt; hdfsAuditLogEventStream[(src == &#39;/tmp/private&#39;)]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue &amp;gt;= 5 insert into outputStream;
    173 &lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt; hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue &amp;gt;= 5 insert into outputStream;
    170 174 &lt;/code&gt;&lt;/pre&gt;
    175 &lt;/div&gt;
    171 176 &lt;/li&gt;
    172 177 &lt;/ul&gt;
    173 178
    174 179 &lt;p&gt;&lt;strong&gt;查询服务(Query Service)&lt;/strong&gt; Eagle 提供类SQL的REST API用来实现针对海量数据集的综合计算、查询和分析的能力,支持例如过滤、聚合、直方运算、排序、top、算术表达式以及分页等。Eagle优先支持HBase 作为其默认数据存储,但是同时也支持基JDBC的关系型数据库。特别是当选择以HBase作为存储时,Eagle便原生拥有了HBase存储和查询海量监控数据的能力,Eagle 查询框架会将用户提供的类SQL查询语法最终编译成为HBase 原生的Filter 对象,并支持通过HBase Coprocessor进一步提升响应速度。&lt;/p&gt;
    175 180
    176 &lt;pre&gt;&lt;code&gt;query=AlertDefinitionService[@dataSource=&quot;hiveQueryLog&quot;]{@policyDef}&amp;amp;pageSize=100000
    181 &lt;div class=&quot;highlighter-rouge&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;query=AlertDefinitionService[@dataSource=&quot;hiveQueryLog&quot;]{@policyDef}&amp;amp;pageSize=100000
    177 182 &lt;/code&gt;&lt;/pre&gt;
    183 &lt;/div&gt;
    178 184
    179 &lt;h2 id=&quot;eagleebay&quot;&gt;Eagle在eBay的使用场景&lt;/h2&gt;
    185 &lt;h2 id=&quot;eagle在ebay的使用场景&quot;&gt;Eagle在eBay的使用场景&lt;/h2&gt;
    180 186 &lt;p&gt;目前,Eagle的数据行为监控系统已经部署到一个拥有2500多个节点的Hadoop集群之上,用以保护数百PB数据的安全,并正计划于今年年底之前扩展到其他上十个Hadoop集群上,从而覆盖eBay 所有主要Hadoop的10000多台节点。在我们的生产环境中,我们已针对HDFS、Hive 等集群中的数据配置了一些基础的安全策略,并将于年底之前不断引入更多的策略,以确保重要数据的绝对安全。目前,Eagle的策略涵盖多种模式,包括从访问模式、频繁访问数据集,预定义查询类型、Hive 表和列、HBase 表以及基于机器学习模型生成的用户Profile相关的所有策略等。同时,我们也有广泛的策略来防止数据的丢失、数据被拷贝到不安全地点、敏感数据被未授权区域访问等。Eagle策略定义上极大的灵活性和扩展性使得我们未来可以轻易地继续扩展更多更复杂的策略以支持更多多元化的用例场景。&lt;/p&gt;
    181 187
    182 &lt;h2 id=&quot;section-1&quot;&gt;后续计划&lt;/h2&gt;
    188 &lt;h2 id=&quot;后续计划&quot;&gt;后续计划&lt;/h2&gt;
    183 189 &lt;p&gt;过去两年中,在eBay 除了被用于数据行为监控以外,Eagle 核心框架还被广泛用于监控节点健康状况、Hadoop应用性能指标、Hadoop 核心服务以及整个Hadoop集群的健康状况等诸多领域。我们还建立一系列的自动化机制,例如节点修复等,帮助我们平台部门极大得节省了我们人工劳力,并有效地提升了整个集群资源地利用率。&lt;/p&gt;
    184 190
    185 191 &lt;p&gt;以下是我们目前正在开发中地一些特性:&lt;/p&gt;
     
    196 202 &lt;/li&gt;
    197 203 &lt;/ul&gt;
    198 204
    199 &lt;h2 id=&quot;section-2&quot;&gt;关于作者&lt;/h2&gt;
    205 &lt;h2 id=&quot;关于作者&quot;&gt;关于作者&lt;/h2&gt;
    200 206 &lt;p&gt;&lt;a href=&quot;https://github.com/haoch&quot;&gt;陈浩&lt;/a&gt;,Apache Eagle Committer 和 PMC 成员,eBay 分析平台基础架构部门高级软件工程师,负责Eagle的产品设计、技术架构、核心实现以及开源社区推广等。&lt;/p&gt;
    201 207
    202 208 &lt;p&gt;感谢以下来自Apache Eagle社区和eBay公司的联合作者们对本文的贡献:&lt;/p&gt;
     
    210 216
    211 217 &lt;p&gt;eBay 分析平台基础架构部(Analytics Data Infrastructure)是eBay的全球数据及分析基础架构部门,负责eBay在数据库、数据仓库、Hadoop、商务智能以及机器学习等各个数据平台开发、管理等,支持eBay全球各部门运用高端的数据分析解决方案作出及时有效的作业决策,为遍布全球的业务用户提供数据分析解决方案。&lt;/p&gt;
    212 218
    213 &lt;h2 id=&quot;section-3&quot;&gt;参考资料&lt;/h2&gt;
    219 &lt;h2 id=&quot;参考资料&quot;&gt;参考资料&lt;/h2&gt;
    214 220
    215 221 &lt;ul&gt;
    216 222 &lt;li&gt;Apache Eagle 文档:&lt;a href=&quot;http://goeagle.io&quot;&gt;http://goeagle.io&lt;/a&gt;&lt;/li&gt;
     
    218 224 &lt;li&gt;Apache Eagle 项目:&lt;a href=&quot;http://incubator.apache.org/projects/eagle.html&quot;&gt;http://incubator.apache.org/projects/eagle.html&lt;/a&gt;&lt;/li&gt;
    219 225 &lt;/ul&gt;
    220 226
    221 &lt;h2 id=&quot;section-4&quot;&gt;引用链接&lt;/h2&gt;
    227 &lt;h2 id=&quot;引用链接&quot;&gt;引用链接&lt;/h2&gt;
    222 228 &lt;ul&gt;
    223 229 &lt;li&gt;&lt;strong&gt;CSDN&lt;/strong&gt;: &lt;a href=&quot;http://www.csdn.net/article/2015-10-29/2826076&quot;&gt;http://www.csdn.net/article/2015-10-29/2826076&lt;/a&gt;&lt;/li&gt;
    224 230 &lt;li&gt;&lt;strong&gt;OSCHINA&lt;/strong&gt;: &lt;a href=&quot;http://www.oschina.net/news/67515/apache-eagle&quot;&gt;http://www.oschina.net/news/67515/apache-eagle&lt;/a&gt;&lt;/li&gt;
  • eagle/site/index.html

     
    88 88 </div>
    89 89 </div>
    90 90
    91 <div class="download" style="margin-left:10px;" title="Documents"><a href="docs/latest">DOCS</a></div>
    91 <div class="download" style="margin-left:10px;" title="Documents"><a href="docs/">DOCS</a></div>
    92 92 <div class="download"><a href="docs/download-latest.html" title="Downloads">DOWNLOAD</a></div>
    93 93 </div>
    94 94 </div>
  • eagle/site/post/2015/10/27/apache-eagle-announce-cn.html

     
    93 93
    94 94 <p>日前,eBay公司隆重宣布正式向开源业界推出分布式实时安全监控方案 - Apache Eagle (http://goeagle.io),该项目已于2015年10月26日正式加入Apache 成为孵化器项目。Apache Eagle提供一套高效分布式的流式策略引擎,具有高实时、可伸缩、易扩展、交互友好等特点,同时集成机器学习对用户行为建立Profile以实现智能实时地保护Hadoop生态系统中大数据的安全。</p>
    95 95
    96 <h2 id="section">背景</h2>
    96 <h2 id="背景">背景</h2>
    97 97 <p>随着大数据的发展,越来越多的成功企业或者组织开始采取数据驱动商业的运作模式。在eBay,我们拥有数万名工程师、分析师和数据科学家,他们每天访问分析数PB级的数据,以为我们的用户带来无与伦比的体验。在全球业务中,我们也广泛地利用海量大数据来连接我们数以亿计的用户。</p>
    98 98
    99 99 <p>近年来,Hadoop已经逐渐成为大数据分析领域最受欢迎的解决方案,eBay也一直在使用Hadoop技术从数据中挖掘价值,例如,我们通过大数据提高用户的搜索体验,识别和优化精准广告投放,充实我们的产品目录,以及通过点击流分析以理解用户如何使用我们的在线市场平台等。</p>
     
    130 130 <li><strong>开源</strong>:Eagle一直根据开源的标准开发,并构建于诸多大数据领域的开源产品之上,因此我们决定以Apache许可证开源Eagle,以回馈社区,同时也期待获得社区的反馈、协作与支持。</li>
    131 131 </ul>
    132 132
    133 <h2 id="eagle">Eagle概览</h2>
    133 <h2 id="eagle概览">Eagle概览</h2>
    134 134
    135 135 <p><img src="/images/posts/eagle-group.png" alt="" /></p>
    136 136
    137 <h4 id="data-collection-and-storage">数据流接入和存储(Data Collection and Storage)</h4>
    137 <h4 id="数据流接入和存储data-collection-and-storage">数据流接入和存储(Data Collection and Storage)</h4>
    138 138 <p>Eagle提供高度可扩展的编程API,可以支持将任何类型的数据源集成到Eagle的策略执行引擎中。例如,在Eagle HDFS 审计事件(Audit)监控模块中,通过Kafka来实时接收来自Namenode Log4j Appender 或者 Logstash Agent 收集的数据;在Eagle Hive 监控模块中,通过YARN API 收集正在运行Job的Hive 查询日志,并保证比较高的可伸缩性和容错性。</p>
    139 139
    140 <h4 id="data-processing">数据实时处理(Data Processing)</h4>
    140 <h4 id="数据实时处理data-processing">数据实时处理(Data Processing)</h4>
    141 141
    142 142 <p><strong>流处理API(Stream Processing API)Eagle</strong> 提供独立于物理平台而高度抽象的流处理API,目前默认支持Apache Storm,但是也允许扩展到其他任意流处理引擎,比如Flink 或者 Samza等。该层抽象允许开发者在定义监控数据处理逻辑时,无需在物理执行层绑定任何特定流处理平台,而只需通过复用、拼接和组装例如数据转换、过滤、外部数据Join等组件,以实现满足需求的DAG(有向无环图),同时,开发者也可以很容易地以编程地方式将业务逻辑流程和Eagle 策略引擎框架集成起来。Eagle框架内部会将描述业务逻辑的DAG编译成底层流处理架构的原生应用,例如Apache Storm Topology 等,从事实现平台的独立。</p>
    143 143
    144 144 <p><strong>以下是一个Eagle如何处理事件和告警的示例:</strong></p>
    145 145
    146 <pre><code>StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); // storm env
    146 <div class="highlighter-rouge"><pre class="highlight"><code>StormExecutionEnvironment env = ExecutionEnvironmentFactory.getStorm(config); // storm env
    147 147 StreamProducer producer = env.newSource(new KafkaSourcedSpoutProvider().getSpout(config)).renameOutputFields(1) // declare kafka source
    148 148 .flatMap(new AuditLogTransformer()) // transform event
    149 149 .groupBy(Arrays.asList(0)) // group by 1st field
     
    151 151 .alertWithConsumer(“userActivity“,”userProfileExecutor“) // ML policy evaluation
    152 152 env.execute(); // execute stream processing and alert
    153 153 </code></pre>
    154 </div>
    154 155
    155 156 <p><strong>告警框架(Alerting Framework)Eagle</strong>告警框架由流元数据API、策略引擎服务提供API、策略Partitioner API 以及预警去重框架等组成:</p>
    156 157
     
    160 161 <li>
    161 162 <p><strong>扩展性</strong> Eagle的策略引擎服务提供API允许你插入新的策略引擎</p>
    162 163
    163 <pre><code> public interface PolicyEvaluatorServiceProvider {
    164 <div class="highlighter-rouge"><pre class="highlight"><code> public interface PolicyEvaluatorServiceProvider {
    164 165 public String getPolicyType(); // literal string to identify one type of policy
    165 166 public Class&lt;? extends PolicyEvaluator&gt; getPolicyEvaluator(); // get policy evaluator implementation
    166 167 public List&lt;Module&gt; getBindingModules(); // policy text with json format to object mapping
     
    171 172 public void onPolicyDelete(); // invoked when policy is deleted
    172 173 }
    173 174 </code></pre>
    175 </div>
    174 176 </li>
    175 177 <li><strong>策略Partitioner API</strong> 允许策略在不同的物理节点上并行执行。也允许你自定义策略Partitioner类。这些功能使得策略和事件完全以分布式的方式执行。</li>
    176 178 <li>
    177 179 <p><strong>可伸缩性</strong> Eagle 通过支持策略的分区接口来实现大量的策略可伸缩并发地运行</p>
    178 180
    179 <pre><code> public interface PolicyPartitioner extends Serializable {
    181 <div class="highlighter-rouge"><pre class="highlight"><code> public interface PolicyPartitioner extends Serializable {
    180 182 int partition(int numTotalPartitions, String policyType, String policyId); // method to distribute policies
    181 183 }
    182 184 </code></pre>
    185 </div>
    183 186
    184 187 <p><img src="/images/posts/policy-partition.png" alt="" /></p>
    185 188
     
    236 239 <li>
    237 240 <p>单一事件执行策略(用户访问Hive中的敏感数据列)</p>
    238 241
    239 <pre><code> from hiveAccessLogStream[sensitivityType=='PHONE_NUMBER'] select * insert into outputStream;
    242 <div class="highlighter-rouge"><pre class="highlight"><code> from hiveAccessLogStream[sensitivityType=='PHONE_NUMBER'] select * insert into outputStream;
    240 243 </code></pre>
    244 </div>
    241 245 </li>
    242 246 <li>
    243 247 <p>基于窗口的策略(用户在10分钟内访问目录 /tmp/private 多余 5次)</p>
    244 248
    245 <pre><code> hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue &gt;= 5 insert into outputStream;
    249 <div class="highlighter-rouge"><pre class="highlight"><code> hdfsAuditLogEventStream[(src == '/tmp/private')]#window.externalTime(timestamp,10 min) select user, count(timestamp) as aggValue group by user having aggValue &gt;= 5 insert into outputStream;
    246 250 </code></pre>
    251 </div>
    247 252 </li>
    248 253 </ul>
    249 254
    250 255 <p><strong>查询服务(Query Service)</strong> Eagle 提供类SQL的REST API用来实现针对海量数据集的综合计算、查询和分析的能力,支持例如过滤、聚合、直方运算、排序、top、算术表达式以及分页等。Eagle优先支持HBase 作为其默认数据存储,但是同时也支持基JDBC的关系型数据库。特别是当选择以HBase作为存储时,Eagle便原生拥有了HBase存储和查询海量监控数据的能力,Eagle 查询框架会将用户提供的类SQL查询语法最终编译成为HBase 原生的Filter 对象,并支持通过HBase Coprocessor进一步提升响应速度。</p>
    251 256
    252 <pre><code>query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}&amp;pageSize=100000
    257 <div class="highlighter-rouge"><pre class="highlight"><code>query=AlertDefinitionService[@dataSource="hiveQueryLog"]{@policyDef}&amp;pageSize=100000
    253 258 </code></pre>
    259 </div>
    254 260
    255 <h2 id="eagleebay">Eagle在eBay的使用场景</h2>
    261 <h2 id="eagle在ebay的使用场景">Eagle在eBay的使用场景</h2>
    256 262 <p>目前,Eagle的数据行为监控系统已经部署到一个拥有2500多个节点的Hadoop集群之上,用以保护数百PB数据的安全,并正计划于今年年底之前扩展到其他上十个Hadoop集群上,从而覆盖eBay 所有主要Hadoop的10000多台节点。在我们的生产环境中,我们已针对HDFS、Hive 等集群中的数据配置了一些基础的安全策略,并将于年底之前不断引入更多的策略,以确保重要数据的绝对安全。目前,Eagle的策略涵盖多种模式,包括从访问模式、频繁访问数据集,预定义查询类型、Hive 表和列、HBase 表以及基于机器学习模型生成的用户Profile相关的所有策略等。同时,我们也有广泛的策略来防止数据的丢失、数据被拷贝到不安全地点、敏感数据被未授权区域访问等。Eagle策略定义上极大的灵活性和扩展性使得我们未来可以轻易地继续扩展更多更复杂的策略以支持更多多元化的用例场景。</p>
    257 263
    258 <h2 id="section-1">后续计划</h2>
    264 <h2 id="后续计划">后续计划</h2>
    259 265 <p>过去两年中,在eBay 除了被用于数据行为监控以外,Eagle 核心框架还被广泛用于监控节点健康状况、Hadoop应用性能指标、Hadoop 核心服务以及整个Hadoop集群的健康状况等诸多领域。我们还建立一系列的自动化机制,例如节点修复等,帮助我们平台部门极大得节省了我们人工劳力,并有效地提升了整个集群资源地利用率。</p>
    260 266
    261 267 <p>以下是我们目前正在开发中地一些特性:</p>
     
    272 278 </li>
    273 279 </ul>
    274 280
    275 <h2 id="section-2">关于作者</h2>
    281 <h2 id="关于作者">关于作者</h2>
    276 282 <p><a href="https://github.com/haoch">陈浩</a>,Apache Eagle Committer 和 PMC 成员,eBay 分析平台基础架构部门高级软件工程师,负责Eagle的产品设计、技术架构、核心实现以及开源社区推广等。</p>
    277 283
    278 284 <p>感谢以下来自Apache Eagle社区和eBay公司的联合作者们对本文的贡献:</p>
     
    286 292
    287 293 <p>eBay 分析平台基础架构部(Analytics Data Infrastructure)是eBay的全球数据及分析基础架构部门,负责eBay在数据库、数据仓库、Hadoop、商务智能以及机器学习等各个数据平台开发、管理等,支持eBay全球各部门运用高端的数据分析解决方案作出及时有效的作业决策,为遍布全球的业务用户提供数据分析解决方案。</p>
    288 294
    289 <h2 id="section-3">参考资料</h2>
    295 <h2 id="参考资料">参考资料</h2>
    290 296
    291 297 <ul>
    292 298 <li>Apache Eagle 文档:<a href="http://goeagle.io">http://goeagle.io</a></li>
     
    294 300 <li>Apache Eagle 项目:<a href="http://incubator.apache.org/projects/eagle.html">http://incubator.apache.org/projects/eagle.html</a></li>
    295 301 </ul>
    296 302
    297 <h2 id="section-4">引用链接</h2>
    303 <h2 id="引用链接">引用链接</h2>
    298 304 <ul>
    299 305 <li><strong>CSDN</strong>: <a href="http://www.csdn.net/article/2015-10-29/2826076">http://www.csdn.net/article/2015-10-29/2826076</a></li>
    300 306 <li><strong>OSCHINA</strong>: <a href="http://www.oschina.net/news/67515/apache-eagle">http://www.oschina.net/news/67515/apache-eagle</a></li>