HTTP listener for incoming web traffic logs, streams categorised logs to Kafka.
c97fde47-0158-1000-f03d-70c65cc94d3b
Streaming Example
eaecb55d-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
1 GB
10000
c97fde47-0158-1000-0000-000000000000
eaec29f4-0158-1000-0000-000000000000
PROCESSOR
0 sec
1
matched
c97fde47-0158-1000-0000-000000000000
e8042e4e-0158-1000-0000-000000000000
PROCESSOR
0
ed0e5d8e-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
1 GB
10000
c97fde47-0158-1000-0000-000000000000
ed0647db-0158-1000-0000-000000000000
PROCESSOR
0 sec
1
success
c97fde47-0158-1000-0000-000000000000
c98038dc-0158-1000-0000-000000000000
PROCESSOR
0
ed5587da-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
1 GB
10000
c97fde47-0158-1000-0000-000000000000
ed4e7eb3-0158-1000-0000-000000000000
PROCESSOR
0 sec
1
unmatched
c97fde47-0158-1000-0000-000000000000
eaec29f4-0158-1000-0000-000000000000
PROCESSOR
0
11794866-0159-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
1 GB
10000
c97fde47-0158-1000-0000-000000000000
e8042e4e-0158-1000-0000-000000000000
PROCESSOR
0 sec
1
success
c97fde47-0158-1000-0000-000000000000
ed0647db-0158-1000-0000-000000000000
PROCESSOR
0
4b8b6395-0159-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
1 GB
10000
c97fde47-0158-1000-0000-000000000000
ed4f8536-0158-1000-0000-000000000000
PROCESSOR
0 sec
1
matched
c97fde47-0158-1000-0000-000000000000
eaec29f4-0158-1000-0000-000000000000
PROCESSOR
0
4b8c450c-0159-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
1 GB
10000
c97fde47-0158-1000-0000-000000000000
ed4b15f5-0158-1000-0000-000000000000
PROCESSOR
0 sec
1
success
c97fde47-0158-1000-0000-000000000000
ed4f8536-0158-1000-0000-000000000000
PROCESSOR
0
4b8c553e-0159-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
1 GB
10000
c97fde47-0158-1000-0000-000000000000
ed4b15f5-0158-1000-0000-000000000000
PROCESSOR
0 sec
1
success
c97fde47-0158-1000-0000-000000000000
ed4e7eb3-0158-1000-0000-000000000000
PROCESSOR
0
c98038dc-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
253.44178122311757
0.0
WARN
1
Base Path
Base Path
Listening Port
Listening Port
Max Data to Receive per Second
Max Data to Receive per Second
SSL Context Service
org.apache.nifi.ssl.SSLContextService
SSL Context Service
Authorized DN Pattern
Authorized DN Pattern
Max Unconfirmed Flowfile Time
Max Unconfirmed Flowfile Time
HTTP Headers to receive as Attributes (Regex)
HTTP Headers to receive as Attributes (Regex)
false
30 sec
Base Path
contentListener
Listening Port
8081
Max Data to Receive per Second
SSL Context Service
Authorized DN Pattern
.*
Max Unconfirmed Flowfile Time
60 secs
HTTP Headers to receive as Attributes (Regex)
0
0 sec
TIMER_DRIVEN
1 sec
Watch Proxy Logs
false
success
org.apache.nifi.processors.standard.ListenHTTP
e8042e4e-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
255.4417724609375
273.9999737033378
WARN
1
Destination
Destination
Return Type
Return Type
Path Not Found Behavior
Path Not Found Behavior
Null Value Representation
Null Value Representation
auth.status
auth.status
response.status
response.status
false
30 sec
Destination
flowfile-attribute
Return Type
auto-detect
Path Not Found Behavior
ignore
Null Value Representation
empty string
auth.status
$.authenticated
response.status
$.res_status
0
0 sec
TIMER_DRIVEN
1 sec
Capture Response and Auth Status
true
failure
false
matched
true
unmatched
org.apache.nifi.processors.standard.EvaluateJsonPath
eaec29f4-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
255.4417724609375
537.9999737033378
WARN
1
Routing Strategy
Routing Strategy
matched
matched
false
30 sec
Routing Strategy
Route to Property name
matched
${response.status:equals(200)}
0
0 sec
TIMER_DRIVEN
1 sec
On Success
false
matched
false
unmatched
org.apache.nifi.processors.standard.RouteOnAttribute
ed0647db-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
884.857421875
11.168739725066274
WARN
1
Rate Control Criteria
Rate Control Criteria
Maximum Rate
Maximum Rate
Rate Controlled Attribute
Rate Controlled Attribute
Time Duration
Time Duration
Grouping Attribute
Grouping Attribute
false
30 sec
Rate Control Criteria
data rate
Maximum Rate
1 MB
Rate Controlled Attribute
Time Duration
1 min
Grouping Attribute
0
0 sec
TIMER_DRIVEN
1 sec
ControlRate
true
failure
false
success
org.apache.nifi.processors.standard.ControlRate
ed4b15f5-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
293.72320556640625
1089.053440500213
WARN
1
bootstrap.servers
bootstrap.servers
security.protocol
security.protocol
sasl.kerberos.service.name
sasl.kerberos.service.name
ssl.context.service
org.apache.nifi.ssl.SSLContextService
ssl.context.service
topic
topic
acks
acks
kafka-key
kafka-key
key-attribute-encoding
key-attribute-encoding
message-demarcator
message-demarcator
max.request.size
max.request.size
max.block.ms
max.block.ms
partitioner.class
partitioner.class
compression.type
compression.type
false
30 sec
bootstrap.servers
localhost:9092
security.protocol
PLAINTEXT
sasl.kerberos.service.name
ssl.context.service
topic
${kafka.topic}
acks
0
kafka-key
key-attribute-encoding
utf-8
message-demarcator
max.request.size
1 MB
max.block.ms
30 sec
partitioner.class
org.apache.kafka.clients.producer.internals.DefaultPartitioner
compression.type
none
0
0 sec
TIMER_DRIVEN
1 sec
Enqueue Processed Logs
true
failure
true
success
org.apache.nifi.processors.kafka.pubsub.PublishKafka
ed4e7eb3-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
575.4296875
800.9344219455253
WARN
1
Delete Attributes Expression
Delete Attributes Expression
kafka.topic
kafka.topic
false
30 sec
Delete Attributes Expression
kafka.topic
error_logs
0
0 sec
TIMER_DRIVEN
1 sec
Kafka Topic: Error Logs
false
success
org.apache.nifi.processors.attributes.UpdateAttribute
ed4f8536-0158-1000-0000-000000000000
c97fde47-0158-1000-0000-000000000000
0.0
803.5218242892753
WARN
1
Delete Attributes Expression
Delete Attributes Expression
kafka.topic
kafka.topic
false
30 sec
Delete Attributes Expression
kafka.topic
success_logs
0
0 sec
TIMER_DRIVEN
1 sec
Kafka Topic: Success Logs
false
success
org.apache.nifi.processors.attributes.UpdateAttribute
12/31/2016 12:08:01 GMT