This presentation describes a dataflow pipeline created in Apache NiFi that uses Idyl E3 Entity Extraction Engine to extract entities from natural language text.

1. Entity Extraction from Natural Language Text in a
Data Flow Pipeline
Mountain Fog
Copyright 2017 Mountain Fog, Inc. All Rights Reserved.

2. Goals
● Ingest text files from the file system.
● Extract entities from the text.
● Store entities in a MongoDB database.
Text Entities
Apache NiFi Dataflow
S3

3. Tools in Use
● Apache NiFi
● Facilitates data flow
between disparate
sources and services.
● https://nifi.apache.org/
● Idyl E3 Entity Extraction
Engine
● Extracts entities from natural
language text via user-
generated entity models
through a REST API.
● http://www.mtnfog.com/

4. Idyl E3 Entity Extraction Engine
Launch via the AWS Marketplace.
Comes with an entity model for English-language
person entities.
REST API for entity extraction.
Free to use.

5. NiFi Processors
A processor executes the dataflow work “of data
routing, transformation, or mediation between
systems.” [1]
GetFile Processor
IdylE3 Processor
PutMongoDB Processor
[1] https://nifi.apache.org/docs/nifi-docs/html/overview.html

6. Ingest Text Files
Processor’s properties set to read all files from /ingest.

7. Send Text to Idyl E3
The Idyl E3 endpoint is set in the processor’s properties.

8. Store Entities in MongoDB
The MongoDB URI set in the processor’s properties.

9. The Flow

10. The Result
Start the NiFi flow.
Files are removed from the ingest directory.
Entities appear in the MongoDB collection.
Take a well-deserved break.

11. Going Further
Scale Idyl E3 behind a load balancer.
Query entities via the
Entity Query Language (EQL) processor.
Extract other entity types through custom entity
models.

12. support@mtnfog.com
www.mtnfog.com