Kafka
The Kafka feed type in ArcGIS Velocity subscribes to and consumes messages from an externally accessible Kafka broker. Kafka is an open-source distributed streaming platform that you can use to do the following:
Publish and subscribe to a stream of messages through topics.
Reliably store streams of messages.
Process streams of messages.
Examples
The following are example use cases for the feed:
A data analyst wants to ingest sensor data through a Kafka broker.
A transportation department configures a Kafka feed to consume vehicle positions and attribute information.
Usage notes
Keep the following in mind when working with the feed:
This feed type supports connectivity to Confluent Kafka.
Do not use a single consumer group for multiple topics. Create a consumer group for each topic.
If there are multiple consumers (a Velocity Kafka feed is a consumer) using a single consumer group to consume data from a topic, each consumer (or feed) only receives part of the data depending on the number of partitions of the topic.
If there are more consumers of a topic using the same consumer group than the number of partitions, some consumers do not receive any data.
Do not use the same consumer group referenced by a Velocity Kafka feed outside of Velocity.
After configuring feed connection parameters, configure input data to define the schema and the key parameters.
Parameters
The following are the parameters for the feed:
|
Parameter |
Description |
Data type |
|---|---|---|
|
Broker |
A comma-delimited list of Kafka brokers specified as |
String |
|
Topic |
The Kafka topic from which Velocity consumes messages. |
String |
|
Consumer group ID (optional) |
The Kafka consumer group Velocity joins as a member. |
String |
|
Authentication |
Specifies the type of authentication that is used to access the HTTP endpoint. Authentication options are None, SASL/PLAIN, SASL/SCRAM-SHA-256, and SASL/SCRAM-SHA-512. |
String |
|
Use SSL |
Species whether SSL (SASL_SSL) is used when connecting to the Kafka broker. Note:This should match the configuration the broker has on a specified port. If false, Velocity connects via PLAINTEXT. |
Boolean |
|
Username |
The username for accessing the Kafka broker. This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512. |
String |
|
Password |
The password for accessing the Kafka broker. This parameter only applies when Authentication is set to SASL/PLAIN, SASL/SCRAM-SHA-256, or SASL/SCRAM-SHA-512. |
String |