Data Ingestion

Intelligent Data Store ingests data in the form of a normalised Kafka topic. To achieve this:
  1. The enterprise or keyspace name is deduced from groupName attribute available in the Kafka messages.

    The data collected from Connect are broadly classified into different types of events:

    • LifeCycle events: The lifecycle events are:
      • registration of devices
      • de-registration of devices
      • expiration of devices
      • updates of devices
      For lifecycle events, devices/gateway are selected based on the criteria provided in the request which can be either a list of serial numbers or a combination of manufacturer related information like make, model, and firmware version. If it is not possible to observe some or all of the events on any device, the IMPACT server neither reports data nor responds with an error within the scope of an asynchronous response notification. If a subscription request for a particular device which cannot be observed at the moment, then IMPACT server validates the request and accepts it for processing later.
    • Observationevents: The observation events contain data of a particular resourcePath and its values.
    • Response events: The response events contain the responses from the device to an operation, like read, write, exec, or delete.
    • DJR events: The DJR (Device Job Result) events contain the results of the jobs run through the IMPACT.
    • Alert events: Alert events contain the data of matching a particular rule condition defined in the rule.
    • Aggregation events: Aggregation events contain the data for which aggregate queries match the condition in the defined rule.
    • MONTE events: Network Exposure events contain the monitoring data for IoT devices.
    • SMS events: SMS events contain the delivery receipts for the SMS that are transported from SMS connector to the devices.

    For use cases and examples of different event types, refer Data Access APIs.

    For more information on different event types, see Intelligent Data Store API Swagger documentation.

  2. The data is stored in separate tables based on the message type to facilitate time-series like querying.