ElastAlert Metadata Index

ElastAlert uses Elasticsearch to store various information about its state. This not only allows for some level of auditing and debugging of ElastAlert’s operation, but also to avoid loss of data or duplication of alerts when ElastAlert is shut down, restarted, or crashes. This cluster and index information is defined in the global config file with es_host, es_port and writeback_index. ElastAlert must be able to write to this index. The script, elastalert-create-index will create the index with the correct mapping for you, and optionally copy the documents from an existing ElastAlert writeback index. Run it and it will prompt you for the cluster information.

ElastAlert will create three different types of documents in the writeback index:

elastalert_status

elastalert_status is a log of the queries performed for a given rule and contains:

  • @timestamp: The time when the document was uploaded to Elasticsearch. This is after a query has been run and the results have been processed.
  • rule_name: The name of the corresponding rule.
  • starttime: The beginning of the timestamp range the query searched.
  • endtime: The end of the timestamp range the query searched.
  • hits: The number of results from the query.
  • matches: The number of matches that the rule returned after processing the hits. Note that this does not necessarily mean that alerts were triggered.
  • time_taken: The number of seconds it took for this query to run.

elastalert_status is what ElastAlert will use to determine what time range to query when it first starts to avoid duplicating queries. For each rule, it will start querying from the most recent endtime. If ElastAlert is running in debug mode, it will still attempt to base its start time by looking for the most recent search performed, but it will not write the results of any query back to Elasticsearch.

elastalert

elastalert is a log of information about every alert triggered and contains:

  • @timestamp: The time when the document was uploaded to Elasticsearch. This is not the same as when the alert was sent, but rather when the rule outputs a match.
  • rule_name: The name of the corresponding rule.
  • alert_info: This contains the output of Alert.get_info, a function that alerts implement to give some relevant context to the alert type. This may contain alert_info.type, alert_info.recipient, or any number of other sub fields.
  • alert_sent: A boolean value as to whether this alert was actually sent or not. It may be false in the case of an exception or if it is part of an aggregated alert.
  • alert_time: The time that the alert was or will be sent. Usually, this is the same as @timestamp, but may be some time in the future, indicating when an aggregated alert will be sent.
  • match_body: This is the contents of the match dictionary that is used to create the alert. The subfields may include a number of things containing information about the alert.
  • alert_exception: This field is only present when the alert failed because of an exception occurring, and will contain the exception information.
  • aggregate_id: This field is only present when the rule is configured to use aggregation. The first alert of the aggregation period will contain an alert_time set to the aggregation time into the future, and subsequent alerts will contain the document ID of the first. When the alert_time is reached, all alerts with that aggregate_id will be sent together.

elastalert_error

When an error occurs in ElastAlert, it is written to both Elasticsearch and to stderr. The elastalert_error type contains:

  • @timestamp: The time when the error occurred.
  • message: The error or exception message.
  • traceback: The traceback from when the error occurred.
  • data: Extra information about the error. This often contains the name of the rule which caused the error.

silence

silence is a record of when alerts for a given rule will be suppressed, either because of a realert setting or from using –silence. When an alert with realert is triggered, a silence record will be written with until set to the alert time plus realert.

  • @timestamp: The time when the document was uploaded to Elasticsearch.
  • rule_name: The name of the corresponding rule.
  • until: The timestamp when alerts will begin being sent again.
  • exponent: The exponential factor which multiplies realert. The length of this silence is equal to realert * 2**exponent. This will be 0 unless exponential_realert is set.

Whenever an alert is triggered, ElastAlert will check for a matching silence document, and if the until timestamp is in the future, it will ignore the alert completely. See the Running ElastAlert section for information on how to silence an alert.