Timestamp definitions

This document explains common timestamps for events and detections. For more information about timestamps, see Date function.

The following timestamps are related to events:

• Event timestamp: Time when an event occurred and is stored in the metadata.event_timestamp UDM field. Rules and UDM searches use the metadata.event_timestamp field for queries.
• Collected timestamp: Time when an event was collected by the local collection infrastructure, such as the forwarder. This is stored in the metadata.collected_timestamp UDM field.
• Ingested timestamp: Time when an event was ingested by Google Security Operations. This is stored in the metadata.ingested_timestamp UDM field.

Indexing and data availability timestamps

In addition to the event, collection, and ingestion timestamps, understanding when data becomes fully searchable and usable within Google Security Operations is crucial. This involves the indexing process, which makes ingested logs and UDM events available for various analysis and detection features.

The following section outlines data availability in relation to the indexing process:

• Raw Log Search Availability: Raw logs become available for searching using the Raw log search feature very quickly after ingestion. This search method directly scans the original, unparsed log text for your query. Expected availability for raw log searching is often within seconds to a few minutes after the log arrives at the Google SecOps ingestion point.

• UDM Search Availability: After raw logs are ingested, they go through parsing and normalization to be converted into Unified Data Model (UDM) events. Subsequently, these UDM events undergo indexing. The indexed UDM data is required for a UDM search and becomes available for querying typically within 2–15 minutes after the initial ingestion. A more precise average time is around 5 minutes from when data is sent to the Google SecOps ingestion service.

• Detection Rule Availability: UDM events are used by the Detection Engine. For real-time and scheduled rules to evaluate against new data, the UDM events must be parsed, normalized, and indexed. Rule execution typically occurs within 5–10 minutes after the event arrives at the Google SecOps ingestion point, depending on the rule type and scheduling.

Detecting Delays:

To detect delays between log ingestion and when data is searchable, customers can monitor the time differences between:

1. metadata.ingested_timestamp (when Google SecOps received the event).
2. The time when the event becomes queryable using UDM Search or available for rule execution.

While specific indexing completion timestamps are not directly exposed as UDM fields, the documented availability times for UDM Search and Detection Engine provide a guide to the indexing latency. Significant deviations from the expected 2-15 minute window could indicate processing delays.

The following timestamps are stored with detections:

• Detection window: For rules with a match section, a detection is created over the time range, called the detection window. The event timestamps for events that triggered the detection are within the detection window.
• Detection timestamp: For rules with a match section, the detection timestamp is the end time of the detection window. Otherwise, the detection timestamp is the metadata.event_timestamp of the event that generated the detection.
• Detection created timestamp: Date and time the detection was created by detection engine.

Where timestamps appear in the application

The following sections define where you can view these timestamps in the UI.

UDM Event viewer

To open the UDM Event view, do the following:

1. Perform a UDM Search.
2. In the Events tab, select an event to open the Event viewer

3. The UDM event pane displays the following data:

   • Event timestamp is stored in the metadata.event_timestamp UDM field (1).
   • Ingested timestamp is stored in the metadata.ingested_timestamp UDM field (2).

   

Detections panel

To open the Detections view, do the following:

1. Open Detections > Rules & Detections, and then click the Dashboard button.

2. Click the rule name link under the Rule name column. The Detections panel appears and displays the following:

   • Detection timestamp appears in rows that identify a detection (1).
   • Event timestamp appears in rows that identify events (2).

   

Alert view

To open the Alert view, do the following:

1. Open Detections > Alerts & IOCs.
2. Under the Alerts tab, click the alert name link in the Name column.

3. Click the Overview tab to display the following:

   • Alert (or Detection) created timestamp appears in the Alert Details pane > Created field (1).
   • Detection window appears in the Detection Summary pane > Detection window field (2).
   • Detection timestamp appears is in the Detection Summary pane > Alerts detected at field (3).

   

Need more help? Get answers from Community members and Google SecOps professionals.