Querying historical data in SCADA systems - Full Explanation

Introduction

Imagine you want to understand how a machine behaved yesterday or last week to find out why it stopped working. Querying historical data helps you look back at past information collected by a system to find patterns or problems.

Explanation

Data Collection

SCADA systems continuously collect data from sensors and devices in real time. This data is stored in a database or historian for future use. The quality and frequency of data collection affect how detailed the historical records are.

Historical data depends on how and when the system collects and stores information.

Data Storage

Collected data is saved in a specialized database called a historian. This storage is optimized to handle large amounts of time-stamped data efficiently. It allows quick access to past records without slowing down the system.

A historian stores time-stamped data to enable fast and organized retrieval.

Querying Methods

Users can ask the system questions about past data using queries. These queries specify what data to retrieve, such as values from a certain sensor during a specific time period. The system then returns the requested information for analysis.

Queries let users select specific historical data based on time and source.

Data Analysis

Once data is retrieved, it can be analyzed to find trends, detect anomalies, or understand events. This helps in making decisions like maintenance scheduling or improving processes. Visualization tools often help show this data clearly.

Analyzing historical data reveals insights to improve system performance.

Real World Analogy

Think of a security camera that records everything happening in a store. Later, if something goes wrong, you can watch the recordings to see what happened and when. Querying historical data is like searching through those recordings to find specific moments.

Data Collection → The security camera continuously recording video footage.

Data Storage → The hard drive where all the video recordings are saved.

Querying Methods → Using the video player to jump to a specific date and time.

Data Analysis → Watching the footage to understand what caused an incident.

Diagram

Diagram

┌───────────────┐     ┌───────────────┐     ┌───────────────┐     ┌───────────────┐
│ Data          │     │ Data          │     │ Querying      │     │ Data          │
│ Collection    │────▶│ Storage       │────▶│ Methods       │────▶│ Analysis      │
│ (Sensors)     │     │ (Historian)   │     │ (User Queries)│     │ (Insights)    │
└───────────────┘     └───────────────┘     └───────────────┘     └───────────────┘

This diagram shows the flow from collecting data, storing it, querying it, and finally analyzing the results.

Key Facts

Historian → A specialized database designed to store and retrieve time-stamped data efficiently.

Time-stamped Data → Data entries that include the exact time they were recorded.

Query → A request to retrieve specific data from a database based on criteria like time or source.

Data Analysis → The process of examining data to find useful patterns or information.

Common Confusions

Believing historical data is the same as real-time data.

Believing historical data is the same as real-time data. Historical data refers to past records stored over time, while real-time data is current and continuously updated.

Thinking all data is stored forever.

Thinking all data is stored forever. Storage limits mean older data may be archived or deleted; retention policies control how long data is kept.

Summary

Historical data lets you look back at past system behavior to understand and improve it.

Data is collected continuously, stored efficiently, and retrieved using queries based on time and source.

Analyzing this data helps find patterns and solve problems in SCADA systems.

Practice

1. What is the main purpose of querying historical data in SCADA systems?

easy

A. To control real-time device operations

B. To review past system behavior and analyze trends

C. To update firmware on sensors

D. To configure network settings

Solution

1. Step 1: Understand the role of historical data

   Historical data stores past readings and events from the system.

2. Step 2: Identify the purpose of querying it

   Querying helps analyze past behavior and detect trends or issues.

3. Final Answer:

   To review past system behavior and analyze trends -> Option B

4. Quick Check:

   Historical data = review past behavior [OK]

Hint: Historical data is for past info, not real-time control [OK]

Common Mistakes:

• Confusing historical data with real-time control
• Thinking it updates devices
• Assuming it changes network settings

2. Which SQL query correctly selects temperature readings from sensor 'S1' recorded after '2024-01-01'?

easy

A. SELECT sensor_id, timestamp FROM readings WHERE type = 'temperature' OR sensor_id = 'S1' AND timestamp > '2024-01-01';

B. SELECT * FROM readings WHERE sensor_id = 'S1' AND timestamp < '2024-01-01' AND type = 'temperature';

C. SELECT * FROM readings WHERE sensor_id == 'S1' AND timestamp > '2024-01-01' AND type = 'temperature';

D. SELECT * FROM readings WHERE sensor_id = 'S1' AND timestamp > '2024-01-01' AND type = 'temperature';

Solution

1. Step 1: Check correct SQL syntax for conditions

   Use single equals (=) for comparison and AND to combine conditions.

2. Step 2: Verify logical conditions match requirements

   SELECT * FROM readings WHERE sensor_id = 'S1' AND timestamp > '2024-01-01' AND type = 'temperature'; correctly filters sensor_id = 'S1', timestamp > '2024-01-01', and type = 'temperature'.

3. Final Answer:

   SELECT * FROM readings WHERE sensor_id = 'S1' AND timestamp > '2024-01-01' AND type = 'temperature'; -> Option D

4. Quick Check:

   Correct syntax and filters = SELECT * FROM readings WHERE sensor_id = 'S1' AND timestamp > '2024-01-01' AND type = 'temperature'; [OK]

Hint: Use = for comparison and AND to combine filters [OK]

Common Mistakes:

• Using '==' instead of '=' in SQL
• Mixing AND and OR without parentheses
• Using wrong comparison operators
• Filtering with wrong timestamp direction

3. Given this query:
SELECT timestamp, value FROM readings WHERE sensor_id = 'S2' AND timestamp BETWEEN '2024-03-01' AND '2024-03-05' ORDER BY timestamp DESC LIMIT 3;
What will be the output?

medium

A. The 3 latest readings from sensor S2 between March 1 and 5, sorted descending

B. An error because LIMIT cannot be used with ORDER BY

C. All readings from sensor S2 between March 1 and 5, unsorted

D. The 3 earliest readings from sensor S2 between March 1 and 5, sorted ascending

Solution

1. Step 1: Understand the WHERE and BETWEEN clause

   Filters readings from sensor 'S2' between '2024-03-01' and '2024-03-05'.

2. Step 2: Analyze ORDER BY and LIMIT

   ORDER BY timestamp DESC sorts newest first; LIMIT 3 returns top 3 newest readings.

3. Final Answer:

   The 3 latest readings from sensor S2 between March 1 and 5, sorted descending -> Option A

4. Quick Check:

   ORDER BY DESC + LIMIT 3 = latest 3 readings [OK]

Hint: ORDER BY DESC + LIMIT gets newest records first [OK]

Common Mistakes:

• Confusing ascending vs descending order
• Thinking LIMIT returns earliest records
• Assuming LIMIT causes syntax error with ORDER BY

4. You wrote this query to get pressure data:
SELECT * FROM readings WHERE sensor_id = 'P1' AND timestamp > '2024-02-01' AND type = 'pressure'
But it returns no results, even though data exists. What is the likely problem?

medium

A. The query is missing a GROUP BY clause

B. The sensor_id should be numeric, not string

C. The timestamp format is incorrect and does not match stored data

D. The type column does not exist in the readings table

Solution

1. Step 1: Check timestamp format correctness

   Timestamp format must match stored data format exactly to filter correctly.

2. Step 2: Verify other query parts

   Sensor_id as string is valid; GROUP BY not needed; type column likely exists.

3. Final Answer:

   The timestamp format is incorrect and does not match stored data -> Option C

4. Quick Check:

   Timestamp format mismatch = no results [OK]

Hint: Match timestamp format exactly to stored data [OK]

Common Mistakes:

• Assuming sensor_id must be numeric
• Adding unnecessary GROUP BY
• Ignoring timestamp format differences

5. You want to find the average temperature for each of sensors 'T1' and 'T2' during January 2024, but only for readings above 20°C. Which SQL query achieves this?

hard

A. SELECT sensor_id, AVG(value) FROM readings WHERE (sensor_id = 'T1' OR sensor_id = 'T2') AND timestamp BETWEEN '2024-01-01' AND '2024-01-31' AND value > 20 AND type = 'temperature' GROUP BY sensor_id;

B. SELECT AVG(value) FROM readings WHERE sensor_id IN ('T1', 'T2') AND timestamp >= '2024-01-01' AND timestamp <= '2024-01-31' AND value > 20 AND type = 'temperature';

C. SELECT sensor_id, AVG(value) FROM readings WHERE sensor_id = 'T1' AND sensor_id = 'T2' AND timestamp BETWEEN '2024-01-01' AND '2024-01-31' AND value > 20 AND type = 'temperature' GROUP BY sensor_id;

D. SELECT sensor_id, AVG(value) FROM readings WHERE sensor_id = 'T1' OR sensor_id = 'T2' AND timestamp BETWEEN '2024-01-01' AND '2024-01-31' AND value > 20 AND type = 'temperature';

Solution

1. Step 1: Filter sensors correctly

   Use (sensor_id = 'T1' OR sensor_id = 'T2') or sensor_id IN ('T1', 'T2') to include both sensors.

2. Step 2: Apply date and value filters with grouping

   Filter timestamp between January 1 and 31, value > 20, and type = 'temperature'. Group by sensor_id to get averages per sensor.

3. Step 3: Check query correctness

   SELECT sensor_id, AVG(value) FROM readings WHERE (sensor_id = 'T1' OR sensor_id = 'T2') AND timestamp BETWEEN '2024-01-01' AND '2024-01-31' AND value > 20 AND type = 'temperature' GROUP BY sensor_id; uses correct syntax with parentheses and GROUP BY; SELECT AVG(value) FROM readings WHERE sensor_id IN ('T1', 'T2') AND timestamp >= '2024-01-01' AND timestamp <= '2024-01-31' AND value > 20 AND type = 'temperature'; misses GROUP BY; SELECT sensor_id, AVG(value) FROM readings WHERE sensor_id = 'T1' AND sensor_id = 'T2' AND timestamp BETWEEN '2024-01-01' AND '2024-01-31' AND value > 20 AND type = 'temperature' GROUP BY sensor_id; has impossible condition; SELECT sensor_id, AVG(value) FROM readings WHERE sensor_id = 'T1' OR sensor_id = 'T2' AND timestamp BETWEEN '2024-01-01' AND '2024-01-31' AND value > 20 AND type = 'temperature'; lacks parentheses causing wrong logic.

4. Final Answer:

   SELECT sensor_id, AVG(value) FROM readings WHERE (sensor_id = 'T1' OR sensor_id = 'T2') AND timestamp BETWEEN '2024-01-01' AND '2024-01-31' AND value > 20 AND type = 'temperature' GROUP BY sensor_id; -> Option A

5. Quick Check:

   Correct filters + grouping = SELECT sensor_id, AVG(value) FROM readings WHERE (sensor_id = 'T1' OR sensor_id = 'T2') AND timestamp BETWEEN '2024-01-01' AND '2024-01-31' AND value > 20 AND type = 'temperature' GROUP BY sensor_id; [OK]

Hint: Use parentheses for OR and GROUP BY for averages per sensor [OK]

Common Mistakes:

• Missing GROUP BY when aggregating by sensor
• Using AND instead of OR between sensor_ids
• Incorrect timestamp filtering logic