Applies To:
  • CitectSCADA 3.xx, 4.xx, 5.xx, 6.xx
  • CitectHMI 3.xx, 4.xx, 5.xx, 6.xx

Summary:
When a redundant trend server is started, it automatically backfills its trend history files with any data that it missed while it was down. How does this process work?
 

Solution:
If one trend server is shut down, it will miss the recording of samples during that time. When it is restarted, it checks the time of the last samples it had recorded. It then sends a request to the other trend server for all samples newer than that, and adds them to its own history files. CitectSCADA 5.42 SP A, 5.50 SP A and later include additional backfilling features:

Trend redundancy reconnection including heartbeat — previously only one attempt was made at startup
Startup correctly backfills no matter what order the servers start — previously, only the last server to start up would backfill
Gaps without network loss — when network outages occur, it traps the trend gaps and requests a backfill
Gaps with network loss — when a network connection is lost, it traps the trend gaps and requests a backfill
TrnSetTable — writes to both servers. A Cicode error occurs if redundancy is enabled and the redundant write did not occur on the other server.

The performance and reliability of runtime backfilling has been improved significantly in CitectSCADA 6.00.and again in 6.10

Requirements for Backfilling
"Trends Server supports redundancy" must be ticked in the Computer Setup Wizard on both trend servers
"This Server name" and "Other Server name" must be entered in the Computer Setup Wizard
Two trend servers must be in use
One trend server must already be running while the other starts up (no longer required in Citect 5.42 SP A, 5.50 SP A and later).

Viewing Connection Statistics
Type PAGE TABLE TRAN <Enter> in the Kernel to view the connection to the other trend server. The line that begins with "Trend SerRnd" shows the statistics for the connection from this trend server to the other trend server (e.g. message handle, packets sent and received, LAN stack no.). The server requesting backfilling data should have a hMsg value >= 0. This indicates that a network session has been established. The server providing the data normally shows -1 (no connection) for hMsg as it does not need to create a session to the other server.
To view backfilling status in CitectSCADA versions 6.00 and later type PAGE QUEUE <Enter> in the Kernel. Press <Page Down> until the one of the following pages is displayed:

TrnRdn.GapFillDelayQueue — outstanding work to process
TrnRdn.GapFillSentQueue — items awaiting a response from the redundant server

To view backfilling status CitectSCADA 5.50r0, 5.42r0 and previous versions check the Main window in the Kernel. The following messages will appear on the trend server that is requesting data:

Trend Redundancy: Data request started hh:mm:ss
Trend Redundancy: Samples requested from hh:mm:ss
Trend Redundancy: Data request stopped hh:mm:ss

The "Samples requested from" message shows the time of the oldest sample that is being synchronized. If this server has no data, the message "Trend Redundancy: Requesting all possible samples" will display instead.

Do not shut down either trend server until "Data request stopped" message appears, indicating that backfilling has finished. Otherwise, it will be necessary to copy the history files manually.

Runtime Backfilling Tuning
CitectSCADA 5.42 Service Pack A, 5.50 SP A and higher support runtime backfilling. If necessary, performance can be tuned by setting these citect.ini parameters:

[Trend]EnableBackfill — Enables or disables backfilling
[Trend]DisableRuntimeBackfill — Set to 1 to disable runtime backfilling. It will still backfill once upon connection (5.50 SP C only)
[Trend]BytesReadBeforeSleep — Allows more data to be read from the trend archive before the trend server has to sleep
[Trend]BytesWrittenBeforeSleep — Allows more data to be written to the trend archive before the trend server has to sleep
[Trend]MaxBackfillsAtOnce — Determines the number of trends to be requested for backfill at a time
[Trend]MaxBackfillsAtOnce — Determines the number of the trend to be requested for backfilling at a time
[Trend]WriteWatchTime — The trend write task writes out trend data to disk, and then sleeps for the time specified in this parameter (before writing the next trend file)
[Trend]MaxRdnSamples — Determines the maximum number of samples to be requested for a trend at a time
[Trend]MaxSetTableStnbyPending — Controls the maximum number of pending TrnSetTable requests to the standby trend server
[Trend]MaxSetTableQue — The maximum number of TrnSetTable requests on the primary trend server allowed when these must be transferred to the standby server
[Trend]GapFillTime or GapFillSamples — Sets the threshold between filling gaps with calculated values and backfilling
[Trend]TrendSetErrorHandling — If set to 1 TrnSetTable will return a negative number of samples if they were written only to the primary server
[Trend]ClientRequestTime — Determines the maximum time the client will wait on TrnSetTable() before causing an error
[Trend]TrendDebug — Logs additional backfilling information (may cause increased CPU and disk usage). See the CitectSCADA 6.00 help topic “[Trend]TrendDebug parameter”

First Time Startup Of Redundant Servers
1. Start up one trend server and it will create the history files
2. Copy the trend history files (*.hst and *.000 to *.999) to the other trend server. (See Copying Trend Files Without Shutting Down, below)
3. Start up the other trend server
4. After starting up, it will backfill its trend files from the time the first server had last written to them

If the first server had not yet logged any data to the files, you may ignore the message: "Cannot find files of trend 'trendtagname'". The reason for the message is that it expects the copied files to have data in them, but it may be a number of minutes before the first server's cache fills up and causes it write to the history files. This message can be disabled by setting the citect.ini parameter [Trend]CopyFilesMessage=0

Adding Trends To A Running System
1. Shut down one trend server
2. Add the trend tag definitions to its project
3. Recompile the project
4. Start the Citect runtime

Citect will create the (empty) trend history files. It will attempt to backfill them with data from the other trend server. Since they do not yet exist on the other server, the "Data request stopped" message may not display. Instead, the following message will appear in the syslog.dat file (to view it in the Kernel choose View menu | SysLog):

The Trend 'trendname' Does Not Exist On The Other Trend Server 'otherclustername'
5. After backfilling is complete, shut down the other trend server
6. Add the trend tag definitions to its project
7. Copy the new trend tag's history files from the other server.
8. Recompile the project
9. Start the Citect runtime
10. Backfilling should occur.
11. Restart any Citect Display or Manager clients so the project changes take effect

If the first server had not yet logged any data to the files, the popup message: "Cannot find files of trend 'trendtagname'" may be ignored.

If newly created history files are copied to the other server before they have been logged to, on startup it will say they have to be copied manually (which has just been done). The message can be ignored. Display a trend page on both servers, to verify that they are in sync and are logging. The reason for the error is that Citect expects the copied files to contain data, but it may be a number of minutes before the first server's cache fills up and causes it write to the history files (see Q3941).

Deleting Trends From A Running System
1. Shut down one trend server
2. Delete the trend tag(s) from the Citect project
3. Delete the corresponding history files from this trend server
4. Recompile the project
5. Start the Citect runtime
6. Shut down the other trend server
7. Repeat steps 2-5 for this PC
8. Restart any Citect Display or Manager clients so the project changes take effect

Restoring Trends After A Hard Drive Failure
1. Copy the trend history files from the other trend server. (See Copying Trend Files Without Shutting Down, below)
2. Start up this trend server
3. After starting up, it will backfill its trend files from the time the first server had last written to them

Restoring Trends After A Network Disconnection
If one trend server loses its network connection to the I/O server, it will record sample values as "<NA>" until the connection is restored. Versions that support runtime backfilling will handle this. Older versions won’t trigger backfilling since the server was still running. To correct this:
1. Shut down the trend server that is missing data
2. Copy the trend history files from the other trend server. See the topic: Copying Trend Files Without Shutting Down, below
3. Start the Citect runtime
4. Backfilling should occur

If multiple outages have occurred, some data may only exist on one server, and other data may only exist on the other server. Runtime backfilling support is required to synchronize both servers again.

Copying Trend Files Without Shutting Down
If it is necessary to copy trend files without shutting down the trend server they must be copied in the correct order because they may be updated by Citect during the copy process. Copy all the .HST files first, then the numbered files .001 to .999. This will keep the trend data in a valid state (see Q1265).
 

Keywords:
 

Attachments