| Applies To: |
|
| Summary: |
| When a redundant trend server is started,
it automatically backfills its trend history files with any data
that it missed while it was down. How does this process work? |
| Solution: |
| If one trend server is shut down, it will
miss the recording of samples during that time. When it is
restarted, it checks the time of the last samples it had recorded.
It then sends a request to the other trend server for all samples
newer than that, and adds them to its own history files.
CitectSCADA 5.42 SP A, 5.50 SP A and later include additional
backfilling features: Trend redundancy reconnection including heartbeat — previously only one attempt was made at startup Startup correctly backfills no matter what order the servers start — previously, only the last server to start up would backfill Gaps without network loss — when network outages occur, it traps the trend gaps and requests a backfill Gaps with network loss — when a network connection is lost, it traps the trend gaps and requests a backfill TrnSetTable — writes to both servers. A Cicode error occurs if redundancy is enabled and the redundant write did not occur on the other server. The performance and reliability of runtime backfilling has been improved significantly in CitectSCADA 6.00.and again in 6.10 Requirements for Backfilling "Trends Server supports redundancy" must be ticked in the Computer Setup Wizard on both trend servers "This Server name" and "Other Server name" must be entered in the Computer Setup Wizard Two trend servers must be in use One trend server must already be running while the other starts up (no longer required in Citect 5.42 SP A, 5.50 SP A and later). Viewing Connection Statistics Type PAGE TABLE TRAN <Enter> in the Kernel to view the connection to the other trend server. The line that begins with "Trend SerRnd" shows the statistics for the connection from this trend server to the other trend server (e.g. message handle, packets sent and received, LAN stack no.). The server requesting backfilling data should have a hMsg value >= 0. This indicates that a network session has been established. The server providing the data normally shows -1 (no connection) for hMsg as it does not need to create a session to the other server. To view backfilling status in CitectSCADA versions 6.00 and later type PAGE QUEUE <Enter> in the Kernel. Press <Page Down> until the one of the following pages is displayed: TrnRdn.GapFillDelayQueue — outstanding work to process TrnRdn.GapFillSentQueue — items awaiting a response from the redundant server To view backfilling status CitectSCADA 5.50r0, 5.42r0 and previous versions check the Main window in the Kernel. The following messages will appear on the trend server that is requesting data: Trend Redundancy: Data request started hh:mm:ss Trend Redundancy: Samples requested from hh:mm:ss Trend Redundancy: Data request stopped hh:mm:ss The "Samples requested from" message shows the time of the oldest sample that is being synchronized. If this server has no data, the message "Trend Redundancy: Requesting all possible samples" will display instead. Do not shut down either trend server until "Data request stopped" message appears, indicating that backfilling has finished. Otherwise, it will be necessary to copy the history files manually. Runtime Backfilling Tuning CitectSCADA 5.42 Service Pack A, 5.50 SP A and higher support runtime backfilling. If necessary, performance can be tuned by setting these citect.ini parameters: [Trend]EnableBackfill — Enables or disables backfilling [Trend]DisableRuntimeBackfill — Set to 1 to disable runtime backfilling. It will still backfill once upon connection (5.50 SP C only) [Trend]BytesReadBeforeSleep — Allows more data to be read from the trend archive before the trend server has to sleep [Trend]BytesWrittenBeforeSleep — Allows more data to be written to the trend archive before the trend server has to sleep [Trend]MaxBackfillsAtOnce — Determines the number of trends to be requested for backfill at a time [Trend]MaxBackfillsAtOnce — Determines the number of the trend to be requested for backfilling at a time [Trend]WriteWatchTime — The trend write task writes out trend data to disk, and then sleeps for the time specified in this parameter (before writing the next trend file) [Trend]MaxRdnSamples — Determines the maximum number of samples to be requested for a trend at a time [Trend]MaxSetTableStnbyPending — Controls the maximum number of pending TrnSetTable requests to the standby trend server [Trend]MaxSetTableQue — The maximum number of TrnSetTable requests on the primary trend server allowed when these must be transferred to the standby server [Trend]GapFillTime or GapFillSamples — Sets the threshold between filling gaps with calculated values and backfilling [Trend]TrendSetErrorHandling — If set to 1 TrnSetTable will return a negative number of samples if they were written only to the primary server [Trend]ClientRequestTime — Determines the maximum time the client will wait on TrnSetTable() before causing an error [Trend]TrendDebug — Logs additional backfilling information (may cause increased CPU and disk usage). See the CitectSCADA 6.00 help topic “[Trend]TrendDebug parameter” First Time Startup Of Redundant Servers 1. Start up one trend server and it will create the history files 2. Copy the trend history files (*.hst and *.000 to *.999) to the other trend server. (See Copying Trend Files Without Shutting Down, below) 3. Start up the other trend server 4. After starting up, it will backfill its trend files from the time the first server had last written to them If the first server had not yet logged any data to the files, you may ignore the message: "Cannot find files of trend 'trendtagname'". The reason for the message is that it expects the copied files to have data in them, but it may be a number of minutes before the first server's cache fills up and causes it write to the history files. This message can be disabled by setting the citect.ini parameter [Trend]CopyFilesMessage=0 Adding Trends To A Running System 1. Shut down one trend server 2. Add the trend tag definitions to its project 3. Recompile the project 4. Start the Citect runtime Citect will create the (empty) trend history files. It will attempt to backfill them with data from the other trend server. Since they do not yet exist on the other server, the "Data request stopped" message may not display. Instead, the following message will appear in the syslog.dat file (to view it in the Kernel choose View menu | SysLog): The Trend 'trendname' Does Not Exist On The Other Trend Server 'otherclustername' 5. After backfilling is complete, shut down the other trend server 6. Add the trend tag definitions to its project 7. Copy the new trend tag's history files from the other server. 8. Recompile the project 9. Start the Citect runtime 10. Backfilling should occur. 11. Restart any Citect Display or Manager clients so the project changes take effect If the first server had not yet logged any data to the files, the popup message: "Cannot find files of trend 'trendtagname'" may be ignored. If newly created history files are copied to the other server before they have been logged to, on startup it will say they have to be copied manually (which has just been done). The message can be ignored. Display a trend page on both servers, to verify that they are in sync and are logging. The reason for the error is that Citect expects the copied files to contain data, but it may be a number of minutes before the first server's cache fills up and causes it write to the history files (see Q3941). Deleting Trends From A Running System 1. Shut down one trend server 2. Delete the trend tag(s) from the Citect project 3. Delete the corresponding history files from this trend server 4. Recompile the project 5. Start the Citect runtime 6. Shut down the other trend server 7. Repeat steps 2-5 for this PC 8. Restart any Citect Display or Manager clients so the project changes take effect Restoring Trends After A Hard Drive Failure 1. Copy the trend history files from the other trend server. (See Copying Trend Files Without Shutting Down, below) 2. Start up this trend server 3. After starting up, it will backfill its trend files from the time the first server had last written to them Restoring Trends After A Network Disconnection If one trend server loses its network connection to the I/O server, it will record sample values as "<NA>" until the connection is restored. Versions that support runtime backfilling will handle this. Older versions won’t trigger backfilling since the server was still running. To correct this: 1. Shut down the trend server that is missing data 2. Copy the trend history files from the other trend server. See the topic: Copying Trend Files Without Shutting Down, below 3. Start the Citect runtime 4. Backfilling should occur If multiple outages have occurred, some data may only exist on one server, and other data may only exist on the other server. Runtime backfilling support is required to synchronize both servers again. Copying Trend Files Without Shutting Down If it is necessary to copy trend files without shutting down the trend server they must be copied in the correct order because they may be updated by Citect during the copy process. Copy all the .HST files first, then the numbered files .001 to .999. This will keep the trend data in a valid state (see Q1265). |
| Keywords: |
Related Links
Attachments