Patch-ID# 116466-08 NOTE: *********************************************************************** READ THE TERMS OF THE AGREEMENT ("AGREEMENT") IN THE LEGAL_LICENSE.TXT FILE CAREFULLY BEFORE USING THIS SOFTWARE. BY USING THE SOFTWARE, YOU AGREE TO THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL OF THE TERMS, PROMPTLY DESTROY THE UNUSED SOFTWARE. *********************************************************************** Keywords: sunwscm hme hmadm scm shutdown failover dsstat core scmadm sdbc mdb Synopsis: Availability Suite 3.2 CORE: Patch Date: Dec/22/2005 Install Requirements: Reboot after installation, an alternative may be in Special Install Instructions Install in Single User Mode Solaris Release: 8 9 SunOS Release: 5.8 5.9 Unbundled Product: Sun StorEdge Availability Suite Unbundled Release: 3.2 Xref: Topic: Relevant Architectures: sparc BugId's fixed with this patch: 2130820 4845102 4916708 4956742 4981411 4987621 5005765 5014282 5020256 5020577 5030355 5033048 5033106 5035196 5058411 5082142 5093983 5094206 5094694 6180565 6199262 6206679 6217688 6218008 6222650 6223102 6240841 6242272 6285730 6285819 6287498 6288855 6290572 6307873 Changes incorporated in this version: 2130820 6288855 Patches accumulated and obsoleted by this patch: Patches which conflict with this patch: Patches required with this patch: Obsoleted by: Files included with this patch: /etc/init.d/scm /etc/init.d/sv /etc/rc0.d/K78sv /etc/rc0.d/K84scm /etc/rc2.d/S002scm /etc/rc2.d/S00tsv /usr/kernel/drv/sdbc-5.8 /usr/kernel/drv/sdbc-5.9 /usr/kernel/drv/sparcv9/sdbc-5.8 /usr/kernel/drv/sparcv9/sdbc-5.9 /usr/lib/mdb/kvm/sdbc.so /usr/lib/mdb/kvm/sparcv9/sdbc.so /usr/opt/SUNWesm/SUNWscm/man/man1scm/dscfg.1m /usr/opt/SUNWesm/SUNWscm/man/man1scm/dsstat.1m /usr/opt/SUNWesm/SUNWscm/man/man1scm/scmadm.1m /usr/opt/SUNWesm/SUNWscm/sbin/dsstat /usr/opt/SUNWesm/SUNWscm/sbin/scmadm /usr/opt/SUNWesm/SUNWspsv/sbin/svboot /usr/opt/SUNWesm/cluster/lib/start/15sv /usr/opt/SUNWesm/cluster/lib/stop/10sv /usr/opt/SUNWesm/cluster/sbin/reconfig /usr/opt/SUNWesm/cluster/sbin/sv /usr/opt/SUNWesm/man/man1scm/dscfg.1m /usr/opt/SUNWscm/lib/libdscfg.so.1 /usr/opt/SUNWscm/lib/libunistat.so.1 /usr/opt/SUNWscm/lib/nskernd-5.8 /usr/opt/SUNWscm/lib/nskernd-5.9 /usr/opt/SUNWscm/sbin/dsbitmap-5.8 /usr/opt/SUNWscm/sbin/dsbitmap-5.9 /usr/opt/SUNWscm/sbin/dscfg /usr/opt/SUNWscm/sbin/dsclinfo Problem Description: 6288855 4 node cluster with heavy i/o load on the primary vols causes reference count maxed out WARNINGS 2130820 failover of greater than 64 Remote Mirror sets causes cluster deadlock condition (from 116466-07) 6307873 The change to use 'clinfo' instead of the Availability Suite 'dsclinfo' broke our SMF scripts 6290572 sndr devid limitation of 64 (NSC_MAXPATH constant) 6287498 Solaris 9 Vice-Master node caught in reboot loop when Fujitsu disk MAT3073NC is used 6285819 pgrep launched by NWS.reconfig deadlocks with processes mmapping on global at cluster failover 6285730 node hangs trying to come down after scshutdown is issued 6242272 raw kstat display data output from Availability Suite (sndr, ii, sv, sdbc), contains non-ASCII chars (from 116466-06) 6240841 sdbc fails to perform untimeout() during shutown and causes a panic 6223102 AVS3.2 latest patches on SC31u4: sndradm -P hang, ii_boot resume failed 6222650 writes in disk queue does not get applied to the secondary when in REP state 6218008 AS3.2 + latest patches: SNDR with disk queue + ndrii, scswitch -z -g hangs. (from 116466-05) 6217688 Updated P1 avail_suite/other Improved concurrency in reconfig script causes errors when run under SCATE 6206679 circlular locking hang between Availability Suite and SunCluster 6199262 reconfig script thinks II resource group is SNDR resource group 5094206 calling 'sndradm -I' w/out pre-creating ndr-ii in iiadm can lead to deadlock (from 116466-04) 5093983 After patching with 116466-01 ii cache can no longer be set >128MB 6180565 cache deconfigure does not clean up cache after a failed configure (from 116466-03) 5082142 dscfg -s must assure RAW device is specified 5094694 cfg_open() leaves stale info after failing to open block device (from 116466-02) 4845102 [RFE] dsstat should be able to report if PIT volume(s) are offline for a set 4916708 need to allow other SC resources in the light weight RG 5020256 dsstat -m ii - failed when II set was disabled 5020577 the -n switch should be removed as a valid dscfg command 5030355 dsstat manpage needs to include sndr volume status for queuing 5033048 scmadm -e's NOTICE: message to console and /var/adm/messages are misleading 5033106 entering the default for dscfg -i command is an invalid option 5035196 reconfig script needs to take RT versioning into consideration 5058411 dscfg does not log to ds.log (from 116466-01) 5014282 dsstat should report sndr volume status as queuing 5005765 tuning cache size larger then cache available hangs system 4987621 ::sdbc_vars mdb dcmd prints out wrong values 4956742 3.2 port needed of 3.1 dynamic memory allocation fixes 4981411 reconfig script needs to take RT versioning into consideration Patch Installation Instructions: -------------------------------- For Solaris 7-8 releases, refer to the man pages for instructions on using 'patchadd' and 'patchrm' scripts provided with Solaris. Any other special or non-generic installation instructions should be described below as special instructions. The following example installs a patch to a standalone machine: example# patchadd /var/spool/patch/104945-02 The following example removes a patch from a standalone system: example# patchrm 104945-02 For additional examples please see the appropriate man pages. Special Install Instructions: ----------------------------- Since this patch updates modules that live in the kernel, it is necessary for the user to boot the system up in single user mode to apply the patch and then reboot the system. Availability Suite is enhanced by this patch to operate effectively in an SNDR environment using disk queues, where the configured volume exhibits a high rate of change as a result of a "hot-spot" on one or more blocks of a replicated volume set. Through the use of a resized SNDR bitmap volume, performance issues pertinent to the "hot-spot" are avoided. Details follow. Availability Suite's Remote Mirror Copy software uses a portion of each bitmap volume configured to maintain a per-chunk reference counter. This functionality is required so that multiple revisions of the same disk block or blocks within a 32KB chunk can be queued up for replication, an architectural requirement necessary to maintain write-order consistency between configured SNDR replicas. With the introduction of disk queues in Availability Suite Version 3.2, there is now the potential for tens to hundreds of instances of the same data block or blocks within a 32KB chunk to be queued up for replication. A typical scenario where this is being seen is during a database reload, or complex update operation, where multiple sections of the database experience "hot-spots", or places where large numbers of I/O writes occur to the same disk block or blocks in a relatively short period of time. The behavior is not limited to any database, file system, or application I/O, all which can exhibit "hot-spots" of activity in a configured replicated volume. Prior to this patch, the per-chunk reference counter was maintained in a single byte (8-bits), allowing from 1 to 255 instances of the same block or blocks to be queued up for replication. With memory based queues and early usage of disk based queues, it was presumed that the 255 limit was sufficient, and that reaching this value would only be a momentary, infrequent occurrence. For certain customers and installations using Availability Suite Remote Copy Version 3.2, this is not the case, and the behavior causes poor application performance, which may be viewed as a system hang, along with the following message being seen in /var/adm/messages, from tens to tens-of-thousands of times. WARNING: SNDR: bitmap reference count maxed out for During periods of seeing this message, I/O write operations are being delayed, as long as replication is still active and there is room in the disk queue for other I/Os needed to maintain write order consistency. The problem with these "hot spot" I/Os being "delayed", is that database application writes are now occurring at network I/O replication rates, not disk I/O rates, causing the database, file system or application to appear to be hung or slow. This is further problematic since these "hot spots" are assumed to be a "commit write", meaning that all database, file system or application I/O is serialized and now delayed while this single write I/O waits for the per chunk reference counter to be less then 255 maximum value. This patch resolves this problem by detecting the ability to increasing the reference counter from an 8-bit byte value, to a 32-bit value, allowing the maximum value to go from 255 to 4,294,967,295, a value while is presently unreachable with Remote Mirroring 3.2 software. Each time a Remote Mirror replication set transitions from logging mode to replicating mode, if it has not already done so, it will determine if the current bitmap volume is sized large enough to use 32-bit reference counters, verses 8-bit reference counters, and use the larger size if possible and if only when disk queues are configured. There are various means in which to allow this larger size volume (as determine by the updated dsbitmap utility) to be detected by the Remote Mirror software: 1). The current bitmap volume is already sized large enough based on prior rounding up, or oversizing of the original bitmap volume at its initial configuration time. 2). While in logging mode the bitmap volume is replaced with a larger volume, followed by the use of the SNDR "reconfigure bitmap" commands as follows: sndradm [opts] -R b p [set] reconfig primary bitmap 3). While in logging mode the bitmap volume is resized and then replaced with the "reconfigure bitmap" commands as follows: sndradm [opts] -R b p [set] reconfig primary bitmap 4). At enable time, the bitmap volume is sized based on the "disk queue" sizing as reported by the 'dsbitmap' utility, which has been enhanced to indicate the two sizes of Remote Mirror bitmap volumes to configure, one without disk queues, one with disk queues. For installations unaffected by the bitmap reference count maxed out issue, there is no need to increase their current bitmap volume size, even if using disk queues. Once the patch has been applied to the system and the affected SNDR replicas have their bitmap volumes resized, the bitmap reference count maxed out problem will no longer exist for those volumes. In view of the above changes, be aware of the fact that if the maxed reference count limit is no longer a restriction, then the next limit is an SNDR disk queue full condition. In this case the disk queue operational modes of blocking or non-blocking are relevant. If blocking mode is selected and the disk queue becomes full, then the same performance issues with the bitmap reference count maxed out exists. If non-blocking disk queues are enabled, he SNDR set will automatically drop into logging mode once the disk queue full condition is reached. README -- Last modified date: Thursday, December 22, 2005