The views expressed on this blog are my own and do not necessarily reflect the views of Oracle

March 30, 2014

ASM spfile in a disk group

Starting with ASM version 11.2, the ASM spfile can be stored in an ASM disk group. Indeed, during a new ASM installation, the Oracle Universal Installer (OUI) will place the ASM spfile in the disk group that gets created during the installation. This is true for both Oracle Restart (single instance environments) and Cluster installations.

The ASM spfile stored in the disk group is a registry file, and will always be the ASM metadata file number 253.

New ASMCMD commands

To support this feature, new ASMCMD commands were introduced to back up, copy and move the ASM spfile. The commands are:
  • spbackup - backs up an ASM spfile to a backup file. The backup file is not a special file type and is not identified as an spfile.
  • spcopy - copies an ASM spfile from the source location to an spfile in the destination location.
  • spmove - moves an ASM spfile from source to destination and automatically updates the GPnP profile.

The SQL commands CREATE PFILE FROM SPFILE and CREATE SPFILE FROM PFILE are still valid for the ASM spfile stored in the disk group.

ASM spfile in disk group DATA

In my environment, the ASM spfile is (somewhere) in disk group DATA. Let's find it:

$ asmcmd find --type ASMPARAMETERFILE +DATA "*"
+DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.822856169

As we can see, the ASM spfile is in a special location and it has ASM file number 253.

Of course, we see the same thing from sqlplus:

$ sqlplus / as sysasm

SQL> show parameter spfile

NAME   TYPE   VALUE
------ ------ -------------------------------------------------
spfile string +DATA/ASM/ASMPARAMETERFILE/registry.253.822856169

SQL>

Let's make a backup of that ASM spfile.

$ asmcmd spbackup +DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.822856169 /tmp/ASMspfile.backup

And check out the contents of the file:

$ strings /tmp/ASMspfile.backup
+ASM.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value
+ASM.asm_diskgroups='RECO','ACFS'#Manual Mount
*.asm_power_limit=1
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'

As we can see, this is a copy of the ASM spfile, that includes the parameters and associated comments.

ASM spfile discovery

So, how can the ASM instance read the spfile on startup, if the spfile is in a disk group that is not even mounted yet? Not only that - the ASM doesn't really know which disk group has the spfile and even if the spfile exists.

The trick is in the ASM disk headers. To support the ASM spfile in a disk group, two new fields were added to the ASM disk header:
  • kfdhdb.spfile - Allocation unit number of the ASM spfile.
  • kfdhdb.spfflg - ASM spfile flag. If this value is 1, the ASM spfile is on this disk in allocation unit kfdhdb.spfile.

When the ASM instance starts, it reads the disk headers and looks for the spfile information. Once it finds the disks that have the spfile, it can read the actual initialization parameters.

Let's have a look at my disk group DATA. First check the disk group state and redundancy

$ asmcmd lsdg -g DATA | cut -c1-26
Inst_ID  State    Type
     1  MOUNTED  NORMAL

The disk group is mounted and it's normal redundancy. This means the ASM spfile will be mirrored, so we should see two disks with kfdhdb.spfile and kfdhdb.spfflg values set. Let's have a look:

$ for disk in `asmcmd lsdsk -G DATA --suppressheader`
> do
> echo $disk
> kfed read $disk | grep spf
> done
/dev/sdc1
kfdhdb.spfile:                       46 ; 0x0f4: 0x0000002e
kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001
/dev/sdd1
kfdhdb.spfile:                     2212 ; 0x0f4: 0x000008a4
kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001
/dev/sde1
kfdhdb.spfile:                        0 ; 0x0f4: 0x00000000
kfdhdb.spfflg:                        0 ; 0x0f8: 0x00000000

As we can see, two disks have the ASM spfile. Let's check the contents of the Allocation Unit 46 on disk /dev/sdc1:

$ dd if=/dev/sdc1 bs=1048576 skip=46 count=1 | strings
+ASM.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value
+ASM.asm_diskgroups='RECO','ACFS'#Manual Mount
*.asm_power_limit=1
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0352732 s, 29.7 MB/s

As we can see, the AU 46 on disk /dev/sdc1 indeed contains the ASM spfile.

ASM spfile alias block

In addition to the new ASM disk header fields, there is a new metadata block type - KFBTYP_ASMSPFALS - that describes the ASM spfile alias. The ASM spfile alias block will be the last block in the ASM spfile.

Let's have a look at the last block of the Allocation Unit 46:

$ kfed read /dev/sdc1 aun=46 blkn=255
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           27 ; 0x002: KFBTYP_ASMSPFALS
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                     255 ; 0x004: blk=255
kfbh.block.obj:                     253 ; 0x008: file=253
kfbh.check:                   806373865 ; 0x00c: 0x301049e9
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfspbals.incarn:              822856169 ; 0x000: 0x310bc9e9
kfspbals.blksz:                     512 ; 0x004: 0x00000200
kfspbals.size:                        3 ; 0x008: 0x0003
kfspbals.path.len:                    0 ; 0x00a: 0x0000
kfspbals.path.buf:                      ; 0x00c: length=0

There is not much in this metadata block. Most of the entries have the block header info (fields kfbh.*). The actual ASM spfile alias data (fields kfspbals.*) has only few entries. The spfile file incarnation (822856169) is part of the file name (REGISTRY.253.822856169), the block size is 512 (bytes) and the file size is 3 blocks. The path info is empty, meaning I don't actually have the ASM spfile alias.

Let's create one. I will first create a pfile from the existing spfile and then create the spfile alias from that pfile.

$ sqlplus / as sysasm

SQL> create pfile='/tmp/pfile+ASM.ora' from spfile;

File created.

SQL> shutdown abort;
ASM instance shutdown

SQL> startup pfile='/tmp/pfile+ASM.ora';
ASM instance started

Total System Global Area 1135747072 bytes
Fixed Size                  2297344 bytes
Variable Size            1108283904 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted

SQL> create spfile='+DATA/spfileASM.ora' from pfile='/tmp/pfile+ASM.ora';

File created.

SQL> exit

Looking for the ASM spfile again shows two entries:

$ asmcmd find --type ASMPARAMETERFILE +DATA "*"
+DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.843597139
+DATA/spfileASM.ora

We now see the ASM spfile itself (REGISTRY.253.843597139) and its alias (spfileASM.ora). Having a closer look at spfileASM.ora confirms this is indeed the alias for the registry file:

$ asmcmd ls -l +DATA/spfileASM.ora
Type              Redund  Striped  Time             Sys  Name
ASMPARAMETERFILE  MIRROR  COARSE   MAR 30 20:00:00  N    spfileASM.ora => +DATA/ASM/ASMPARAMETERFILE/REGISTRY.253.843597139

Check the ASM spfile alias block now:

$ kfed read /dev/sdc1 aun=46 blkn=255
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           27 ; 0x002: KFBTYP_ASMSPFALS
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                     255 ; 0x004: blk=255
kfbh.block.obj:                     253 ; 0x008: file=253
kfbh.check:                  2065104480 ; 0x00c: 0x7b16fe60
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfspbals.incarn:              843597139 ; 0x000: 0x32484553
kfspbals.blksz:                     512 ; 0x004: 0x00000200
kfspbals.size:                        3 ; 0x008: 0x0003
kfspbals.path.len:                   13 ; 0x00a: 0x000d
kfspbals.path.buf:        spfileASM.ora ; 0x00c: length=13

Now we see that the alias file name now appears in the ASM spfile alias block. Note the new incarnation number, as this is a new ASM spfile, created from the pfile.

Conclusion

Starting with ASM version 11.2, the ASM spfile can be stored in an ASM disk group. To support this feature, we now have new ASMCMD commands and, under the covers, we have new ASM metadata structures.

February 18, 2014

ACFS disk group rebalance


Starting with Oracle Database version 11.2, an ASM disk group can be used for hosting one or more cluster file systems. These are known as Oracle ASM Cluster File Systems or Oracle ACFS. This functionality is achieved by creating special volume files inside the ASM disk group, which are then exposed to the OS as the block devices. The file systems are then created on those block devices.

This post is about the rebalancing, mirroring and extent management of the ACFS volume files.

The environment used for the examples:
* 64-bit Oracle Linux 5.4, in Oracle Virtual Box
* Oracle Restart and ASM version 11.2.0.3.0 - 64bit
* ASMLib/oracleasm version 2.1.7

Set up ACFS volumes

As this is an Oracle Restart environment (single instance), I have to load ADVM/ACFS drivers manually (as root user).

# acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9322: completed
#

Create a disk group to hold ASM cluster file systems.

$ sqlplus / as sysasm

SQL> create diskgroup ACFS
disk 'ORCL:ASMDISK5', 'ORCL:ASMDISK6'
attribute 'COMPATIBLE.ASM' = '11.2', 'COMPATIBLE.ADVM' = '11.2';

Diskgroup created.

SQL>

While it is possible and supported to have a disk group that holds database files and ACFS volume files, I recommend to have a separate disk group for ACFS volumes. This provides the role/function separation and potential performance benefits to database files.

Check the allocation unit (AU) sizes for all disk groups.

SQL> select group_number "Group#", name "Name", allocation_unit_size "AU size"
from v$asm_diskgroup_stat;

   Group# Name        AU size
---------- -------- ----------
        1 ACFS        1048576
        2 DATA        1048576

SQL>

Note the default AU size (1MB) for both disk groups. I will refer to this later on, when I talk about the extent sizes for the volume files.

Create some volumes in disk group ACFS.

$ asmcmd volcreate -G ACFS -s 4G VOL1
$ asmcmd volcreate -G ACFS -s 2G VOL2
$ asmcmd volcreate -G ACFS -s 1G VOL3

Get the volume info.

$ asmcmd volinfo -a
Diskgroup Name: ACFS

Volume Name: VOL1
Volume Device: /dev/asm/vol1-142
State: ENABLED
Size (MB): 4096
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

Volume Name: VOL2
Volume Device: /dev/asm/vol2-142
State: ENABLED
Size (MB): 2048
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

Volume Name: VOL3
Volume Device: /dev/asm/vol3-142
State: ENABLED
Size (MB): 1024
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

$

Note that the volumes are automatically enabled after creation. On (server) restart we would need to manually load ADVM/ACFS drivers (acfsload start) and enable the volumes (asmcmd volenable -a).

ASM files for ACFS support

For each volume, the ASM creates a volume file. In a redundant disk group, each volume will also have a dirty region logging (DRL) file.

Get some info about our volume files.

SQL> select file_number "File#", volume_name "Volume", volume_device "Device", size_mb "MB", drl_file_number "DRL#"
from v$asm_volume;

File# Volume Device               MB DRL#
------ ------ ----------------- ----- ----
  256 VOL1   /dev/asm/vol1-142  4096  257
  259 VOL2   /dev/asm/vol2-142  2048  258
  261 VOL3   /dev/asm/vol3-142  1024  260

SQL>

In addition to volume names, device names and sizes, this shows ASM files numbers 256, 259 and 261 for volume devices, and ASM file numbers 257, 258 and 260 for the associated DRL files.

Volume file extents

Get the extent distribution info for one of the volume files.

SQL> select xnum_kffxp "Extent", au_kffxp "AU", disk_kffxp "Disk"
from x$kffxp
where group_kffxp=2 and number_kffxp=261
order by 1,2;

  Extent         AU       Disk
---------- ---------- ----------
        0       6256          0
        0       6256          1
        1       6264          0
        1       6264          1
        2       6272          1
        2       6272          0
        3       6280          0
        3       6280          1
...
      127       7272          0
      127       7272          1
2147483648       6252          0
2147483648       6252          1
2147483648 4294967294      65534

259 rows selected.

SQL>

First thing to note is that each extent is mirrored, as the volume is in a normal redundancy disk group.

We also see that the volume file 261 has 128 extents. As the volume size is 1GB, that means each extent size is 8MB or 8 AUs. The point here is that the volume files have their own extent size, unlike the standard ASM files that inherit the (initial) extent size from the disk group AU size.

ASM based cluster file systems

We can now use the volumes to create ASM cluster file systems and let everyone use them (this needs to be done as root user, of course):

# mkdir /acfs1
# mkdir /acfs2
# mkdir /acfs3

# chmod 777 /acfs?

# /sbin/mkfs -t acfs /dev/asm/vol1-142
mkfs.acfs: version         = 11.2.0.3.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume          = /dev/asm/vol1-142
mkfs.acfs: volume size     = 4294967296
mkfs.acfs: Format complete.

# /sbin/mkfs -t acfs /dev/asm/vol2-142
mkfs.acfs: version         = 11.2.0.3.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume          = /dev/asm/vol2-142
mkfs.acfs: volume size     = 2147483648
mkfs.acfs: Format complete.

# /sbin/mkfs -t acfs /dev/asm/vol3-142
mkfs.acfs: version         = 11.2.0.3.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume          = /dev/asm/vol3-142
mkfs.acfs: volume size     = 1073741824
mkfs.acfs: Format complete.

# mount -t acfs /dev/asm/vol1-142 /acfs1
# mount -t acfs /dev/asm/vol2-142 /acfs2
# mount -t acfs /dev/asm/vol3-142 /acfs3

# mount | grep acfs
/dev/asm/vol1-142 on /acfs1 type acfs (rw)
/dev/asm/vol2-142 on /acfs2 type acfs (rw)
/dev/asm/vol3-142 on /acfs3 type acfs (rw)

Copy some files into the new file systems.

$ cp diag/asm/+asm/+ASM/trace/* /acfs1
$ cp diag/rdbms/db/DB/trace/* /acfs1
$ cp oradata/DB/datafile/* /acfs1

$ cp diag/asm/+asm/+ASM/trace/* /acfs2
$ cp oradata/DB/datafile/* /acfs2

$ cp fra/DB/backupset/* /acfs3

Check the used space.

$ df -h /acfs?
Filesystem         Size  Used Avail Use% Mounted on
/dev/asm/vol1-142  4.0G  1.3G  2.8G  31% /acfs1
/dev/asm/vol2-142  2.0G  1.3G  797M  62% /acfs2
/dev/asm/vol3-142  1.0G  577M  448M  57% /acfs3

ACFS disk group rebalance

Let's add one disk to the ACFS disk group and monitor the rebalance operation.

SQL> alter diskgroup ACFS add disk 'ORCL:ASMDISK4';

Diskgroup altered.

SQL>

Get the ARB0 PID from the ASM alert log.

$ tail alert_+ASM.log
Sat Feb 15 12:44:53 2014
SQL> alter diskgroup ACFS add disk 'ORCL:ASMDISK4'
NOTE: Assigning number (2,2) to disk (ORCL:ASMDISK4)
...
NOTE: starting rebalance of group 2/0x80486fe8 (ACFS) at power 1
SUCCESS: alter diskgroup ACFS add disk 'ORCL:ASMDISK4'
Starting background process ARB0
Sat Feb 15 12:45:00 2014
ARB0 started with pid=27, OS id=10767
...

And monitor the rebalance by tailing the ARB0 trace file.

$ tail -f ./+ASM_arb0_10767.trc

*** ACTION NAME:() 2014-02-15 12:45:00.151
ARB0 relocating file +ACFS.1.1 (2 entries)
ARB0 relocating file +ACFS.2.1 (1 entries)
ARB0 relocating file +ACFS.3.1 (42 entries)
ARB0 relocating file +ACFS.3.1 (1 entries)
ARB0 relocating file +ACFS.4.1 (2 entries)
ARB0 relocating file +ACFS.5.1 (1 entries)
ARB0 relocating file +ACFS.6.1 (1 entries)
ARB0 relocating file +ACFS.7.1 (1 entries)
ARB0 relocating file +ACFS.8.1 (1 entries)
ARB0 relocating file +ACFS.9.1 (1 entries)
ARB0 relocating file +ACFS.256.839587727 (120 entries)

*** 2014-02-15 12:46:58.905
ARB0 relocating file +ACFS.256.839587727 (117 entries)
ARB0 relocating file +ACFS.256.839587727 (1 entries)
ARB0 relocating file +ACFS.257.839587727 (17 entries)
ARB0 relocating file +ACFS.258.839590377 (17 entries)

*** 2014-02-15 12:47:50.744
ARB0 relocating file +ACFS.259.839590377 (119 entries)
ARB0 relocating file +ACFS.259.839590377 (1 entries)
ARB0 relocating file +ACFS.260.839590389 (17 entries)
ARB0 relocating file +ACFS.261.839590389 (60 entries)
ARB0 relocating file +ACFS.261.839590389 (1 entries)
...

We see that the rebalance is per ASM file. This is exactly the same behaviour as with database files - ASM performs the rebalance on a per file basis. The ASM metadata files (1-9) get rebalanced first. The ASM then rebalances the volume file 256, DRL file 257, and so on.

From this we see that the ASM rebalances volume files (and other ASM files), not the OS files in the associated file system(s).

Disk online operation in an ACFS disk group

When an ASM disk goes offline, the ASM creates the staleness registry and staleness directory, to track the extents that should be modified on the offline disk. Once the disk comes back online, the ASM uses that information to perform the fast mirror resync.

That functionality is not available to volume files in ASM version 11.2. Instead, to online the disk, the ASM rebuilds the entire content of that disk. This is why the disk online performance, for disk groups with volume files, is inferior to the disk group with standard database files.

The fast mirror resync functionality for volume files is available in ASM version 12.1 and later.

Conclusion

ASM disk groups can be used to host a general purpose cluster file systems. ASM does this by creating volume files inside the disk groups, that are exposed to the operating system as block devices.

Existing ASM disk group mirroring functionality (normal and high redundancy) can be used to protect the user files at the file system level. ASM does this by mirroring extents for the volume files, in the same fashion it does this for any other ASM file. The volume files have their own extent sizes, unlike the standard database files that inherit the (initial) extent size from the disk group AU size.

The rebalance operation, in an ASM disk group that hosts ASM cluster file system volumes, is per volume file, not per the individual user files stored in the associated file system(s).