Alex Kernel Unix guy

root@r4275 [/nim] su – roy
Cannot set process environment.

The problem was not only with this one account so we assumed it has to do something with the enviroment files. As recommended we checked permissions on the files in /etc/ and /etc/security and then on / as well but these seemed fine. The mystery was revealed after debugging the command with truss:

root@r4275 [/home] truss su – roy
...
execve("/usr/bin/ksh", 0xF05A45A4, 0x201065F8) Err#13 EACCES
kopen("/usr/lib/nls/msg/en_US/su.cat", O_RDONLY) Err#13 EACCES
Cannot set process environmentkwrite(2, " C a n n o t s e t p".., 30) = 30
...

The directories /usr/bin and /usr/lib had mode 666 instead of 755. Satanic attack, isn't it?

During my trials to compile Net-SNMP 5.4.2.1 for AIX 6.1 a came across an make error that i resolved by small workaround. I was getting this error:

 gcc -I../../include -I. -I../../agent -I../../agent/mibgroup -I../../snmplib -g -O2 -Uaix6 -Daix6=aix6 -c mibII/ip.c  -DPIC -o mibII/.libs/ip.o
In file included from /usr/include/sys/corral.h:25,
                 from /usr/include/libperfstat.h:28,
                 from mibII/ip.c:37:
/usr/include/netinet/in6_var.h:65: error: array type has incomplete element type
make: 1254-004 The error code from the last command is 1.

Stop.
make: 1254-004 The error code from the last command is 2.

Stop.
make: 1254-004 The error code from the last command is 2.

Stop.

I compared the mentioned in6_var.h and in_var.h files in /usr/include/netinet and this first one had different declaration of some struct. I'm no programmer but i found on google, that it must be called as a pointer and not as a static variable so i changed the definition from this:

extern  CONST struct protosw inet6sw[];

to this:

extern  CONST struct protosw *inet6sw[];

Now it works but we don't use any network statistics monitoring on AIX servers so i don't know if all will work as it should. The compilation however was successful. If anyone came across this problem and found proper solution, please leave your comments, it will be highly appreciated, thank you.

I received an order to increase VG by additional vpath. So i tried it via smitty storage -> Volume Groups -> Add a Datapath Physical Volume to a Volume Group but this error message appeared:

Method error (/usr/lib/methods/chgvpath):
        0514-047 Cannot access a device.

0516-1182 extendvg Open Failure on vpath12.
0516-792 extendvg: Unable to extend volume group.

Honestly i have seen this error for the first time so i was not sure what to do but my more experienced colleague helped – what i forgot was to look into errpt, where i could have found the source of trouble:

---------------------------------------------------------------------------
LABEL:          VPATH_RESV_CFLICT
IDENTIFIER:     72206E77

Date/Time:       Thu May  3 12:03:24 DFT
Sequence Number: 17572
Machine Id:      00530F7A4C00
Node Id:         dom
Class:           H
Type:            PEND
Resource Name:   vpath12        
Resource Class:  disk
Resource Type:   vpath
Location:       

Description
REQUESTED OPERATION CANNOT BE PERFORMED

Probable Causes
SOFTWARE PROGRAM

Failure Causes
DEVICE LOCKED BY ANOTHER USER

        Recommended Actions
        RELEASE DEVICE PERSISTENT RESERVATION

Detail Data
SENSE DATA
0000 0031 8000 0028 0000 000C 0000 0001 0000 0000 0000 0000 0000 000C

Device locked by another user? What? Ok, there is a command that solves this cause, it's a low level command lquerypr. The problem here was that the PV has been previously assigned in some of the VGs (possibly on some other host) and got assigned a host key that didn't match with the new VG so it has to be cleaned in 3. steps:

# lquerypr -Vh /dev/vpath12
open device /dev/vpath12
setkey.compcode = 0
setkey.returncode = 1
c70d8d05
Reserved with different key c70d8d05, current host key 530f7a01

# lquerypr -Vph /dev/vpath12
open device /dev/vpath12
setkey.compcode = 0
setkey.returncode = 0
c70d8d05
Reserved with different key c70d8d05, current host key 530f7a01

# lquerypr -Vh /dev/vpath12
open device /dev/vpath12
setkey.compcode = 0
setkey.returncode = 0
Not reserved.

The first command will show the reservation, second (added the -p switch) will clean the reservation and third will show that the reservation has been cleaned. Now the vpath is ready to be assigned to new VG without trouble.

We have to remove unused SAP VGs for instance SES. It's called (surprisingly) sapvg_SES. And there is second – dbvg_SES. Let's check the layout:

# lsvg -l sapvg_SES
sapvg_SES:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
lv_SES_sapmnt       jfs2       16    16    1    open/syncd    /sapmnt/SES
loglv00             jfs2log    3     3     1    open/syncd    N/A
lv_SES_sap          jfs2       16    16    1    open/syncd    /usr/sap/SES
lv_SES_admhome      jfs2       1     1     1    open/syncd    /home/sesadm
lv_SES_saptrans     jfs2       83    83    1    open/syncd    /usr/sap/SEP/trans
lv_SES_mirrlogA     jfs2       4     4     1    open/syncd    /oracle/SES/mirrlogA
lv_SES_origlogB     jfs2       4     4     1    open/syncd    /oracle/SES/origlogB
lv_SES_archive      jfs2       120   120   1    open/syncd    /oracle/SES/oraarch

# lsvg -l dbvg_SES
dbvg_SES:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
lv_SES_oracle       jfs2       64    64    1    open/syncd    /oracle/SES
loglv01             jfs2log    3     3     1    open/syncd    N/A
lv_SES_origlogA     jfs2       4     4     1    open/syncd    /oracle/SES/origlogA
lv_SES_mirrlogB     jfs2       4     4     1    open/syncd    /oracle/SES/mirrlogB
lv_SES_sapdata1     jfs2       938   938   2    open/syncd    /oracle/SES/sapdata1

Good, not mirrored, will be easy. First let's unmount all the filesystems. After unsuccessfully trying to umount /oracle/SES i realized one thing and thought "man you're a dumb monkey, THINK before you TYPE":

# umount /oracle/SES
umount: 0506-349 Cannot unmount /dev/lv_SES_oracle: The requested resource is busy.

# fuser -c /oracle/SES
/oracle/SES:

# fuser -cuxk /oracle/SES
/oracle/SES:

# umount /oracle/SES
umount: 0506-349 Cannot unmount /dev/lv_SES_oracle: The requested resource is busy.

You probably already noticed the mistake yourself. I was desperately trying to unmount and unused filesystem that still had some underlying filesystems mounted. It's not that important, just thought it might be worth mentioning. With all the filesystems unmounted, let's do the magic and remove VGs.

# lsvg -p sapvg_SES
sapvg_SES:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk2            active            260         13          00..00..00..00..13

# reducevg -df sapvg_SES hdisk2
rmlv: Logical volume lv_SES_sapmnt is removed.
rmlv: Logical volume loglv00 is removed.
rmlv: Logical volume lv_SES_sap is removed.
rmlv: Logical volume lv_SES_admhome is removed.
rmlv: Logical volume lv_SES_saptrans is removed.
rmlv: Logical volume lv_SES_mirrlogA is removed.
rmlv: Logical volume lv_SES_origlogB is removed.
rmlv: Logical volume lv_SES_archive is removed.
ldeletepv: Volume Group deleted since it contains no physical volumes.
3001-047 There is no matching file entry for /dev/sapvg_SES.

# lsvg -p dbvg_SES
dbvg_SES:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk3            active            521         0           00..00..00..00..00
hdisk6            active            521         29          00..00..00..00..29

# reducevg -df dbvg_SES hdisk6
You have mail in /usr/spool/mail/root

# reducevg -df dbvg_SES hdisk3
rmlv: Logical volume lv_SES_oracle is removed.
rmlv: Logical volume loglv01 is removed.
rmlv: Logical volume lv_SES_origlogA is removed.
rmlv: Logical volume lv_SES_mirrlogB is removed.
rmlv: Logical volume lv_SES_sapdata1 is removed.
ldeletepv: Volume Group deleted since it contains no physical volumes.
3001-047 There is no matching file entry for /dev/dbvg_SES.

What about the options -d and -f? The first option -d causes the reducevg command to deallocate all the partitions before removing the VG, in other words it will first destroy the logical volumes and the -f takes care that you don't have to answer Yes to a confirmation question. Notice that dbvg_SES was deleted first after i have removed the last PV from VG. If you're using some kind of filesystems monitoring don't forget to adjust configuration files.

Imagine we have these two machines PC1 and PC2 in a cluster. There are two volume groups, one is shared in a cluster,  and one is not, so this is out of our focus. There is vgsapr3per that is active on PC1 and shared to PC2 as standby node for this VG.

First we have to "unlock" the VG to be able to work with it on the other node:

PC1:
# varyonvg -bu vgsapr3per

Next you need to "learn" the changes in VG on second node and use the name of one of the disks that are in VG:

PC1:
# lsvg -p vgsapr3per
vgsapr3per:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
...
vpath12           active            260         0           00..00..00..00..00
vpath11           active            260         0           00..00..00..00..00
vpath13           active            260         202         52..00..46..52..52
vpath14           active            260         202         52..00..46..52..52

The only free disks were vpath13 and 14 so i thought ok let's use vpath13 but no:

PC2:
# importvg -L vgsapr3per vpath13
0516-1141 redefinevg: Mismatch between VGID information in ODM and VGDA.
        Check the physical volume name specified.
0516-780 importvg: Unable to import volume group from vpath13.

Aha! The problem here is you need to get PVID of some disk that is common on both nodes. Note that the disks don't have to be named the same but they are identified by the PVID. In my example i have chosen vpath10 that exists as vpath8, note the PVIDs are the same and VG is active on PC1. To make it more simple you can just grep for the PVID on the second node.

PC2:
# lspv
...
vpath10         00cbd98e37203180                 vgsapr3per
...

PC1:
...
vpath8          00cbd98e37203180                 vgsapr3per      active
...

Now we try to learn it again and voila, it works:

PC2:
# importvg -L vgsapr3per vpath10
vgsapr3per

Run this command on both nodes to synchronize the timestamps on VGs on both nodes:

PC1, PC2:
# /usr/sbin/cluster/utilities/clupdatevgts vgsapr3per

And again lock the VG for PC1 only:

PC1:
# varyonvg vgsapr3per

That's it. Problem solved.

Remember that in clustered enviroment you must think about the way both nodes see the changes. Both nodes must be informed about changes in shared VGs to prevent paritioning of a cluster and data inconsistency. First let's do some checks and collect data. I need to increase a clustered filesystem /globalPER

# df
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
...
/dev/lv_contrm    1048576    450288   58%     1744     4% /controlm
/dev/lv_contrm610    4194304   2136536   50%     3149     2% /controlm610
/dev/lvglobalPER   90701824   9456672   90%   284364    21% /globalPER
...

Note i want to record the size in 512-byte blocks.

# lsvg
rootvg
hbvg
vgsapr3per

# lsvg -l vgsapr3per
vgsapr3per:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
lvglobalPER         jfs2       173   346   10   open/syncd    /globalPER
...

# lsvg vgsapr3per
VOLUME GROUP:       vgsapr3per               VG IDENTIFIER:  00cbd98e00004c000000010437205038
VG STATE:           active                   PP SIZE:        256 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      3380 (865280 megabytes)
MAX LVs:            256                      FREE PPs:       420 (107520 megabytes)
...

We can see, the LV is mirrored and there are enough free PPs in VG. So far so good.

# lsvg -p vgsapr3per
vgsapr3per:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
...
vpath9            active            260         0           00..00..00..00..00
vpath10           active            130         0           00..00..00..00..00
vpath12           active            260         0           00..00..00..00..00
vpath11           active            260         0           00..00..00..00..00
vpath13           active            260         210         52..02..52..52..52
vpath14           active            260         210         52..02..52..52..52

Only vpath13 and vpath14 have free PPs, so we will use only these two but don't forget to check with datapath command if they're mirrored over different datacenters to keep redundancy.

# datapath query device | egrep -p "vpath13 | vpath14"
DEV#:  13  DEVICE NAME: vpath13  TYPE: 2105800         POLICY:    Optimized
SERIAL: 50224568
===========================================================================
Path#      Adapter/Hard Disk          State     Mode     Select     Errors
    0         fscsi0/hdisk54           OPEN   NORMAL      22476          0
    1         fscsi0/hdisk55           OPEN   NORMAL      22394          0
    2         fscsi1/hdisk58           OPEN   NORMAL      22394          0
    3         fscsi1/hdisk59           OPEN   NORMAL      22564          0

DEV#:  14  DEVICE NAME: vpath14  TYPE: 2105800         POLICY:    Optimized
SERIAL: 50218370
===========================================================================
Path#      Adapter/Hard Disk          State     Mode     Select     Errors
    0         fscsi0/hdisk56           OPEN   NORMAL      22453          0
    1         fscsi0/hdisk57           OPEN   NORMAL      22233          0
    2         fscsi1/hdisk60           OPEN   NORMAL      22377          0
    3         fscsi1/hdisk61           OPEN   NORMAL      22767          0

Ok, last 5 numbers show these disks are stored in different boxes, so we are sure it will be spread to different places. Let's head to smitty hacmp. System Management (C-SPOC) > HACMP Logical Volume Management > Shared Logical Volumes > Set Characteristics of a Shared Logical Volume > Increase the Size of a Shared Logical Volume. In the popup menu select the correct LV, in our case it's lvglobalPER. In next menu select correct vpaths (vpath13, vpath14). In the next menu the only thing you need to put in are the additional LPs.

                                            Increase the Size of a Shared Logical Volume

Type or select values in entry fields.
Press Enter AFTER making all desired changes.
 
                                                        [Entry Fields]
  Resource Group Name                                 sap_prod_PER
  LOGICAL VOLUME name                                 lvglobalPER
  Reference node                                      pesappc1
* Number of ADDITIONAL logical partitions            [8]                   #
  PHYSICAL VOLUME names                               vpath13 vpath14
  POSITION on physical volume                         outer_middle         +
  RANGE of physical volumes                           minimum              +
  MAXIMUM NUMBER of PHYSICAL VOLUMES                 []                    #
    to use for allocation
  Allocate each logical partition copy                yes                  +
    on a SEPARATE physical volume?
  File containing ALLOCATION MAP                     []

I needed to increase by 2GB, PP size is 256MB so 8 LPs are just the right number. Confirm the selection and let's move on. In the HACMP Logical Volume Management menu choose Shared File Systems then ours is Enhanced Journaled File Systems > Change / Show Characteristics of a Shared Enhanced Journaled File System, in the popup menu choose the correct FS, we have /globalPER.

                              Change/Show Characteristics of a Shared Enhanced Journaled File System

Type or select values in entry fields.
Press Enter AFTER making all desired changes.
 
                                                        [Entry Fields]
  Resource Group Name                                 sap_prod_PER
  File system name                                    /globalPER
  NEW mount point                                    [/globalPER]
  SIZE of file system                                [94896128]
  Mount GROUP                                        [per]
  PERMISSIONS                                        read/write             +
  Mount OPTIONS                                      []                     +
  Start Disk Accounting?                              no                    +
  Block Size (bytes)                                  4096
  Inline Log?                                         no
  Inline Log size (MBytes)                            0

Remember the 512-byte blocks? Here you see them, so if you added 2GB it's 2 GB x 1024 MB x1024 kB x 2 (512 byte blocks) = 4194304 additional. Add this to the previous value and you have the NEW size of the FS after increase. Confirm and voila, that's it! 

It's good for those working with AIX that it offers Logical Volume Manager. And with help of smitty menus, managing disk space couldn't be ever easier. Since AIX 5.3 you can not only increase the filesystems but there is a possibility to even shrink them. In AIX 5.2 it was needed to do a complete filesystem backup, destroy the fs and create it a new with smaller size. Then do the restore of data. But with 5.3 that's over now, so no need to focus on that.

First do a check if you have enough free PPs in the volume group:

# lsvg oravg
...
MAX LVs: 256 FREE PPs: 34 (4352 megabytes)
...

Ok we have about 4 GB. For simple unmirrored disk you can directly increase/decrease, example:

# chfs -a size=+2G /u01

This would add 2 GB to the /u01 filesystem. If you use the -2G option to the size attribute it will shrink it by 2 GB assuming there is at least 2 GB of free space. Otherwise LVM wouldn't be able to relocate the LPs. This is too simple, more fun you'll have when increasing mirrored filesystem.

My personal experience is that usually only rootvg is mirrored, so the OS is available even after disk failure. Application VGs are mirrored mostly on cluster servers. And this is still nothing complicated if you have only 2 mirrored disks in the VG because LVM can spread the mirrors only between these two disks. It's better when you have 4 disks so you have to choose the correct 2 original and the other 2 that serve as mirror. Imagine situation: 4 disks, 2 in one datacenter, 2 in other datacenter (best a few km away to be protected from local disaster =)

dc-1   dc-2
vpath2 vpath7
vpath3 vpath8

In this case it's better first increase the logical volume by desired amount of LPs and select explicitly the 2 disks that are mirrored (and that have free PPs). So we'll first increase the LV by let's say 16 partitions:

# extendlv lv_u01 16 vpath2 vpath7

Imagine you don't have enough free PPs on vpath2 and 7. What now? Well simple, just issue the extendlv command again and use the other mirrored disks. After that you can simply increase the FS and you are sure that PPs will be mirrored correctly.

1. preparation is an important step and it's good to be well prepared, it means less unexpected problems later.

Prepare checklist in printed paper form so you can cross out the steps you've done. Also keep in mind early planning helps you collect as many things to fix as possible. You can prepare needed filesets, microcode drivers or check backups if they're ok, check connections, for reboot it's good to have console connection as well.

2. try to gather as many fixes as you can.

My colleague advised me using the invscout to get a list of what needs to be updated from hardware as well. It's always good to patch as much as you can handle when you have the opportunity in productive environment. Invscout will create a file with microcode information that you can upload on Microcode Discovery Service (which takes some time to find).

Alternatively you can look for the applications that need to be updated, beware of dependencies and compatibility warnings, it's good to dig into the long readme files to prevent patching from smooth processing.

3. patch order that makes sense.

It's good to have some order in the patching so you don't end in mess. I usually commit everything and get rid of the non-rootvg VGs first. That means stop all applications that might use these VGs, unmount all filesystems, swapoff paging space, remove dump device, vary off volume groups and after that start patching TL and SP. If you're short on time you can update as many filesets as possible and after that do the reboot. And if you're updating microcode levels, you usually will need a reboot anyway.

4. don't panic!

If you're running out of time or something doesn't come out as you planned, the rule is keep your head cool. Think of the solution and not of the consequences of late system handover, better late than in inconsistent and bad shape. We're only human and we tend to make mistakes. And if you solve some problem, you get the experience and the appreciation. If you'll try to hide it, the problem will pop-up later anyway and can have worse consequences.