Alex Kernel Unix guy

We experienced problem with nmon_grapher script that it won't give you outputs even if everything seems to be running with exit code 0. It might be because in AIX 6.1 there has been a change in timezone naming conventions (value of the $TZ variable):

AIX 5.3: CET-1CEST,M3.5.0,M10.5.0/03:00
AIX 6.1: Europe/Prague

As the script relies on getting data from TZ variable that are not present in new format, it won't work correctly and will plan at jobs for data collection at incorrect time. You can adjust the script and add a condition that will set the correct TZ format for script:

if [ `oslevel` == "5.3.0.0" ] ; then
TZ_FIRST="`echo $TZ | awk -F , '{ print }'`"
TZ_OFFSET=`echo $TZ_FIRST | sed "s/[a-zA-Z]//g"`
[[ $TZ_OFFSET = [0-9]* ]] && TZ_OFFSET="+$TZ_OFFSET"
[[ $TZ_OFFSET != *[0-9] ]] && TZ_OFFSET="+0"
(( offset = -$timetorun $TZ_OFFSET ))
TZ_NEW="TST${offset}TST${TZ#$TZ_FIRST}"
fi
# Hotfix for AIX 6.1 timezone 15.9.2011 JBoc
if [ `oslevel` == "6.1.0.0" ] ; then
TZ_NEW="TST-2TST,M3.5.0,M10.5.0/03:00"
fi
nexttime=`TZ=$TZ_NEW;date +'%Y%m%d%H00'`
(( timetorun = timetorun * 3600 ))

If you ever come across similar message, it may be a problem in the snmp communication, either way try to refresh the clinfoES subsystem, then it should work as normal 

clstat

There is more than one cluster accessible.
Either provide the cluster to look for, or run
the program interactively (or try again).

usage: clstat [-c cluster ID | -n cluster name] [-i] [-r seconds] [-a|-o] [-s]

  -c cluster ID > run in automatic (non-interactive) mode for the
                  specified cluster.
  -n name       > run in automatic (non-interactive) mode for the
                  specified cluster name.
  -i            > run in interactive mode
  -r seconds    > number of seconds between each check of the
                  cluster status
  -a            > run ascii version.
  -o            > run once and exit.
  -s            > display both up and down service labels.

If you ever needed for some reasons to create SHA256 hashes in AIX, then coded directly into BASE64, there is a nice tool for it. Although it's written in Java. It's called Jacksum: http://www.jonelo.de/java/jacksum/

With native AIX commands you have only the possibility to calculate SHA-1 that comes with the command csum. With old Perl and Python in AIX Linux toolbox repository, Perl being there on version 5.8.2 and Python 2.3.4 currently, you don't even have too much options to create some Perl or Python scripts with this functionality. SHA256 function is available i guess in Perl 5.9.3 and Python 2.5 and higher. In this field AIX looks a bit aged, so the small Java script is exactly what i needed. If you know of native AIX tools that have the functionality of Linux commands sha256sum and base64, please leave any comments.

root@r4275 [/nim] su – roy
Cannot set process environment.

The problem was not only with this one account so we assumed it has to do something with the enviroment files. As recommended we checked permissions on the files in /etc/ and /etc/security and then on / as well but these seemed fine. The mystery was revealed after debugging the command with truss:

root@r4275 [/home] truss su – roy
...
execve("/usr/bin/ksh", 0xF05A45A4, 0x201065F8) Err#13 EACCES
kopen("/usr/lib/nls/msg/en_US/su.cat", O_RDONLY) Err#13 EACCES
Cannot set process environmentkwrite(2, " C a n n o t s e t p".., 30) = 30
...

The directories /usr/bin and /usr/lib had mode 666 instead of 755. Satanic attack, isn't it?

During my trials to compile Net-SNMP 5.4.2.1 for AIX 6.1 a came across an make error that i resolved by small workaround. I was getting this error:

 gcc -I../../include -I. -I../../agent -I../../agent/mibgroup -I../../snmplib -g -O2 -Uaix6 -Daix6=aix6 -c mibII/ip.c  -DPIC -o mibII/.libs/ip.o
In file included from /usr/include/sys/corral.h:25,
                 from /usr/include/libperfstat.h:28,
                 from mibII/ip.c:37:
/usr/include/netinet/in6_var.h:65: error: array type has incomplete element type
make: 1254-004 The error code from the last command is 1.

Stop.
make: 1254-004 The error code from the last command is 2.

Stop.
make: 1254-004 The error code from the last command is 2.

Stop.

I compared the mentioned in6_var.h and in_var.h files in /usr/include/netinet and this first one had different declaration of some struct. I'm no programmer but i found on google, that it must be called as a pointer and not as a static variable so i changed the definition from this:

extern  CONST struct protosw inet6sw[];

to this:

extern  CONST struct protosw *inet6sw[];

Now it works but we don't use any network statistics monitoring on AIX servers so i don't know if all will work as it should. The compilation however was successful. If anyone came across this problem and found proper solution, please leave your comments, it will be highly appreciated, thank you.

I received an order to increase VG by additional vpath. So i tried it via smitty storage -> Volume Groups -> Add a Datapath Physical Volume to a Volume Group but this error message appeared:

Method error (/usr/lib/methods/chgvpath):
        0514-047 Cannot access a device.

0516-1182 extendvg Open Failure on vpath12.
0516-792 extendvg: Unable to extend volume group.

Honestly i have seen this error for the first time so i was not sure what to do but my more experienced colleague helped – what i forgot was to look into errpt, where i could have found the source of trouble:

---------------------------------------------------------------------------
LABEL:          VPATH_RESV_CFLICT
IDENTIFIER:     72206E77

Date/Time:       Thu May  3 12:03:24 DFT
Sequence Number: 17572
Machine Id:      00530F7A4C00
Node Id:         dom
Class:           H
Type:            PEND
Resource Name:   vpath12        
Resource Class:  disk
Resource Type:   vpath
Location:       

Description
REQUESTED OPERATION CANNOT BE PERFORMED

Probable Causes
SOFTWARE PROGRAM

Failure Causes
DEVICE LOCKED BY ANOTHER USER

        Recommended Actions
        RELEASE DEVICE PERSISTENT RESERVATION

Detail Data
SENSE DATA
0000 0031 8000 0028 0000 000C 0000 0001 0000 0000 0000 0000 0000 000C

Device locked by another user? What? Ok, there is a command that solves this cause, it's a low level command lquerypr. The problem here was that the PV has been previously assigned in some of the VGs (possibly on some other host) and got assigned a host key that didn't match with the new VG so it has to be cleaned in 3. steps:

# lquerypr -Vh /dev/vpath12
open device /dev/vpath12
setkey.compcode = 0
setkey.returncode = 1
c70d8d05
Reserved with different key c70d8d05, current host key 530f7a01

# lquerypr -Vph /dev/vpath12
open device /dev/vpath12
setkey.compcode = 0
setkey.returncode = 0
c70d8d05
Reserved with different key c70d8d05, current host key 530f7a01

# lquerypr -Vh /dev/vpath12
open device /dev/vpath12
setkey.compcode = 0
setkey.returncode = 0
Not reserved.

The first command will show the reservation, second (added the -p switch) will clean the reservation and third will show that the reservation has been cleaned. Now the vpath is ready to be assigned to new VG without trouble.

We have to remove unused SAP VGs for instance SES. It's called (surprisingly) sapvg_SES. And there is second – dbvg_SES. Let's check the layout:

# lsvg -l sapvg_SES
sapvg_SES:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
lv_SES_sapmnt       jfs2       16    16    1    open/syncd    /sapmnt/SES
loglv00             jfs2log    3     3     1    open/syncd    N/A
lv_SES_sap          jfs2       16    16    1    open/syncd    /usr/sap/SES
lv_SES_admhome      jfs2       1     1     1    open/syncd    /home/sesadm
lv_SES_saptrans     jfs2       83    83    1    open/syncd    /usr/sap/SEP/trans
lv_SES_mirrlogA     jfs2       4     4     1    open/syncd    /oracle/SES/mirrlogA
lv_SES_origlogB     jfs2       4     4     1    open/syncd    /oracle/SES/origlogB
lv_SES_archive      jfs2       120   120   1    open/syncd    /oracle/SES/oraarch

# lsvg -l dbvg_SES
dbvg_SES:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
lv_SES_oracle       jfs2       64    64    1    open/syncd    /oracle/SES
loglv01             jfs2log    3     3     1    open/syncd    N/A
lv_SES_origlogA     jfs2       4     4     1    open/syncd    /oracle/SES/origlogA
lv_SES_mirrlogB     jfs2       4     4     1    open/syncd    /oracle/SES/mirrlogB
lv_SES_sapdata1     jfs2       938   938   2    open/syncd    /oracle/SES/sapdata1

Good, not mirrored, will be easy. First let's unmount all the filesystems. After unsuccessfully trying to umount /oracle/SES i realized one thing and thought "man you're a dumb monkey, THINK before you TYPE":

# umount /oracle/SES
umount: 0506-349 Cannot unmount /dev/lv_SES_oracle: The requested resource is busy.

# fuser -c /oracle/SES
/oracle/SES:

# fuser -cuxk /oracle/SES
/oracle/SES:

# umount /oracle/SES
umount: 0506-349 Cannot unmount /dev/lv_SES_oracle: The requested resource is busy.

You probably already noticed the mistake yourself. I was desperately trying to unmount and unused filesystem that still had some underlying filesystems mounted. It's not that important, just thought it might be worth mentioning. With all the filesystems unmounted, let's do the magic and remove VGs.

# lsvg -p sapvg_SES
sapvg_SES:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk2            active            260         13          00..00..00..00..13

# reducevg -df sapvg_SES hdisk2
rmlv: Logical volume lv_SES_sapmnt is removed.
rmlv: Logical volume loglv00 is removed.
rmlv: Logical volume lv_SES_sap is removed.
rmlv: Logical volume lv_SES_admhome is removed.
rmlv: Logical volume lv_SES_saptrans is removed.
rmlv: Logical volume lv_SES_mirrlogA is removed.
rmlv: Logical volume lv_SES_origlogB is removed.
rmlv: Logical volume lv_SES_archive is removed.
ldeletepv: Volume Group deleted since it contains no physical volumes.
3001-047 There is no matching file entry for /dev/sapvg_SES.

# lsvg -p dbvg_SES
dbvg_SES:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk3            active            521         0           00..00..00..00..00
hdisk6            active            521         29          00..00..00..00..29

# reducevg -df dbvg_SES hdisk6
You have mail in /usr/spool/mail/root

# reducevg -df dbvg_SES hdisk3
rmlv: Logical volume lv_SES_oracle is removed.
rmlv: Logical volume loglv01 is removed.
rmlv: Logical volume lv_SES_origlogA is removed.
rmlv: Logical volume lv_SES_mirrlogB is removed.
rmlv: Logical volume lv_SES_sapdata1 is removed.
ldeletepv: Volume Group deleted since it contains no physical volumes.
3001-047 There is no matching file entry for /dev/dbvg_SES.

What about the options -d and -f? The first option -d causes the reducevg command to deallocate all the partitions before removing the VG, in other words it will first destroy the logical volumes and the -f takes care that you don't have to answer Yes to a confirmation question. Notice that dbvg_SES was deleted first after i have removed the last PV from VG. If you're using some kind of filesystems monitoring don't forget to adjust configuration files.

Imagine we have these two machines PC1 and PC2 in a cluster. There are two volume groups, one is shared in a cluster,  and one is not, so this is out of our focus. There is vgsapr3per that is active on PC1 and shared to PC2 as standby node for this VG.

First we have to "unlock" the VG to be able to work with it on the other node:

PC1:
# varyonvg -bu vgsapr3per

Next you need to "learn" the changes in VG on second node and use the name of one of the disks that are in VG:

PC1:
# lsvg -p vgsapr3per
vgsapr3per:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
...
vpath12           active            260         0           00..00..00..00..00
vpath11           active            260         0           00..00..00..00..00
vpath13           active            260         202         52..00..46..52..52
vpath14           active            260         202         52..00..46..52..52

The only free disks were vpath13 and 14 so i thought ok let's use vpath13 but no:

PC2:
# importvg -L vgsapr3per vpath13
0516-1141 redefinevg: Mismatch between VGID information in ODM and VGDA.
        Check the physical volume name specified.
0516-780 importvg: Unable to import volume group from vpath13.

Aha! The problem here is you need to get PVID of some disk that is common on both nodes. Note that the disks don't have to be named the same but they are identified by the PVID. In my example i have chosen vpath10 that exists as vpath8, note the PVIDs are the same and VG is active on PC1. To make it more simple you can just grep for the PVID on the second node.

PC2:
# lspv
...
vpath10         00cbd98e37203180                 vgsapr3per
...

PC1:
...
vpath8          00cbd98e37203180                 vgsapr3per      active
...

Now we try to learn it again and voila, it works:

PC2:
# importvg -L vgsapr3per vpath10
vgsapr3per

Run this command on both nodes to synchronize the timestamps on VGs on both nodes:

PC1, PC2:
# /usr/sbin/cluster/utilities/clupdatevgts vgsapr3per

And again lock the VG for PC1 only:

PC1:
# varyonvg vgsapr3per

That's it. Problem solved.