Troubleshooting kernel package installation

If after applying my kernel 2.6.9-89.0.20 package the lcd shows 'IP Address not set' , there is a failure in the upgrade installation. This is because yum had installed newer kernels and the rpm software don't want to install the rpm included in this package.

1.- To repair it you need to boot the server with a terminal connected to the serial console port. When the message 'Press spacebar to enter ROM mode' appears, press the space bar.
2.- Once in the ROM menu type 'bfd /boot/vmlinux_old_002.bz2'. This command forces to boot from the old kernel.
3.- Login to the server as root.
4.- # cd /boot
5.- # rpm -i --force http://www.titox-net.es/downloads/cobalt/BlueQuartz/kernel/linux-2.6.9-89.0.20-cobalt/kernel-2.6.9-89.0.20_cobalt.i386.rpm
6.- # ln -s vmlinux-2.6.9-89.0.20.EL-cobalt.bz2 vmlinux.bz2
7.- # reboot

There is a mistake in the kernel config and RAID devices are not working... OK I will compile a new version soon as possible, I need for my new server!!!!!!! Sorry guys.

A big-little scare: Burning RaQ 4 ROM failure!!!!

Yesterday I started to update an spare RaQ 4i to use it as main server at home. The old RaQ 4 have worked the last 3 years without any stop and I feel that I need to change the server for caution.

This second RaQ 4i has two extra ethernets ports and two 120GB Seagate hard disks but now it's loaded with the old Cobalt OS (fully upgraded). I tried to compile MySQL 5.1.46 on it but I has some problems and decided to upgrade the kernel to 2.4.36 (Zeffie's kernel). But this new kernel needs ROM version 2.10.3-ext3.

First I started doing a backup of the old ROM:

[root /root]# ./flashtool -v -r > cobalt-2.3.34-1M.rom
./flashtool: searching for PCI 10b9:7101 : found it at /proc/bus/pci/00/03.0
./flashtool: systype = COBT_3K
./flashtool: bank 0: ST Microelectronics M29F080A 1MB
./flashtool: reading page 0
./flashtool: reading page 1
./flashtool: reading page 2
./flashtool: reading page 3
./flashtool: reading page 4
./flashtool: reading page 5
./flashtool: reading page 6
./flashtool: reading page 7
./flashtool: reading page 8
./flashtool: reading page 9
./flashtool: reading page 10
./flashtool: reading page 11
./flashtool: reading page 12
./flashtool: reading page 13
./flashtool: reading page 14
./flashtool: reading page 15
./flashtool: flushing buffers

Like I did before in other servers I started using Tim Hocking's flashtool and OUCH!!! FAILED!!!!:

[root /root]# ./flashtool -v -w cobalt-2.10.3-ext3-1M.rom
./flashtool: searching for PCI 10b9:7101 : found it at /proc/bus/pci/00/03.0
./flashtool: systype = COBT_3K
./flashtool: bank 0: ST Microelectronics M29F080A 1MB
./flashtool: writing page 0
./flashtool: buffer page 0 does not exist - creating it
./flashtool: writing page 1
./flashtool: buffer page 1 does not exist - creating it
./flashtool: writing page 2
./flashtool: buffer page 2 does not exist - creating it
./flashtool: writing page 3
./flashtool: buffer page 3 does not exist - creating it
./flashtool: writing page 4
./flashtool: buffer page 4 does not exist - creating it
./flashtool: writing page 5
./flashtool: buffer page 5 does not exist - creating it
./flashtool: writing page 6
./flashtool: buffer page 6 does not exist - creating it
./flashtool: writing page 7
./flashtool: buffer page 7 does not exist - creating it
./flashtool: writing page 8
./flashtool: buffer page 8 does not exist - creating it
./flashtool: writing page 9
./flashtool: buffer page 9 does not exist - creating it
./flashtool: writing page 10
./flashtool: buffer page 10 does not exist - creating it
./flashtool: writing page 11
./flashtool: buffer page 11 does not exist - creating it
./flashtool: writing page 12
./flashtool: buffer page 12 does not exist - creating it
./flashtool: writing page 13
./flashtool: buffer page 13 does not exist - creating it
./flashtool: writing page 14
./flashtool: buffer page 14 does not exist - creating it
./flashtool: writing page 15
./flashtool: buffer page 15 does not exist - creating it
./flashtool: flushing buffers
./flashtool:   flushing block 0 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 0! Bad flash chip?
 done
./flashtool:   flushing block 1 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 1! Bad flash chip?
 done
./flashtool:   flushing block 2 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 2! Bad flash chip?
 done
./flashtool:   flushing block 3 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 3! Bad flash chip?
 done
./flashtool:   flushing block 4 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 4! Bad flash chip?
 done
./flashtool:   flushing block 5 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 5! Bad flash chip?
 done
./flashtool:   flushing block 6 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 6! Bad flash chip?
 done
./flashtool:   flushing block 7 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 7! Bad flash chip?
 done
./flashtool:   flushing block 8 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 8! Bad flash chip?
 done
./flashtool:   flushing block 9 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 9! Bad flash chip?
 done
./flashtool:   flushing block 10 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 10! Bad flash chip?
 done
./flashtool:   flushing block 11 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 11! Bad flash chip?
 done
./flashtool:   flushing block 12 to ROM..../flashtool: flashrom_flush_buffers(): write failed for block 12! Bad flash chip?
 done
./flashtool:   flushing block 13 to ROM... verifying...  done
./flashtool:   flushing block 14 to ROM... verifying...  done
./flashtool:   flushing block 15 to ROM... verifying...  done
./flashtool: flashrom_cleanup(): device error flushing buffers

As you can see this server has the standard old ST ROM chip. I upgraded this brand roms with this tool but yesterday it won't run. Fortunately I have a copy of Duncan's flashtool (the small one - 400 KB) and it worked fine. I uploaded via FTP and burned the rom:

[root /root]# ./flashtool -v -w cobalt-2.10.3-ext3-1M.rom
./flashtool: searching for PCI 10b9:7101 : found it at /proc/bus/pci/00/03.0
./flashtool: systype = COBT_3K
./flashtool: bank 0: ST Microelectronics M29F080A 1MB
./flashtool: Using pthread POSIX real time scheduling.
./flashtool: writing page 0
./flashtool: buffer page 0 does not exist - creating it
./flashtool: writing page 1
./flashtool: buffer page 1 does not exist - creating it
./flashtool: writing page 2
./flashtool: buffer page 2 does not exist - creating it
./flashtool: writing page 3
(...)

Very important in this case was to not power off the server: if after a failed rom burn the server is powered off, the server will be a brick, totally dead.

For this reason now I have both flashtool programs mirrored in my server here: http://www.titox-net.es/downloads/cobalt/rom/

Sun Fire V120 as NAS part 7: Gigabit networking

Well, one week ago I received a new Sun Quad GigaSwift PCI card for this server. It's a full lenght PCI card with 64 bits PCI bus, but works in 32 bits PCI bus also.

As you can see in the pictures, the card installation is very easy. Only take out the blank bracket and the small grey plastic, then mount the card in th server slot. Finally return to set the grey plastic which locks the rear part of the GigaSiwft.

The network cards can be configured via the ifconfig utility in Solaris but to make this configurations stored we have to do some more file editting:

1.- The first step is to check if the server detects the GigaSwift card:

# grep ce /etc/path_to_inst
"/pci@1f,0/pci@1/pci@5/pci@0/network@0" 0 "ce"
"/pci@1f,0/pci@1/pci@5/pci@0/network@1" 1 "ce"
"/pci@1f,0/pci@1/pci@5/pci@4/network@2" 2 "ce"
"/pci@1f,0/pci@1/pci@5/pci@4/network@3" 3 "ce"

2.- Second step: create the /etc/hostname.ce0 file containing the name of the server. This name can't be the same in two different ethernet ports. In the 100MB port I'm using v120 as hostname, now I added a 'g' to the name:

# cat /etc/hostname.ce0
v120g

ce0 is the instance number for the port I'm going to use. I only need one port of the four installed with this card. If we want to use more port, more hostname.ce'instance_number' are needed.

3.- Now we have to edit the /etc/inet/hosts file but the vi editor says it's write protected (and I'm doing it beeing root). Like other changes I did before, I uploaded this file to my laptop via FTP, modified with a text editor and then downloaded to the server.

# cd /etc/inet
# cp /export/home/admin/hosts .

# cat /etc/inet/hosts
#
# Internet host table
#
::1     localhost
127.0.0.1       localhost
192.168.2.120   v120    loghost
192.168.5.120   v120g
~

4.- With the two configuration files ready, the card can be turned on:

# ifconfig ce0 plumb 192.168.5.120 up

Unfortunately the trasfer speed is the same as the speed obtained with the 100MB onboard port. Yes, I know there is a lot of parameters to tune, but I will look on the next days. Another problem I found is that: with both Ethernet ports configured (onboard an GigaSwift), when I unplug the onboard port cable, I have a lot of problems to connect with the Gigaswift (well, I can't connect). I think this is because Solaris uses the main port to check against DNS server. Then, I disconfigured the onboard ethernet port:

1.- Unplumb the card:

# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
        inet 192.168.5.120 netmask ffffff00 broadcast 192.168.5.255
        ether 0:3:ba:5c:2b:7d
eri0: flags=1000843 mtu 1500 index 3
        inet 192.168.2.120 netmask ffffff00 broadcast 192.168.2.255
        ether 0:3:ba:5c:2b:7d

# ifconfig eri0 unplumb

# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
        inet 192.168.5.120 netmask ffffff00 broadcast 192.168.5.255
        ether 0:3:ba:5c:2b:7d

2.- Delete the /etc/hosts entry (I modified the file in my laptop):

# cd /etc/inet
# cp hosts hosts.old
# cp /export/home/admin/hosts .
# cat hosts

#
# Internet host table
#
::1     localhost
127.0.0.1       localhost
192.168.5.120   v120g   loghost

3. Delete the /etc/hostname.eri0 file (here I moved it):

# cd /etc
# ls hostname*
hostname.ce0   hostname.eri0
# mv hostname.eri0 hostname.ri0.old

4.- Unplumb the ethernet port (and reboot):

# ifconfig eri0 unplumb
# reboot

Now everything works good with only the GigaSwift port 0 attached to the network. Something that is killing me is the low speed transfer via FTP. I can only reach 4.5 MB/s. This is a very low value. While doing a file tranfer I checked the server status with iostat:

# iostat
   tty        sd0           sd3           sd16          sd17           cpu
 tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv   us sy wt id
   0    1   2   0    4  439   8   28  789   8   26  758   8   31   60 26  0 14

Here I can see that the CPU is hard working. Maybe all the calculations needed by Samba, ftp and the other services joined by the ZFS filesystem are eating a lot of resources. I think I will check to optimize the network parameters for play and then I will reformat the hard disks using SVM instead of ZFS.

P.D.: Do you know what is this? (Check picture)

Sun Fire V120 as NAS part 6: Network shares and RAM upgrade

Ok. Time to start some file sharing protocols.

First the easy part: NFS.
Solaris comes with NFS4 (backwards compatible with NFS2 and NFS3). Not very important in a normal home but in my case, this is the best option to share information with my SGI boxes. Only three steps are needed to start sharing, Fantastic!

1. Make writable to everybody the folder where I want to share. I use the same array than before:

# chmod 777 /fsshared

2. Add the share information to the file /etc/dfs/dfstab

#       Place share(1M) commands here for automatic execution
#       on entering init state 3.
#
#       Issue the command 'svcadm enable network/nfs/server' to
#       run the NFS daemon processes and the share commands, after adding
#       the very first entry to this file.
#
#       share [-F fstype] [ -o options] [-d ""] [resource]
#       .e.g,
#       share  -F nfs  -o rw=engineering  -d "home dirs"  /export/home2
share -F nfs -o rw -d "fsshared V120" /fsshared

3. Start sharing:

# shareall

The dfstab entry goes:

• -F nfs .- Type of sharing.
• -o rw .- The options. Here I only used read-write but there is a lot more (including loggin), but I just don't need.
• -d "fsshared V120" .- Obviously is only a comment.
• /fsshared .- Here comes the folder I want to export.

Here some captures from my Fuel mounting the exported folder of the Sun Fure V120:

Second part is SAMBA!
This is the standard Windows protocol. At the time the most important in any home or company. From a lot of time ago samba.org is publising the source code and it's widely used but all the test I did before (and some test done by friends) are telling Samba is slow, in Solaris and in Linux.

Configure Samba in Solaris is not as easy as I was thinking before to use it.
1. Check if Samba is installed. Well, installing Solaris with OEM option, Samba must be installed:

# /usr/sfw/sbin/smbd -V
Version 3.0.35

2. Now we have to create the smb.conf file. I copied the exampled provided with the Solaris installation:

# cd /etc/sfw
# cp smb.conf-example smb.conf

3. Edit the configuratin file. I modified three parts: The allowed ip ranges (my network subnet and localhost):

# This option is important for security. It allows you to restrict
# connections to machines which are on your local network. The
# following example restricts access to two C class networks and
# the "loopback" interface. For more examples of the syntax see
# the smb.conf man page
   hosts allow = 192.168.2. 127.

 Set the Windows workgroup name:

# workgroup = NT-Domain-Name or Workgroup-Name, eg: MIDEARTH
   workgroup = WORKGROUP

And the last one, the shared folder. I want the folder open read-write to everybody, then the guest ok option must be set and the guest only option must be removed! The docs says guest only is discarded with guest ok but in my case guest ok was discarded with both options in the configuration of the folder.

# A publicly accessible directory, read/write to all users. Note that all files
# created in the directory by users will be owned by the default user, so
# any user with access can delete any other user's files. Obviously this
# directory must be writable by the default user. Another user could of course
# be specified, in which case all files would be owned by that user instead.
[public]
   path = /fsshared
   public = yes
   guest ok = yes
   writable = yes
   printable = no
   browseable = yes

4. Test the configuration file:

# /usr/sfw/bin/testparm

Load smb config files from /etc/sfw/smb.conf
Processing section "[homes]"
Processing section "[printers]"
Processing section "[public]"
Loaded services file OK.
Server role: ROLE_STANDALONE
Press enter to see a dump of your service definitions

5. Add a user to the Samba database:

#  /usr/sfw/bin/smbpasswd -a admin
New SMB password:
Retype new SMB password:
Added user admin.

6. The last part must be start the samba service:

# svcadm enable samba

----------------------------------------------------------------------------------------------------------
UPDATE: 2010/09/08
7. Now Samba is working but I can't see the computer in the Windows Explorer (or other smb bowsers): The last step is activate WINS:

# svcadm enable wins

Here you have the old post part (italic letter). NOT NECESSARY.
A new test showed that I don't need to do the next part. At the end 
but in my case it didn't work. After some googling I found this guide related to Solaris u4, and I followed. Samba didn't start and after a new search I found a fault in my smb.conf: I writed /fssaredc instead of the name of my shared folder.
Now I have two commands to start samba:

# svcadm enable samba:smbd
# svcadm enable samba:nmbd

End of the update
----------------------------------------------------------------------------------------------------------
But it's working:

And here comes the bad part. I only get 4.3 MB/s downloading the test file via FTP and 1.9 MB/s via Samba! The V120 is connected to my main router via a non manageable HP ProCurve gigabit switch and the client is a Dell Precsion 390 with the onboard Broadcom gigabit network chipset. I did some checks in the V120 network port configuration but I didn't find anything wrong. Well, I will check later. Anyway I have a Quad GigaSwift PCI card at the mail coming home.

RAM tests.
Thursday arrived some more RAM sticks, 4x1GB ECC registered sticks!!!

I did various test with different amount of memory: 512 MB (one stick that came with the server), 2 GB and 4 GB.

                                 512MB         2 GB        4GB

-----------------------------------------------------------------------

From RAID5 to system disk.      12.8 MB/s    17.8 MB/s   26.9 MB/s

From system disk to RAID5.      11.3 MB/s    18.6 MB/s   21.6 MB/s

From RAID5 to disk 2.           12,5 MB/s    20.9 MB/s   32.0 MB/s

I was expecting a great improvement from 512 MB to 2 Gb, but I also discovered a great rise in performance with 4 GB using the RAID5 array. Yes, ZFS needs a lot of memory!
Note that tests are done by hand and I only did the test one or two times but are enough for comparision. At least for me!

Sun Fire V120 as NAS part 5: Users, rows and some tips

Well, I had some problems with the screen rows (lines) when using the vi editor. Some time surfing the net later I found a good solution: the TERM environment variable. The /etc/profile script only can set this variable only in two ways: sun-color or sun:

(...)
if [ "$TERM" = "" ]
then
        if /bin/i386
        then
                TERM=sun-color
        else
                TERM=sun
        fi
        export TERM
(...)

This could be changed at the Solaris prompt typing:

# TERM=vt100
# export TERM

But with this commands the values are lost when the user logouts. Due all the connections to the server console or terminal will use a vt100 terminal simulator I decided to modify the /etc/profile script as follows:

(...)
if [ "$TERM" = "" ]
then
        if /bin/i386
        then
                TERM=sun-color
        else
                TERM=vt100
        fi
        export TERM
(...)

One problem solved. Go to the second one: with only the root account nobody can login into the server via ftp, smb or something like that. I decided to create an admin account like my linux servers:

# useradd -d /export/home/admin -m -s /bin/ksh -c admin admin
# passwd admin
New password:
(...)

Ok now I can open a FTP session. -d specifies the user folder, -s sets the shell he will use and -c specifies the full name of the user (use " to enclose the name if it has spaces). Obviously the las word is the account name.

Another problem I had was that the man pages were not working. Man seemed installed but not the documentation files. I tried adding the routes to the path but finally the solution is more easy: the package SUNWman was not installed while the package SUNWdoc was perfectly on the hard disk:

# cd /cdrom/sol_10_1009_sparc/Solaris_10/Product
# pkgadd -d . SUNWman

One more thing: How to complete shutdown (well in this server it means 'put the server on standby'):

# shutdown -y -g 0 -i 5

Sun Fire V120 as NAS part 4: Testing ZFS RAID-Z

I wanted to test a RAID5 configuration but having my server booting from a ZFS disk, seems that the Solaris Volume Manager (SVM) databases are not created and I can't use it. Anyway my main project is focuses in ZFS. Start!!!!!
From the SSM software or the format utility I get the disks names (SCSI nodes 1 to 3, scsi bus 2 - onle array disk):

• c2t1d0
• c2t2d0
• c2t3d0

Create a ZFS RAIDZ array is easy, just type:

# zpool create fsshared raidz c2t1d0 c2t2d0 c2t3d0

Now I have a new folder named /fsshared:

# ls
bin       dev       export    kernel    mnt       platform  sbin      usr
boot      devices   fsshared  lib       net       proc      system    var
cdrom     etc       home      lom2      opt       rpool     tmp       vol
# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
fsshared                   89,9K  33,1G  28,0K  /fsshared
rpool                      4,67G  28,6G    97K  /rpool
rpool/ROOT                 3,67G  28,6G    21K  legacy
rpool/ROOT/s10s_u8wos_08a  3,67G  28,6G  3,67G  /
rpool/dump                  512M  28,6G   512M  -
rpool/export                 44K  28,6G    23K  /export
rpool/export/home            21K  28,6G    21K  /export/home
rpool/swap                  512M  28,8G   230M  -

At the list you can see the main boot zfs systems named as root and my new fsshared array. Now it's time to check speed.
Create a new 2 GB file and watch how the array free space decreases:

# cd /fsshared
# mkfile 2g testfile
# ls
testfile
# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
fsshared                   2,00G  31,1G  2,00G  /fsshared
rpool                      4,74G  28,5G    97K  /rpool
rpool/ROOT                 3,74G  28,5G    21K  legacy
rpool/ROOT/s10s_u8wos_08a  3,74G  28,5G  3,74G  /
rpool/dump                  512M  28,5G   512M  -
rpool/export                 44K  28,5G    23K  /export
rpool/export/home            21K  28,5G    21K  /export/home
rpool/swap                  512M  28,8G   230M  -

To check the write speed I copied the testfile to / and measured the time manualy

# cp /fsshared/testfile /
2048 MB  -  161s  - 12.8 MB/s
# rm /fsshared/testfile
# cp /testfile /fsshared
2048 MB  -  181 s  -  11.3 MB/s

This is not a good result and I'm start to think that the small amount of  RAM in the server is one of the causes (with the slow CPU of course). Writing from the main disk to the array is slower than the reverse option. This speed is not fantastic but it's a write speed, now I need a gigabit card to check read speed. If read performance is ok, the V120 will be enough as NAS.

A new test copying the same file from the array to the disk 2 of the server:

# zpool create disc2 c1t1d0
# cp /fsshared/testfile /disc2
2048 MB  -  164 s  -  12,5 MB/s

Not better results here... And from the main disk to the second one of the server? And reverse?

# rm /disc2/testfile
# cp /testfile /disc2
2048 MB  -  121s  - 16.9 MB/s

Wow... Maybe the ZFS RAID calculations are too heavy for this server.
 Ok. Next must be test NFS, FTP and SMB trasfer results.

Sun Fire V120 as NAS part 3: Enabling ssh root login

Ok. I'm tired. This server is too loud to work behind it. Because I don't want to create users yet and I want to admin the server from another room I decided to enable login with the root account, like my other linux servers

Very easy to do, from the serial console logged as root: edit /etc/ssh/sshd_config and change:

PermitRootLogin no >>PermitRootLogin yes

SPECIAL TIP!!!! I can't use the vi editor from the console as I can't see all the lines of the file. I uploaded to my main server via ftp, modified it in my workstation an then it downloaded again.

# svcadm disable svc:/network/ssh:default
# ftp 192.168.2.254
(...)
# svcadm enable svc:/network/ssh:default