Building a distro kernel
First things first.
As always.. a warning.
Re-building and re-booting a distro kernel is conceptually, and in practice, a fairly easy thing to do. But it is also very easy to mess up your system with a single simple typo, with the practical consequences that you may need to reinstall your system. Being unable to re-boot a shared system where someone else has data that wasn’t backed up is a “most unpleasant” feeling. Be Wary
First. We strongly urge you to practice these steps on a victim system, where you have no problems what-so-ever of completely reinstalling the system. A “victim system” means there is nothing of importance on the system. No important data. No critical users or applications. No key work-in-progress.
Second. These steps are provided simply as a step-by-step guide of how several of us re-built the kernel in the labs on freshly installed victim systems. We most-certainly are not the official source for the right steps, nor do we claim this will work for everyone, and you should not expect any level of support from us on this.
Third. This should be intuitive, but amazingly-enough often isn’t clearly understood. So we’ll say it here explicitly. If you re-build your own kernel and you use that kernel, you have invalidated any hope for “official support” from your service provider while using that kernel.
SLES 11. Find the kernel-source “src.rpm”
On SLES 11, there are two DVD iso images.
SLES-11-DVD-ppc64-GM-DVD1.iso
SLES-11-DVD-ppc64-GM-DVD2.iso
If you mount the second DVD, it has the “source” files. Here we assume you’ve downloaded the DVD iso images to a local directory, let’s say: /usr/local/iso
# cd /usr/local/iso
# ls
SLES-11-DVD-ppc64-GM-DVD1.iso SLES-11-DVD-ppc64-GM-DVD2.iso
# mkdir /dvd
# mount -t iso9660 -o loop SLES-11-DVD-ppc64-GM-DVD2.iso /dvd
# find /dvd -name kernel-source*
/dvd/suse/src/kernel-source-2.6.27.19-5.1.src.rpm
You’ll need the *.src.rpm file.
This rpm cleverly doesn’t show up as installed if you do a rpm -qa | grep kernel-source. To tell whether you have installed the package, look under /usr/src/packages/SPECS. If there’s nothing there, then you have not installed the correct rpm.
Install the kernel-source “src.rpm”
You want the “src.rpm”, not just the “.rpm” file.
cd /dvd/suse/src
rpm -i kernel-source-2.6.27.19-5.1.src.rpm
Copy the “spec” file from SOURCES over to the SPECS sub-directory.
cd /usr/src/packages/SOURCES
cp kernel-ppc64.spec ../SPECS
If the kernel-ppc64.spec file does not exist, you will first have to build the platform specific spec files…
# cd /usr/src/packages/SOURCES
# ls
kernel-binary.spec.in kernel-module-subpackage kernel-source.rpmlintrc kernel-spec-macros
kernel-docs.spec.in kernel-source.changes kernel-source.spec.in kernel-syms.spec.in
# ./mkspec
and then copy the kernel-ppc64.spec file to ../SPECS as previously stated.
Edit the ppc64 “spec” file to add a unique identifier
cd /usr/src/packages/SPECS
vi kernel-ppc64.spec
Change the Release identifier adding an identifier denoting your new kernel build (i.e. from “5” to something like “5.001”)
Release: 5
to
Release: 5.001
Save and file.
Note the current kernel level which will be used to compare the version after the rebuild/successful boot.
# uname -r
2.6.27-19-5-ppc64
rpmbuild -ba kernel-ppc64.spec
rpmbuild is the clever step that packages all of the patches that constitute the “whole” of the kernel.
# rpmbuild -ba kernel-ppc64.spec
error: Failed build dependencies:
sparse is needed by kernel-ppc64-2.6.27.19-5.001.ppc64
fdupes is needed by kernel-ppc64-2.6.27.19-5.001.ppc64
kernel-dummy is needed by kernel-ppc64-2.6.27.19-5.001.ppc64
Now you need the SLES 11 SDK iso image.
mkdir /sdk
mount -t iso9660 -o loop /usr/local/iso/SLE-11-SDK-DVD-ppc64-GM-Media1.iso /sdk
cd /sdk/suse/ppc64/
rpm -i sparse-0.4.1.git1-1.21.ppc64.rpm
rpm -i fdupes-1.40-42.22.ppc64.rpm
Try again (noticing that there is no such thing as a kernel-dummy rpm file).
# rpmbuild -ba kernel-ppc64.spec
error: Failed build dependencies:
kernel-dummy is needed by kernel-ppc64-2.6.27.19-5.001.ppc64
Eh?
From the README.SUSE file in /usr/src/packages/SOURCES
* kernel-dummy
This package is relevant inside the SUSE build system only. We use
it to synchronize release numbers among the kernel packages. When
building packages locally, the kernel-dummy package can safely be
ignored.
Ok. Comment out that line.
%if ! 0%{?opensuse_bs}
# BuildRequires: kernel-dummy
%endif
We’re ready to go. But before we start, we want to modify the “spec” file to compile in parallel.
vi kernel-ppc64.spec
After the “build_flavor” line, add a line to define “jobs” to be the number of available CPUs.
%define build_flavor “ppc64”
%define jobs %(cat /proc/cpuinfo | grep processor | wc -l)
Try again.
rpmbuild -ba kernel-ppc64.spec
<lots of compilation messages>
<eventually…>
Wrote: /usr/src/packages/SRPMS/kernel-ppc64-2.6.27.19-5.001.nosrc.rpm
Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-2.6.27.19-5.001.ppc64.rpm
Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-base-2.6.27.19-5.001.ppc64.rpm
Wrote: /usr/src/packages/RPMS/ppc64/kernel-ppc64-extra-2.6.27.19-5.001.ppc64.rpm
( or rpmbuild -ba –define ‘jobs N’ kernel-ppc64.spec where N = cores can also be used)
Building in parallel
Quick tangent.
To confirm you’re building in parallel, in a separate window fire up “top” and press the “1” key. Here’s an example of an 8 core (16 CPUs with SMT on) Power 6 system running SLES 11.
top – 09:51:50 up 17:30, 2 users, load average: 20.87, 8.87, 3.80
Tasks: 277 total, 19 running, 256 sleeping, 0 stopped, 2 zombie
Cpu0 : 41.5%us, 3.4%sy, 0.0%ni, 12.4%id, 6.1%wa, 0.0%hi, 0.0%si, 36.5%st
Cpu1 : 52.8%us, 2.1%sy, 0.0%ni, 8.4%id, 0.0%wa, 0.0%hi, 0.3%si, 36.4%st
Cpu2 : 55.7%us, 1.3%sy, 0.0%ni, 7.4%id, 0.0%wa, 0.0%hi, 0.0%si, 35.6%st
Cpu3 : 40.4%us, 2.4%sy, 0.0%ni, 21.0%id, 0.0%wa, 0.0%hi, 0.0%si, 36.2%st
Cpu4 : 45.3%us, 3.7%sy, 0.0%ni, 13.9%id, 0.0%wa, 0.0%hi, 0.3%si, 36.8%st
Cpu5 : 49.2%us, 1.6%sy, 0.0%ni, 12.4%id, 0.0%wa, 0.0%hi, 0.0%si, 36.8%st
Cpu6 : 48.7%us, 2.1%sy, 0.0%ni, 4.7%id, 0.5%wa, 0.0%hi, 0.0%si, 43.9%st
Cpu7 : 46.8%us, 1.8%sy, 0.0%ni, 6.3%id, 0.0%wa, 0.0%hi, 0.0%si, 45.0%st
Cpu8 : 46.1%us, 3.2%sy, 0.0%ni, 10.0%id, 0.0%wa, 0.0%hi, 0.0%si, 40.8%st
Cpu9 : 48.2%us, 2.4%sy, 0.0%ni, 8.9%id, 0.0%wa, 0.0%hi, 0.0%si, 40.5%st
Cpu10 : 50.4%us, 1.8%sy, 0.0%ni, 6.8%id, 2.6%wa, 0.3%hi, 0.0%si, 38.1%st
Cpu11 : 45.3%us, 2.4%sy, 0.0%ni, 10.8%id, 2.4%wa, 0.0%hi, 0.0%si, 39.2%st
Cpu12 : 51.7%us, 2.6%sy, 0.0%ni, 9.5%id, 0.0%wa, 0.0%hi, 0.0%si, 36.1%st
Cpu13 : 41.2%us, 3.4%sy, 0.0%ni, 18.7%id, 0.0%wa, 0.0%hi, 0.0%si, 36.7%st
Cpu14 : 55.5%us, 1.8%sy, 0.0%ni, 10.3%id, 0.0%wa, 0.3%hi, 0.0%si, 32.1%st
Cpu15 : 39.4%us, 2.9%sy, 0.0%ni, 24.3%id, 0.0%wa, 0.0%hi, 0.0%si, 33.4%st
Mem: 130335424k total, 19881792k used, 110453632k free, 1300288k buffers
Swap: 4096384k total, 0k used, 4096384k free, 17655936k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15460 root 20 0 192m 180m 9664 R 64 0.1 0:07.15 cc1
20488 root 20 0 67520 61m 9664 R 30 0.0 0:01.13 cc1
21028 root 20 0 50304 39m 9664 R 13 0.0 0:00.48 cc1
21114 root 20 0 50176 45m 9664 R 10 0.0 0:00.39 cc1
21236 root 20 0 50112 44m 9664 R 7 0.0 0:00.27 cc1
21251 root 20 0 49280 33m 9664 R 6 0.0 0:00.24 cc1
21332 root 20 0 49216 36m 8384 R 5 0.0 0:00.19 cc1
21386 root 20 0 49216 30m 9664 R 5 0.0 0:00.19 cc1
21313 root 20 0 36160 28m 9472 R 4 0.0 0:00.17 cc1
21267 root 20 0 49216 29m 9600 R 4 0.0 0:00.16 cc1
21374 root 20 0 32832 27m 9664 R 3 0.0 0:00.12 cc1
21401 root 20 0 32768 25m 7360 R 2 0.0 0:00.09 cc1
21346 root 20 0 32768 24m 7360 R 2 0.0 0:00.08 cc1
21425 root 20 0 32832 20m 7360 R 2 0.0 0:00.06 cc1
3 root RT -5 0 0 0 S 0 0.0 0:00.07 migration/0
90 root 20 0 0 0 0 S 0 0.0 0:06.30 pdflush
15481 root 20 0 16512 11m 4096 S 0 0.0 0:00.04 as
18123 root 20 0 7232 4288 3072 S 0 0.0 0:00.01 make
18300 root 20 0 7232 4352 3072 S 0 0.0 0:00.02 make
19403 root 20 0 7232 4416 3072 D 0 0.0 0:00.01 make
19487 root 20 0 7232 4224 3072 S 0 0.0 0:00.01 make
19611 root 20 0 7232 4224 3072 S 0 0.0 0:00.01 make
1 root 20 0 3648 1088 832 S 0 0.0 0:01.83 init
Install the newly built kernel.
So then we can install the new kernel rpm.
# cd /usr/src/packages/RPMS/ppc64
# rpm -i kernel-ppc64-2.6.27.19-5.001.ppc64.rpm
Setting up /lib/modules/2.6.27.19-5.001-ppc64
We recommend that you reboot to this kernel and confirm that it boots correctly.
We recommend that you do NOT change the default boot kernel, leaving the test kernel as an alternative choice.
Adding patches into the kernel build process.
There are several steps required to add a patch set into the build process.
Prepare the patch set.
The patch set it prepared by creating a patch directory and adding the patches to the directory. Once the patches are added to the directory, create a tar/bzip file of the patches.
tar -cjf patches.osjitter.tar.bz2 patches.osjitter
Add (copy) the patch set to the /usr/src/packages/SOURCES directory.
cp -p patches.osjitter.tar.bz2 /usr/src/packages/SOURCES/
Edit the kernel-ppc64.spec file and specify the patch set in 2 places.
1) Insert the patch set name after the last patch in the kernel-ppc64.spec file. Note to increment the Source number (Source121 is used for this case).
Source113: patches.kabi.tar.bz2
Source120: kabi.tar.bz2
Source121: patches.osjitter.tar.bz2 <— add the patch set
BuildRoot: %{_tmppath}/%{name}-%{version}-build
ExclusiveArch: ppc ppc64
# These files are found in the kernel-source
2) In the same spec file, find the %setup command and add the source number of the patch set to the reference line.
In this case, “-a 121” is added at the end of the %setup script.
…
# Unpack all sources and patches
%setup -q -c -T -a 0 -a 100 -a 101 -a 102 -a 103 -a 104 -a 105 -a 107 -a 108 -a 109 -a 110 -a 111 -a 112 -a 113 -a 120 -a 121 <— -a 121 is added
mkdir -p %kernel_build_dir
…
Update the ../packages/SOURCE/series.conf file adding the specific names of each patch. The order of placement should be considered.
…
patches.fixes/oprofile_bios_ctr.patch
patches.trace/utrace-core
patches.osjitter/patch_osjitter
…
The patch file has now been added to the kernel build process. The final step is to build the kernel using the rpmbuild command.
The patch can be verified by creating an output log of the build process and then examining the log to make sure the patch has been applied. In our case, we see reference to the patch as follows:
…
+ tar -xf –
+ STATUS=0
+ ‘[‘ 0 -ne 0 ‘]’
+ /usr/bin/bzip2 -dc /usr/src/packages/SOURCES/patches.osjitter.tar.bz2
+ tar -xf –
+ STATUS=0
+ ‘[‘ 0 -ne 0 ‘]’
…
…
…
+ patch -s -F0 -E -p1 –no-backup-if-mismatch -i ../patches.fixes/oprofile_bios_ctr.patch
+ patch -s -F0 -E -p1 –no-backup-if-mismatch -i ../patches.trace/utrace-core
+ patch -s -F0 -E -p1 –no-backup-if-mismatch -i ../patches.osjitter/patch_osjitter
+ patch -s -F0 -E -p1 –no-backup-if-mismatch -i ../patches.fixes/ia64-configure-HAVE_UNSTABLE_SCHED_CLOCK-for-SGI_SN.patch
+ patch -s -F0 -E -p1 –no-backup-if-mismatch -i ../patches.fixes/kvm-ioapic.patch
+ patch -s -F0 -E -p1 –no-backup-if-mismatch -i ../patches.fixes/kvm-macos.patch
Building a recent mainline kernel
For completeness, here’s the quick steps for building a mainline kernel.
Go to kernel.org, download a stable or recent kernel. For example, at the time this is being updated, the current stable kernel we recommend is 2.6.32.12. We would suggest keeping within 2.6.32 stable kernels.
In this example, we download the 2.6.32.12 mainline kernel
http://www.kernel.org/pub/linux/kernel/v2.6/testing/linux-2.6.32.12.tar.bz2
Quick steps are available here: http://kernelnewbies.org/KernelBuild
# cd /root
# tar -jxf linux-2.6.32.12.tar.bz2
# cd linux-2.6.32.12
# make pseries_defconfig
You should update and/or verify the .config settings using make menuconfig command as follows:
make menuconfig
<select Kernel options>
<change Page Size to 64k page size – default is 4k>
<change Maximum zone order to 9 – default is 13>
<select Exit>
<select Processor support>
<change Maximum number of CPUs to 4x number of cores – you can specify 256>
<select Exit>
<select General setup>
<select Kernel Performance Events and Counters>
<verify Kernel performance events and counters is turned on>
<deselect/verify Kernel performance counters (old config option)
<select Exit>
<select Exit>
<select Exit>
<Yes, save the new config>
If you wish, you can confirm the new settings in the .config file:
…
CONFIG_PPC_HAS_HASH_64K=y
# CONFIG_PPC_4K_PAGES is not set
# CONFIG_PPC_16K_PAGES is not set
CONFIG_PPC_64K_PAGES=y
# CONFIG_PPC_256K_PAGES is not set
CONFIG_FORCE_MAX_ZONEORDER=9
# CONFIG_PPC_SUBPAGE_PROT is not set
…
CONFIG_NR_CPUS=256
…
#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
CONFIG_EVENT_PROFILE=y
# CONFIG_PERF_COUNTERS is not set
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
For the build..
# make all
# su -c “make modules_install install”
If running under ABAT, use
# boot-to –kernel /boot/vmlinux-2.6.32.12 –initrd /boot/initrd-2.6.32.12.img