OpenBSD i8254 trap
I have found that OpenBSD 7.3 produces fatal trap:
wdc_atapi_start: not ready, st = 50 fatal protection fault in supervisor mode
trap type 4 code 0 rip ffffffff810089d9 cs 8 rflags 10282 cr2 2377bf000 cpl 6 rsp ffff8000150a2f50
gsbase 0xffffffff818fbff0 kgsbase 0x0
panic: trap type 4, code=0, pc=ffffffff810089d9
syncing disks...6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 giving up
dump to dev 17,1 not possible
rebooting...
Please look here for complete explanation:
- https://marc.info/?l=openbsd-bugs&m=168564039922088&w=2 Text below is kept for reference because it contains lot of useful information, how to debug OpenBSD kernel and installation.
This occurs under libvirt/KVM using IDE CD emulation and Virtio-BLK for disk.
Already reported here:
When installing sets from CD (install iso is: install73.iso )
How to get messages?
-
when
bootprompt appears press ESC or SPACE to stop countdown -
then connect from regular console to libvirt console:
# optional: log all messages: script openbsd.log virsh console --force openbsd7.3 -
now back in VGA console issue this last command:
- see https://man.openbsd.org/man8/amd64/boot.8 for manual
set tty com0 -
this should be last message in VGA and you can now fully control OpenBSD from serial console
Here are boot messages:
>> OpenBSD/amd64 CDBOOT 3.55
boot> boot
cannot open cd0a:/etc/random.seed: No such file or directory
booting cd0a:/7.3/amd64/bsd.rd: ... redacted ..
entry point at 0xffffffff81001000
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2023 OpenBSD. All rights reserved. https://www.OpenBSD.org
OpenBSD 7.3 (RAMDISK_CD) #1063: Sat Mar 25 10:41:49 MDT 2023
[email protected]:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
real mem = 1056800768 (1007MB)
avail mem = 1020813312 (973MB)
random: good seed from bootblocks
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf59c0 (9 entries)
bios0: vendor SeaBIOS version "rel-1.15.0-0-g2dd4b9b-rebuilt.opensuse.org" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: tables DSDT FACP APIC WAET
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Celeron(R) CPU N3450 @ 1.10GHz, 360.80 MHz, 06-5c-09
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,CX16,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,SHA,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: apic clock running at 999MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
acpicpu at acpi0 not configured
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
"Intel 82371SB ISA" rev 0x00 at pci0 dev 1 function 0 not configured
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
pciide0: channel 1 disabled (no drives)
"Intel 82371AB Power" rev 0x03 at pci0 dev 1 function 3 not configured
vga1 at pci0 dev 2 function 0 "Red Hat QXL Video" rev 0x05
vga1: aperture needed
wsdisplay1 at vga1 mux 1
wsdisplay1: screen 0 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 52:54:00:f4:0c:46
virtio0: msix shared
uhci0 at pci0 dev 4 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 11
uhci1 at pci0 dev 4 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci2 at pci0 dev 4 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 10
ehci0 at pci0 dev 4 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
virtio1 at pci0 dev 5 function 0 "Qumranet Virtio Console" rev 0x00
virtio1: no matching child driver; not configured
virtio2 at pci0 dev 6 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus1 at vioblk0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 20480MB, 512 bytes/sector, 41943040 sectors
virtio2: msix shared
virtio3 at pci0 dev 7 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
virtio3: no matching child driver; not configured
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at mainbus0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
wskbd0: connecting to wsdisplay1
softraid0 at root
scsibus2 at softraid0: 256 targets
root on rd0a swap on rd0b dump on rd0b
WARNING: CHECK AND RESET THE DATE!
erase ^?, werase ^W, kill ^U, intr ^C, status ^T
Welcome to the OpenBSD/amd64 7.3 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? i
...
Problem:
-
we need to add
DDBandDDB_SAFE_CONSOLEto RAMDISK (rd kernel). -
the process is very special because basically there is added filesystem and result is gzippped
-
This
Makefilelooks promising:/usr/src/distrib/amd64/ramdisk_cd/Makefile- and target of our interest is
bsd.gz
-
here are interesting parts:
bsd.gz: bsd.rd
objcopy -g -x -R .comment -R .SUNW_ctf \
-K rd_root_size -K rd_root_image \
bsd.rd bsd.strip
gzip -9cn bsd.strip > bsd.gz
bsd.rd: mr.fs bsd
cp bsd bsd.rd
rdsetroot bsd.rd mr.fs
bsd:
cd ${.CURDIR}/../../../sys/arch/${MACHINE}/compile/${RAMDISK} && \
su ${BUILDUSER} -c '${MAKE} config && ${MAKE} clean && exec ${MAKE} ${MFLAGS}'
cp -p ${.CURDIR}/../../../sys/arch/${MACHINE}/compile/${RAMDISK}/obj/bsd bsd
mr.fs: instbin
rm -rf [email protected]
install -d -o root -g wheel [email protected]
mtree -def ${MTREE} -p [email protected] -u
CURDIR=${.CURDIR} OBJDIR=${.OBJDIR} OSrev=${OSrev} \
TARGDIR=$@.d UTILS=${UTILS} RELEASEDIR=${RELEASEDIR} \
sh ${UTILS}/runlist.sh ${LISTS}
rm [email protected]/instbin
makefs ${MRMAKEFSARGS} $@ [email protected]
instbin.mk instbin.cache instbin.c: instbin.conf
crunchgen -E -D ${.CURDIR}/../../.. -L ${DESTDIR}/usr/lib \
-c instbin.c -e instbin -m instbin.mk instbin.conf
instbin: instbin.mk instbin.cache instbin.c
${MAKE} ${MFLAGS} -f instbin.mk SRCLIBDIR=${.CURDIR}/../../../lib all
instbin.conf: ${LISTS}
awk -f ${UTILS}/makeconf.awk ${LISTS} > instbin.confWARNING! Important thing is to keep ROOTSIZE array large enough to be edited.
-
We can find in
/usr/src/sys/arch/amd64/conf/RAMDISK_CD:option MINIROOTSIZE=7360 -
it is in sector as we can see on
/usr/src/sys/dev/rd.c#ifndef MINIROOTSIZE #define MINIROOTSIZE 512 #endif #define ROOTBYTES (MINIROOTSIZE << DEV_BSHIFT) /* * This array will be patched to contain a file-system image. * See the program: src/distrib/common/rdsetroot.c */ u_int32_t rd_root_size = ROOTBYTES; char rd_root_image[ROOTBYTES] = "|This is the root ramdisk!\n";
-
the
MINIROOTSIZEis passed to cc:cc -g -Werror -Wall ... -DRAMDISK_HOOKS -DMINIROOTSIZE="0x1cc0" -
here is debug output from rdsetroot (option
-d):cp bsd bsd.rd rdsetroot -d bsd.rd mr.fs rd_root_size_off: 0x1770 rd_root_image_off: 0x1780 rd_root_size val: 0x398000 (7360 blocks) copying root image... ...copied 3768320 bytes objcopy -g -x -R .comment -R .SUNW_ctf -K rd_root_size -K rd_root_image bsd.rd bsd.strip gzip -9cn bsd.strip > bsd.gz -
currently it is tip-top, we can see:
$ printf '%d\n' 0x1cc0 7360 $ expr 7360 \* 512 3768320 $ ls -g /usr/src/distrib/amd64/ramdisk_cd/obj/mr.fs -rw-r--r-- 1 wobj 3768320 May 28 09:07 /usr/src/distrib/amd64/ramdisk_cd/obj/mr.fs
-
what is
instbin? It is like Linux BusyBox - bundled binary containing all base commands -
reference https://man.openbsd.org/release
So what I'm doing:
-
install these sets:
sysandsrc -
modify this file:
diff -u /usr/src/sys/arch/amd64/conf/RAMDISK_CD.orig /usr/src/sys/arch/amd64/conf/RAMDISK_CD --- /usr/src/sys/arch/amd64/conf/RAMDISK_CD.orig Fri May 26 19:22:21 2023 +++ /usr/src/sys/arch/amd64/conf/RAMDISK_CD Fri May 26 19:23:27 2023 @@ -3,6 +3,10 @@ machine amd64 maxusers 4 +# HP - added +option DDB +option DDB_SAFE_CONSOLE + option SMALL_KERNEL option NO_PROPOLICE option BOOT_CONFIG
-
do not build kernel - it needs to get few ramdisk variables to build properly...
cd /usr/src/sys/arch/amd64/conf config RAMDISK_CD # Do not buld kernel yet! - it needs few special ramdisk variables...
-
Now tricky part - short circuit to make kernel with ramdisk...
cd /usr/src make obj # workaround for error "-lstubs not found" cd /usr/src/distrib/special/libstubs make # now back to work... cd /usr/src/distrib/amd64/ramdisk_cd/ # don't do this BUILDUSER=root on production! make bsd.gz BUILDUSER=root
-
now
obj/bsd.gzis your own(!) gzipped kernel with ramdisk!
And voila!!!
#### NEW TRACE INCLUDED WITH REGISTER ####
OpenBSD 7.3-stable (RAMDISK_CD) #0: Sun May 28 09:15:59 CEST 2023
[email protected]:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
...
Installing bsd.rd 100% |**************************| 4547 KB 00:01
Installing bsd.rd 100% |**************************| 4547 KB 00:01
Installing base73.tgz 3% | | 14208 KB 02:07 ETA
wdc_atapi_start: not ready, st = 50
kernel: protection fault trap, code=0
Stopped at wdcstart+0x19: movl 0x58(%rdi),%eax
ddb> trace
wdcstart(ffff80000007c168,ffff80000007c168,0,ffff80002171a920,10282,8)
at wdcstart+0x19
wdc_atapi_the_machine(ffff80000007c168,fffffd807eb0bea8,2,ffff80000007c168,ffff80000007c168,fffffd807eb0bea8)
at wdc_atapi_the_machine+0x14a
wdc_atapi_intr(ffff80000007c168,fffffd807eb0bea8,1,ffff80000007c168,fffffd807eb0bea8,ffff80000007c168)
at wdc_atapi_intr+0x47
wdcintr(ffff80000007c168,ffff80000007c168,0,0,6,1)
at wdcintr+0xae
intr_handler(ffff80002171aa70,ffff800000065500,ffff800000065680,0,ffffffff81212216,ffff80002171aa60)
at intr_handler+0x26
Xintr_ioapic_edge14_untramp(0,206,602,0,ffffffff8192c390,6)
at Xintr_ioapic_edg e14_untramp+0x18f
spllower(0,0,1,0,0,ffff80002171ab20)
at spllower+0x36
scsi_pending_start(ffffffff8192c390,ffff800000024dc0,ffff800000024dc0,ffff800000024d00,ffffffff812234c4,ffff80002171ab70)
at scsi_pending_start+0x2f
scsi_xsh_runqueue(ffff800000024d00,ffff800000024d00,0,ffff800000024d00,ffff800000118a00,ffff800000023970)
at scsi_xsh_runqueue+0x2b
scsi_xsh_add(ffff800000023970,ffff800000023970,fffffd807e459018,ffff800000023800,fffffd807e459108,6)
at scsi_xsh_add+0x80
cdstrategy(fffffd807e459018,fffffd807e459018,fffffd807e459018,fffffd807e459018,fffffd807e459108,150)
at cdstrategy+0xe4
spec_strategy(ffff80002171ac70,ffff80002171ac70,fffffd807e459018,1f,ffffffff81142a01,ffff80002171ac60)
at spec_strategy+0x3f
VOP_STRATEGY(fffffd807df8e1b0,fffffd807e459018,fffffd807e459018,fffffd807df8e1b0,ffffffff81140d6b,ffff80002171ac90)
at VOP_STRATEGY+0x3c
cd9660_strategy(ffff80002171ace0,ffff80002171ace0,fffffd807e459018,4,fffffd806f96e6e8,fffffd806f96e6e8)
at cd9660_strategy+0xce
VOP_STRATEGY(fffffd806f96e6e8,fffffd807e459018,fffffd807e459018,fffffd806f96e6e8,ffffffff81140d6b,ffff80002171ad00)
at VOP_STRATEGY+0x3c
bio_doread(fffffd806f96e6e8,1f89,800,4,ffff800000148100,20)
at bio_doread+0x67
breadn(fffffd806f96e6e8,1f69,800,ffff800000148000,ffff800000148100,20)
at breadn+0x71
cd9660_read(ffff80002171ae60,ffff80002171ae60,fffffd807edfe350,fffffd806f96e6e8,0,ffff80002171af90)
at cd9660_read+0x12b
VOP_READ(fffffd806f96e6e8,ffff80002171af90,0,fffffd807f7bff08,fffffd807f7bff08,ffff800000000000)
at VOP_READ+0x31
vn_read(fffffd807edfe350,ffff80002171af90,0,fffffd807edfe350,fffffd807edfe350,20000)
at vn_read+0xad
dofilereadv(ffff8000216cdb40,3,ffff80002171af90,0,ffff80002171b050,ffff8000216cdb40)
at dofilereadv+0x62
sys_read(ffff8000216cdb40,ffff80002171b000,ffff80002171b050,ffff8000216cdb40,b800,2a9270800)
at sys_read+0x43
syscall(ffff80002171b0b0,ffff80002171b0b0,0,ffff8000216cdb40,0,0)
at syscall+0x201
Xsyscall(0,3,0,3,0,28b087000) at Xsyscall+0x128
end of kernel
End trace frame: 0x6f829d5a1800, count: -24
ddb> show registers
# rdi should contain new address from chp = xfer->chp
rdi 0x90d05d78134f6961 ### it should be address!
rsi 0xfffffd807eb0bea8
rbp 0xffff80002171a930
rbx 0xffff80002171a948
rdx 0xffffffff8192c1f8 wdc_xfer_pool+0x40
rcx 0xfffffd807eb0bed8
rax 0xffff800000028270
r8 0xffffffff81944d58 cleancache+0x48
r9 0x51
r10 0
r11 0
r12 0x1
r13 0x30
r14 0xfffffd807eb0bea8
r15 0xffff80000007c168
rip 0xffffffff8100921a wdcstart+0x19
cs 0x8
rflags 0x10282 __ALIGN_SIZE+0xf282
rsp 0xffff80002171a920
ss 0
wdcstart+0x19: movl 0x58(%rdi),%eax
ddb> show proc
PROC (ftp) pid=159390 stat=onproc
flags process=100003<CONTROLT,EXEC,PLEDGE> proc=0
pri=17, usrpri=50, nice=20
forw=0x0, list=0xffff8000216cd318,0xffff8000216cd898
process=0xffff8000217247e8 user=0xffff800021716000, vmspace=0xfffffd807e9e3
9e0
estcpu=0, cpticks=0, pctcpu=0.49
user=0, sys=0, intr=0
Here is thread (with older stacktrace):
Unfortunately my web client messed up subject in case of 2 messages so here are individual links:
- https://marc.info/?l=openbsd-bugs&m=168510238127493&w=2
- https://marc.info/?l=openbsd-bugs&m=168512767809137&w=2
Similarly sender was screwed by web client so here are two lists:
First let's see where that message appears:
- message:
wdc_atapi_start: not ready, st = 50 - using:
fgrep -rn -A 3 -B 3 'wdc_atapi_start: not ready, st'
src/sys/dev/atapiscsi/atapiscsi.c-738- struct ata_drive_datas *drvp = &chp->ch_drive[xfer->drive];
src/sys/dev/atapiscsi/atapiscsi.c-739-
src/sys/dev/atapiscsi/atapiscsi.c-740- if (timeout) {
src/sys/dev/atapiscsi/atapiscsi.c:741: printf("wdc_atapi_start: not ready, st = %02x\n",
src/sys/dev/atapiscsi/atapiscsi.c-742- chp->ch_status);
src/sys/dev/atapiscsi/atapiscsi.c-743-
src/sys/dev/atapiscsi/atapiscsi.c-744- sc_xfer->error = XS_TIMEOUT;
So let's try to decipher:
wdc_atapi_start: not ready, st = 50
kernel: protection fault trap, code=0
Stopped at wdcstart+0x19: movl 0x58(%rdi),%eax
By following:
Found using mc that affected function wdstart is in:
/usr/src/sys/dev/ic/wdc.c
cd /sys/arch/amd64/compile/RAMDISK_CD
# added -S to see also source code - does not work as epxected...
objdump -dlrS obj/wdc.o > /root/wdc.dis2Now we have to find where is offset 0x19 using OpenBSD guide:
-
start address is:
0000000000000df1 <wdcstart>: -
add 0x19 to it:
printf '0x%x\n' $((0xdf1 + 0x19)) 0xe0a
-
so look at
/root/wdc.disagain:
0000000000000df1 <wdcstart>:
wdcstart():
/usr/src/sys/dev/ic/wdc.c:876
void
wdcstart(struct channel_softc *chp)
{
struct wdc_xfer *xfer;
splassert(IPL_BIO);
df1: 55 push %rbp
df2: 48 89 e5 mov %rsp,%rbp
df5: 57 push %rdi
df6: 57 push %rdi
/usr/src/sys/dev/ic/wdc.c:882
/* is there a xfer ? */
if ((xfer = TAILQ_FIRST(&chp->ch_queue->sc_xfer)) == NULL) {
return;
}
df7: 48 8b 87 e0 04 00 00 mov 0x4e0(%rdi),%rax
dfe: 48 8b 30 mov (%rax),%rsi
e01: 48 85 f6 test %rsi,%rsi
e04: 74 4b je e51 <wdcstart+0x60>
/usr/src/sys/dev/ic/wdc.c:887
Line 886: /* adjust chp, in case we have a shared queue */
Line 887: chp = xfer->chp;
e06: 48 8b 7e 08 mov 0x8(%rsi),%rdi
/usr/src/sys/dev/ic/wdcvar.h
#define WDCF_ACTIVE 0x01 /* channel is active */
/usr/src/sys/dev/ic/wdc.c:889
Line 889: if ((chp->ch_flags & WDCF_ACTIVE) != 0 ) {
Line 890: return; /* channel already active */
line 891: }
e0a: 8b 47 58 mov 0x58(%rdi),%eax
e0d: a8 01 test $0x1,%al
e0f: 75 40 jne e51 <wdcstart+0x60>
I will now switch build machine and use sources from CVS (so I can easily make diffs of my changes):
-
following https://www.openbsd.org/anoncvs.html
-
created this
/etc/doas.conf:permit nopass :wheel -
added myself (
ssduser) to these groups:-
wsrc- to be able to modify/usr/src, for example withdoas /usr/sbin/usermod -G wsrc $USER -
wobj- to be able to build - write to/usr/obj, for example withdoas /usr/sbin/usermod -G wobj $USER -
wheel- to be able to become root withdoas sh
-
-
added
async(all data written asynchronously) andnoatime(like Linuxrelatime- update access time only when ctime or mtime changed to reduce disk I/O) to/etc/fstabNOTE: the
softdepwas underperforming on my bare-metal with SSD (when building kernel the system CPU is around 30%). I don't mind loosing data so I rather useasyncinstead ofsoftdep. In such case system cpu dropped to 5% while building kernel. -
cleaned up existing src and obj:
rm -rf /usr/src/* /usr/obj/* -
checkout latest stable (
-z3uses compression to transfer data faster):cd /usr cvs -z3 -d [email protected]:/cvs checkout -rOPENBSD_7_3 -P src
-
we can also add to
~/.cvsrc:cvs -z3 -
making backup of pristine 7.3-stable sources (it never hurts):
cd / tar cvzf ~/cvs-src-stable-7.3.tar.gz ./usr/src/
-
first modification:
cd /usr/src && cvs -qz3 diff:
? .gitignore
Index: sys/arch/amd64/conf/RAMDISK_CD
===================================================================
RCS file: /cvs/src/sys/arch/amd64/conf/RAMDISK_CD,v
retrieving revision 1.200
diff -u -p -r1.200 RAMDISK_CD
--- sys/arch/amd64/conf/RAMDISK_CD 6 Apr 2022 13:23:58 -0000 1.200
+++ sys/arch/amd64/conf/RAMDISK_CD 28 May 2023 06:41:55 -0000
@@ -3,6 +3,10 @@
machine amd64
maxusers 4
+# HP - added
+option DDB
+option DDB_SAFE_CONSOLE
+
option SMALL_KERNEL
option NO_PROPOLICE
option BOOT_CONFIGLet's build RAMDISK kernel
cd /usr/src
make obj
# workaround for error "-lstubs not found"
cd /usr/src/distrib/special/libstubs
make
# now back to work...
cd /usr/src/distrib/amd64/ramdisk_cd/
# don't do this BUILDUSER=root on production!
# I was unable to do this as non-root without full release (even when using noperm /dest):
doas make bsd.gz BUILDUSER=root
# how to rebuild kernel:
How to rebuild modified kernel:
doas sh
cd /usr/src/distrib/amd64/ramdisk_cd/
rm obj/bsd*
doas make bsd.gz BUILDUSER=rootAnd here is artificial panic so we can see what is the timeout path:
The patch:
diff -u -p -r1.120 atapiscsi.c
--- dev/atapiscsi/atapiscsi.c 16 Apr 2022 19:19:58 -0000 1.120
+++ dev/atapiscsi/atapiscsi.c 28 May 2023 08:05:00 -0000
@@ -743,6 +743,7 @@ wdc_atapi_real_start_2(struct channel_so
sc_xfer->error = XS_TIMEOUT;
xfer->next = wdc_atapi_reset;
+ panic("HP: artificial panic on timeout!\n");
return;
} else {
wdc_atapi_update_status(chp);Console:
OpenBSD 7.3-stable (RAMDISK_CD) #1: Sun May 28 09:59:10 CEST 2023
[email protected]:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
Installing bsd 15% |*** | 3712 KB 00:05 ETA
wdc_atapi_start: not ready, st = 50
panic: HP: artificial panic on timeout!
Stopped at db_enter+0x5: popq %rbp
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*272956 35811 0 0x3 0 0 sleep
db_enter(ffffffff8140b6ff,0,ffff80002170d4d8,202,8,ffffffff8121424e)
at db_enter+0x5
panic(ffffffff813f9768,ffffffff813f9768,ffff80000007c168,fffffd807df64ea8,2,ffffffff)
at panic+0xef
wdc_atapi_reset(ffff80000007c168,fffffd807df64ea8,1,ffff80002170d5c0,ffff80000007c168,fffffd807df64ea8)
at wdc_atapi_reset
wdc_atapi_the_machine(ffff80000007c168,fffffd807df64ea8,0,ffff80000007c168,ffff80000007c168,fffffd807df64dd8)
at wdc_atapi_the_machine+0x8d
wdc_atapi_the_machine(ffff80000007c168,fffffd807df64dd8,2,ffff80000007c168,ffff80000007c168,fffffd807df64dd8)
at wdc_atapi_the_machine+0x14a
wdc_atapi_intr(ffff80000007c168,fffffd807df64dd8,1,ffff80000007c168,fffffd807df64dd8,ffff80000007c168)
at wdc_atapi_intr+0x47
wdcintr(ffff80000007c168,ffff80000007c168,fffffd807f8406c0,0,6,1)
at wdcintr+0xae
intr_handler(ffff80002170d758,ffff800000065500,ffff800000065680,ffffffff811f91b0,ffffffff81211216,ffff80002170d748)
at intr_handler+0x26
Xintr_ioapic_edge14_untramp(0,ffffffff811f91b0,0,18041969,fffffd807f8406c0,a)
at Xintr_ioapic_edge14_untramp+0x18f
Xspllower(0,0,0,0,ffffffff81211455,7f1f3405)
at Xspllower+0xc
pmap_unmap_ptes(fffffd807f8406c0,0,0,7f1f3405,ffffffff811ef07a,ffff80002170d870)
at pmap_unmap_ptes+0x1d
pmap_enter(fffffd807f8406c0,36a000,7f1f3000,4,20,fffffd807f8406c0)
at pmap_enter+0x48e
uvm_fault_lower_lookup(ffff80002170db58,ffff80002170db90,ffff80002170dad0,ffff80002170db58,0,ffff80002170db58)
at uvm_fault_lower_lookup+0x155
uvm_fault_lower(ffff80002170db58,ffff80002170db90,ffff80002170dad0,0,ffff80002170db58,ffff80002170da50)
at uvm_fault_lower+0x41
end trace frame: 0xffff80002170dc10, count: 0
ddb> show panic
*cpu0: HP: artificial panic on timeout!
ddb> show registers
rdi 0
rsi 0x14
rbp 0xffff80002170d4d8
rbx 0xffff80002170d5c0
rdx 0x3fd
rcx 0x4e00000000071002
rax 0x29
r8 0x101010101010101
r9 0x8080808080808080
r10 0xffff80002170d510
r11 0x8
r12 0xfffffd807e923e60
r13 0x30
r14 0
r15 0xffffffff813f9768 vga_accessops+0x208b8
rip 0xffffffff8121424e db_enter+0x5
cs 0x8
rflags 0x202
rsp 0xffff80002170d4d8
ss 0
db_enter+0x5: popq %rbp
Here is simple patch how we can confirm that there is memory corruption:
diff -u -p -r1.136 wdc.c
--- dev/ic/wdc.c 31 Dec 2019 10:05:32 -0000 1.136
+++ dev/ic/wdc.c 28 May 2023 08:24:04 -0000
@@ -883,8 +883,10 @@ wdcstart(struct channel_softc *chp)
return;
}
+ printf("HP: xfer=%p orig chp=%p\n",xfer,chp);
/* adjust chp, in case we have a shared queue */
chp = xfer->chp;
+ printf("HP: xfer=%p xfer->chp=%p\n",xfer,chp);
if ((chp->ch_flags & WDCF_ACTIVE) != 0 ) {
return; /* channel already active */And here results:
HP: xfer=0xfffffd807e020c38 orig chp=0xffff80000007c168
HP: xfer=0xfffffd807e020c38 xfer->chp=0xffff80000007c168
HP: xfer=0xfffffd807e020c38 orig chp=0xffff80000007c168
HP: xfer=0xfffffd807e020c38 xfer->chp=0x6e1e3d12d428657b
kernel: protection fault trap, code=0
Stopped at wdcstart+0x49: movl 0x58(%r15),%eax
ddb> trace
wdcstart(ffff80000007c168,ffff80000007c168,ffff80000007c168,fffffd807e020c38,10,ffff800021707a90)
at wdcstart+0x49
wdc_atapi_the_machine(ffff80000007c168,fffffd807e020c38,2,ffff80000007c168,ffff80000007c168,fffffd807e020c38)
at wdc_atapi_the_machine+0x14a
wdc_atapi_intr(ffff80000007c168,fffffd807e020c38,1,ffff80000007c168,fffffd807e020c38,ffff80000007c168)
at wdc_atapi_intr+0x47
wdcintr(ffff80000007c168,ffff80000007c168,0,0,6,1)
at wdcintr+0xaeintr_handler(ffff800021707bf0,ffff800000065500,ffff800000065680,0,ffffffff81212216,ffff800021707be0)
at intr_handler+0x26
Xintr_ioapic_edge14_untramp(0,ffffff9c,ffffffff81483770,0,ffff8000216cd060,75f3d68ef248)
at Xintr_ioapic_edge14_untramp+0x18f
ndinitat(ffff8000216cd060,ffffffffffffff9c,2e96cea10,75f3d68ef248,0,ffff8000216cd060)
at ndinitat
syscall(ffff800021707ef0,ffff800021707ef0,0,ffff8000216cd060,0,0)
at syscall+0x201
Xsyscall(6,26,5,26,2e96cea10,0)
at Xsyscall+0x128
end of kernel
end trace frame: 0x75f3d68ef2f0, count: -9
ddb>
Finally found workaround - both ACPI & HPET
- https://man.openbsd.org/acpihpet.4 has to be enabled to avoid timeout and trap. Awaiting for feedback...
- details are here:
On already installed system with full kernel it is easy to get all clock information using just sysctl, for example:
sysctl | grep -E '^kern.*(timecount|clockrate)'
kern.clockrate=tick = 10000, hz = 100, profhz = 1000, stathz = 100
kern.timecounter.hardware=pvclock0
kern.timecounter.choice=i8254(0) pvclock0(1500) acpitimer0(1000)
kern.timeout_stats=added = 10600, cancelled = 90, deleted = 805, \
late = 32, pending = 29, readded = 211, scheduled = 3563, \
rescheduled = 70, run_softclock = 9787, run_thread = 501, \
softclocks = 6882, thread_wakeups = 449However, when you try same sysctl command on installation CD (which uses bsd.rd special ramdisk
kernel - you will be surprised with very short output:
sysctl
kern.osrelease=7.3
hw.machine=amd64
hw.model=Intel(R) Celeron(R) CPU N3450 @ 1.10GHz
hw.product=Standard PC (i440FX + PIIX, 1996)
hw.disknames=cd0:,sd0:06f652dc156e63c6,rd0:fde630bc293de162
hw.ncpufound=1Yes - there are total 6 entries only.
How is that possible? There are 2 (yes two!) reasons:
- ramdisk uses special light version of sysctl command - source
is under
/usr/src/distrib/special/sysctl/sysctl.c, while full version on installed system is under/usr/src/sbin/sysctl/sysctl.c - there is special
option SMALL_KERNELin case of RAMDISK and various parts of kernel are omitted to make it smaller (!)
Here is simple patch, how to add kern.clockrate and kern.timecounter.*
to "special" version of sysctl:
Index: distrib/special/sysctl/sysctl.c
===================================================================
RCS file: /cvs/src/distrib/special/sysctl/sysctl.c,v
retrieving revision 1.14
diff -u -p -r1.14 sysctl.c
--- distrib/special/sysctl/sysctl.c 4 Jun 2021 00:29:15 -0000 1.14
+++ distrib/special/sysctl/sysctl.c 30 May 2023 17:55:37 -0000
@@ -44,10 +44,21 @@ struct var {
int pstring(struct var *);
int pint(struct var *);
+int pclock(struct var *);
struct var vars[] = {
{ "kern.osrelease", pstring, 2,
{ CTL_KERN, KERN_OSRELEASE }},
+ { "kern.clockrate", pclock, 2,
+ { CTL_KERN, KERN_CLOCKRATE }},
+ { "kern.timecounter.hardware", pstring, 3,
+ { CTL_KERN, KERN_TIMECOUNTER, KERN_TIMECOUNTER_HARDWARE }},
+ { "kern.timecounter.choice", pstring, 3,
+ { CTL_KERN, KERN_TIMECOUNTER, KERN_TIMECOUNTER_CHOICE }},
+ { "kern.timecounter.tick", pint, 3,
+ { CTL_KERN, KERN_TIMECOUNTER, KERN_TIMECOUNTER_TICK }},
+ { "kern.timecounter.timestepwarnings", pint, 3,
+ { CTL_KERN, KERN_TIMECOUNTER, KERN_TIMECOUNTER_TIMESTEPWARNINGS }},
{ "hw.machine", pstring, 2,
{ CTL_HW, HW_MACHINE }},
{ "hw.model", pstring, 2,
@@ -66,6 +77,27 @@ struct var vars[] = {
int nflag;
char *name;
+
+/* Maximum size object to expect from sysctl(2) */
+#define SYSCTL_BUFSIZ 8192
+
+int pclock(struct var *v)
+{
+ char buf[SYSCTL_BUFSIZ];
+ size_t len = sizeof(buf);
+
+ if (sysctl(v->mib, v->nmib, buf, &len, NULL, 0) != -1) {
+ struct clockinfo *clkp = (struct clockinfo *)buf;
+
+ if (nflag == 0)
+ printf("%s=", v->name);
+ (void)printf(
+ "tick = %d, hz = %d, profhz = %d, stathz = %d\n",
+ clkp->tick, clkp->hz, clkp->profhz, clkp->stathz);
+ return (0);
+ }
+ return (1);
+}
int
pint(struct var *v)You can quickly build it (preferably as non-root user) using:
cd /usr/src/distrib/special/sysctl
make obj # do this only on first build
make
obj/sysctlHowever be beware that bsd.rd may have some sysctl missing. Good news is
that all public sysctl calls are very well described on
official manual page: sysctl(2)
Sample outputs:
-
bare metal machine, (mostly) standard full kernel, MSI-7250, AMD X2, 8GB RAM, AHCI SATA + SSD disk:
OpenBSD 7.3 (HPCLEAN.MP) #0: Thu May 25 17:34:56 CEST 2023 [email protected]:/usr/src/sys/arch/amd64/compile/HPCLEAN.MP sysctl | grep -E '^kern.*(timecount|clockrate)' kern.clockrate=tick = 10000, hz = 100, profhz = 1000, stathz = 100 kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=acpihpet0 kern.timecounter.choice=i8254(0) acpihpet0(1000) acpitimer0(1000) -
KVM host, with ACPI=on, HPET=off (problematic configuration):
OpenBSD 7.3-stable (RAMDISK_CD) #6: Tue May 30 20:08:41 CEST 2023 [email protected]:/usr/src/sys/arch/amd64/compile/RAMDISK_CD # sysctl kern.osrelease=7.3 kern.clockrate=tick = 10000, hz = 100, profhz = 1000, stathz = 100 kern.timecounter.hardware=i8254 kern.timecounter.choice=i8254(0) kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 hw.machine=amd64 hw.model=Intel(R) Celeron(R) CPU N3450 @ 1.10GHz hw.product=Standard PC (i440FX + PIIX, 1996) hw.disknames=cd0:,sd0:06f652dc156e63c6,rd0:38549a5b73f99166 hw.ncpufound=1 -
notice different hardware
-
and here is working KVM configuration ACPI=on, HPET=on:
OpenBSD 7.3-stable (RAMDISK_CD) #6: Tue May 30 20:08:41 CEST 2023 [email protected]:/usr/src/sys/arch/amd64/compile/RAMDISK_CD # sysctl kern.osrelease=7.3 kern.clockrate=tick = 10000, hz = 100, profhz = 1000, stathz = 100 kern.timecounter.hardware=acpihpet0 kern.timecounter.choice=i8254(0) acpihpet0(1000) kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 hw.machine=amd64 hw.model=Intel(R) Celeron(R) CPU N3450 @ 1.10GHz hw.product=Standard PC (i440FX + PIIX, 1996) hw.disknames=cd0:,sd0:06f652dc156e63c6,rd0:38549a5b73f99166 hw.ncpufound=1
So now we know - RAMDISK kernel works well when acpihpet clock is used and has timeout issues when i8254 timer is used.
Yes, it is the case - I started new thread:
Please go to OpenBSD - my main wiki page dedicated for OpenBSD.
Copyright © Henryk Paluch. All rights reserved.
