-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathbios.html
1729 lines (1205 loc) · 61.1 KB
/
bios.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<!-- xhtml format, hw6 (tables) had to use transitional instead of strict dtd. -->
<!--
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
the one below for strict html:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
sttrict xhtml:
-->
<!--
CSS validator: http://jigsaw.w3.org/css-validator/#validate_by_uri
HTML validator: http://validator.w3.org/
-->
<head>
<title>Sys Admin Pocket Survival Guide</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<link rel="stylesheet" href="psg2.css" type="text/css" media="screen">
<link rel="stylesheet" href="psg-table.css" type="text/css">
<link rel="stylesheet" href="psg2-links-icons.css" type="text/css">
<link rel="stylesheet" href="psg-positioning.css" type="text/css">
<link rel="stylesheet" href="psg-print.css" type="text/css" media="print">
</head>
<body>
<div id="fixie"> <!-- add a fixed position example for css final project. potentially use to place ad, which will appear like a 3rd column on right -->
<div id="one" class="skinny"><a href="plug/plug.html"><img src="plug/nema-5-20R-drawing.gif" alt="nema5pic" width="100" height="100"></a></div>
<div id="two" class="skinny"><a href="fig/unixoid_hell.gif"><img src="fig/unixoid_hell.gif" alt="vi-hell" width="100" height="77" ></a></div>
<div id="three" class="skinny"><a href="fig/Assistant.gif"><img src="fig/Assistant.gif" alt="clippie" width="100" height="77" ></a></div>
<div id="four" class="skinny"><!-- empty for now --></div>
</div> <!-- closes #fixie -->
<div id="wrapper">
<div id="header">
<div id="title">Sys Admin Pocket Survival Guide</div>
<div id="sub-title">A Quick Reference Guide for Sys Admins with Alzeimer :)</div>
</div> <!-- closes header -->
<div id="navigation">
<!-- the replace was not inside hightlight area, but whole page, finc and put back some strong and em in doc...-->
<!-- strong, em, h5, or div are not allowed to be nested inside ul !! -->
<div class="azul">Unix</div>
<ul>
<li> <a href="sul>.html">Solaris</a></li>
<li> <a href="linux.html">Linux</a></li>
<li> <a href="hpux.html">HP-UX</a> and <br>
<a href="hpux.supl.html">supplement</a></li>
<li> <a href="aix.html">AIX</a> and <br>
<a href="aix_cd_catalog.html">AIX CD catalog</a><br></li>
<li> <a href="irix.html">Irix</a></li>
<li> <a href="dos.html">Windows</a></li>
<li> <a href="apple.html">Apple Mac</a></li>
</ul>
<div class="azul">Storage</div>
<ul>
<li> <a href="netapp.html">NetApp</a></li>
<li> <a href="emc.html">EMC SAN - Clariion</a></li>
<li> <a href="emcCelerra.html">EMC NAS - Celerra</a></li>
<li> <a href="isilon.html">Isilon</a></li>
<li> <a href="fs.html">Unix File System</a></li>
</ul>
<div class="azul">Big Data</div>
<ul>
<li> <a href="lsf.html">LSF, PBS/Torque, LSF.</a></li>
<li> <a href="mpi.html">MPI, PVM</a></li>
<li> <a href="sci-file.html">Science File Format/Info</a></li>
<li><a href="sci-app.html">Sci-App</a></li>
</ul>
<div class="azul">Unix Dev</div>
<ul>
<li><a href="development.html">compilers, etc</a></li>
<li><a href="shellScript.txt">sh/bash, csh/tcsh</a></li>
<li><a href="awk.txt">AWK</a></li>
<li><a href="perl.html">Perl</a></li>
<li><a href="python.html">Python</a></li>
<li><a href="php.txt">PHP</a></li>
<li><a href="javascript_eg.html">javascript_eg</a></li>
<li>gcc</li>
<li><a href="gdb.html">gdb</a></li>
<li>java</li>
<li>rcs, cvs, p4, subversion</li>
<li><a href="vi.html">vi</a></li>
</ul>
<div class="azul">Network</div>
<ul>
<li> <a href="net.html">Network</a></li>
<li> <a href="infiniband.html">InfiniBand</a><br></li>
<li> <a href="acopia.html">Acopia</a></li>
</ul>
<div class="azul">Misc</div>
<ul>
<li> <a href="ldap.html">LDAP</a><BR></li>
<li> <a href="admin.html">General Unix Sys Admin </a></li>
<li> <a href="tool.html">Sys Admin tools and performance tuning</a></li>
<li> <a href="vnc.html">VNC, X Emulation</a></li>
<li> <a href="backup.html">Unix backup</a></li>
<li> <a href="general_unix.html">Generic Unix Commands</a></li>
<li> <a href="veritas.html">Veritas</a></li>
<li> <a href="legato.html">Legato Networker</a></li>
<li> <a href="mysql.html">MySQL</a></li>
<li> <a href="html.txt">HTML tags</a></li>
<li> <a href="wiki.html">WiKi tags</a></li>
<li> <a href="3rdParty">3rd Party and Vendor Docs Cache</a></li>
</ul>
<div class="azul">IMHO</div>
<ul>
<li> <a href="monitor.html">Network monitoring tool review</a></li>
<li> <a href="netArch.html">Network Architecture Approaches</a></li>
<li> <a href="docPlatform.html">Documentation platform</a></li>
</ul>
<div class="azul">Prod Review</div>
<ul>
<li><a href="termSvr.html">Terminal (Serial Console) Servers</a></li>
<li><a href="ent_prod.html">Enterprise Products</a></li>
<li><a href=""></a></li>
<li> </li>
</ul>
</div> <!-- closes #navigation -->
<!-- ########################################################## -->
<!-- ########################################################## -->
<div id="content">
<p>
<div align="CENTER">
<A HREF="http://www.commitstrip.com/en/2016/06/21/childhood-of-a-coder-booting-to-bios/">
<IMG SRC="fig/bios.Strip-Enfance-du-codeur-Le-bios-650-finalenglish-1.jpg"></A>
<BR>
Answer: DEL-F10-F1-DEL-DEL-DEL-F8-F10-F8-F1-DEL-F10-F12-DEL-DEL-F11. <BR>
In that sequence, to be completed within 3 seconds.
</div>
<H1>BIOS</H1>
Do you prefer UEFI? Really?? <BR>
Anyway, racadm, ipmitool, some stuff are probably common to BIOS and UEFI ;-) <BR>
<H2>IPMI support by NIC</H2>
<PRE>
Network-Controller Sideband Interface (NCSI) header:
NC-SI for IPMI support (header/connector, between NIC and mboard).
CX-6 in AIOM/OCP3.0 module, in SMC, did work as shared cable for IPMI (ethernet mode).
</PRE>
<H2>nVidia DGX A100</H2>
DEL or F2 = setup
F11 = boot menu
F12 = PXE boot
Epyc 7742 64-core x2
2TB RAM?
<H2>Penguin Computing</H2>
<PRE>
F2 = Setup
F10 = Display Boot Menu
F12 = Force Network Boot
For BMC/IPMI was said to default to share eth0 with host os when service RJ45 isn't connected.
However, my systems wasn't setup that way.
To make IPMI share eth0 with the host os, set it with:
ipmitool raw 0x2e 0xcc 0x5e 0x2b 0 0x0C 0x01 0x02
Note that ipmi already have a default admin/admin account as user 2.
If setting new admin acc, try to rename the existing one first:
ipmitool user set name 2 oper
ipmitool user list 1
</PRE>
<!-- did not work:
go to BIOS, Server Management, BMC... Use
"Mode 2 NCSI"
This hopefully also means that if there is eth connected to the management port, it will use that automatically.
(default, at least for the batch I am workihg on, is set to "mode 1 dedicated").
-->
<H2>HP Proliant (Compaq) Server</H2>
<PRE>
BIOS key combinations (At least for DL 760, but DL 580 should be same)
1st prompt = memory
F1 for memory config
ESC to bypass
2nd prompt = Smart Array (RAID controller, no software support at OS level?)
F8 for config of logical drive.
ESC to cancel.
If there are failures, F1 to continue w/ drive disabled, F2 (accept data loss and continue)
It will sit and wait till a choice is made in such cases.
3rd prompt = BIOS
F9 = Option ROM config
F10 = enter setup mode of BIOS (seems to be some DOS utility that gets run!)
F12 = PXE boot
The prompt seems to change slightly depending on boot time errors.
Major thing is to have finger ready at the keyboard and know what to expect when.
F8 for RAID config
F10 for BIOS
-------------
HP DL 585 (AMD Opteron)
F9 = BIOS setup. it is diaplayed in bottom right corner while POST is going on during some period.
Ctrl-E = Emulex HBA config
</PRE>
<H2>Lenovo Server</H2>
<PRE>
F1 Setup
F2 Diagnostics
F12 Select Boot Device # Need legacy mode to do PXE boot
prompt 2:
Ctrl-S : Broadcom NIC NexXTreme Ethernet Boot Agent config
prompt 3:
Ctrl-I : MINI config for SERVER RAID 7x controller, only allow minimal logical drive config.
presumably need to boot from CD for full hardware RAID config, such as setup RAID 5 or 1.
</PRE>
<H2>Lenovo Desktop/Laptop</H2>
<PRE>
Enter (or ThinkVantage button)
ctrl-m = marvell bios setup (sata, raid config)
ctrl-s = mac address config
</PRE>
<H2>SuperMicro</H2>
<p>
During Post, keys:
<PRE>
tab Display BIOS POST message
del run setup (BIOS)
F2 run setup (BIOS)
F11 boot menu
F12 Network boot
RAID controller
AVAGO MegaRAID SAS-MFI BIOS v5.50.03.0 July 2015)
^P pause (disk scanning?)
^V skip
^H WebBIOS -- ctrl-h load RAID sw to configure virtual drives
^Y preboot CLI
</PRE>
<H2>Tyan</H2>
<PRE>
del or F2 run setup (AMI BIOS)
F12 boot from ...
ipmitool lan set 8 cipher_privs XXXXXXXXXXXXXXX # tried for Tyan nodes, but ipmi cipher problem persist
ipmitool lan set 8 cipher_privs caaaaaaaaaaaXXX # this might have been the default
to have shared IPMI,
go to bios (ie no ipmitool cli to configure this, PITA)
BMC: "management 2" need to be enabled (was disabled by default).
enabling that restored config, including all the previous settings :)
(had set a static IP+netmask in the bios, but presumably can use ipmitool to set a new IP as well).
</PRE>
<H2>Asus</H2>
<PRE>
del run setup (AMI BIOS)
alt+F2 - EzFlash
F8 = BBS POPUP # ie boot device menu
^B PXE prompt for Intel I350 on-board NIC
</PRE>
<H2>Dell</H2>
PowerEdge
<PRE>
F2 = System Setup
F10 = Lifecycle Controller (BIOS setup)
F11 = Boot Manager
F12 = PXE boot
subsequent prompts sequence:
Ctrl-S : Intel PXE Boot agent config
Ctrl-M : LSI Logic RAID Controller config
Ctrl-A = Avago raid controller
Ctrl-D : Dell Remote Access Controller Setup (build in kvm over IP)
This run on dedicated NIC on service processor card.
Machine does not have to be on, but must have power connected.
Can be configured to use DHCP and register DDNS ('A' record only).
IP will be displayed on boot message.
ssh/http/https (vnc port 5900 seems also open)
There is no setup for username/password in the BIOS settings, default login = root/calvin
DRAC 4 Firmware 1.33 req Java Plugin version 1.4.2. Firefox ok.
Ctrl-E : BMC software IPMI server management
This need to run on one of the NIC on motherboard.
Can be setup for DHCP, but DDNS doesn't work.
IP will be displayed on boot message.
but no ports seems to be open :(
</PRE>
<H3>iDRAC access</H3>
<H5>https</H5>
if there is a browser that can use the ipmi/idrac network, pointing it to the the IP of iDRAC would bring up the web interface.
<H5>ssh</H5>
<PRE>
instead of relying on ipmitool to remotely query/control the service processor, can ssh to it.
ssh ipmi-ip -l admin
get racadm>> prompt
from here, can issue commands (optional to have explicitly called with racadm)
racadm help cmd
racadm techsupreport collect
racadm jobqueue view
racadm techsupreport export -l 10.0.92:/users/users/tin/tmp/tsr_report.zip
# above would work if NFS server allows the racadm network to mount ?
# other protocols are tftp, ftp, cifs. scp is not supported, nor rcp.
racadm get iDRAC.serverboot.firstbootdevice
racadm set iDRAC.serverboot.firstbootdevice PXE
# above is a one time boot config
# after that, it rever to previous setting , ie FirstBootDevice=Normal
# but that's not the BootOnce setting, a bit strange.
racadm get iDRAC.ServerBoot.BootOnce
racadm help iDRAC.serverboot.firstbootdevice
racadm set iDRAC.serverboot.firstbootdevice F11
# F10 (Lifecycle Controller)
# F11 (BIOS Boot Manager);
# no F2 here, but F11 will have menu leading to bios setup.
racadm getsel # get system event log
getsvctag # get service tag aka serial number
other commands to try?
clrsel -- clear the System Event Log (SEL)
cmreset -- perform a Chassis Manager reset operation
coredump -- display the RAC coredump files
coredumpexport -- Export the RAC coredump files
coredumpdelete -- Deletes the core files
driverpack -- display driverpack info
debug -- Field Service Debug Authorization facility commands
exposeisminstallertohost -- Support Assist operations.
gethostnetworkinterfaces -- Display host network interface details
getled -- Get the state of the LED on a module.
getniccfg -- display current network settings # IP, subnetmask, gateway, LOM1
getraclog -- display the RAC log
getractime -- display the current RAC time # internally is GMT?
getremoteservicesstatus -- display remote services status
racadm>>getremoteservicesstatus
Server status: Out of POST
LC status : Ready
RT status : Ready
Status : Ready
TS status : Ready
SEKM status : Ready
racadm>>
getniccfg # YMMV, but IP can hint what host is on.
racadm# get bios.MiscSettings.AssetTag
racadm# set bios.MiscSettings.AssetTag n0029
jobqueue create BIOS.Setup.1-1
# how did i not set this NetworkSettings before?? was looking for it for long time!
get BIOS.NetworkSettings.PxeDev1EnDis # F2 | Bios Settings | Network Settings
set BIOS.NetworkSettings.PxeDev1EnDis Enabled
jobqueue create BIOS.Setup.1-1
</PRE>
<A HREF="https://dl.dell.com/content/manual65464730-integrated-dell-remote-access-controller-9-racadm-cli-guide.pdf?language=en-us">racadm CLI guide</A> iDRAC 9, 2024-09.
<PRE>
racadm cmd:
get -- display RAC configuration properties
get BIOS # and many subcomponents
# also get in these categories:
BIOS -- Configuration of BIOS attributes
iDRAC -- Configuration of iDRAC attributes
LifecycleController -- Configuration of LifecycleController attributes
Nic -- Configuration of NIC attributes
Storage -- Configuration of Storage attributes
System -- Configuration of System attributes
FC -- Configuration of Fiber Channel attributes
InfiniBand -- Configuration of InfiniBand attributes
BIOS.IntegratedDevices ... # F2|System BIOS|Integrated Devices :: User Accessible USB Port, internal SD card, iDRAC Direct USB Port, Embeded NIC1
BIOS.IntegratedDevices ... # Embeded NIC1 has setting of "(Disabled (OS)"
racadm>>get System.UserDefinedString
ERROR: SWC0244 : Invalid Fully Qualified Device Descriptor (FQDD).
# somewhere, can likely set some custom info, for say PO, hostname, or asset tag.
getniccfg # the "get" equivalent for nic portion of the menu
Integrated vs Embeded ?
Integrated is OCP 3.0 slot NIC?
Embeded is the staic shared NIC for iDRAC + OS 1g NIC?
Infiniband shows up separately by itself (Mezaline card, as plain PCI device?)
set nic.nicconfig.1.legacybootproto PXE # F2|Device Settings|(select NIC)|NIC Config|Legacy Boot Protocol (ie boot option to support PXE)
# nicconfig.1 seems like "Integrated Nic" (OCP 3.0 slot?)
# nicconfig.3 seems to corresponds to the embeded NIC (1g RJ45 shared with iDRAC)
jobqueue create NIC.Integrated.1-1-1 -r pwrcycle -s TIME_NOW
FYI:
nic.DeviceLevelConfig.1 # is F2|Device Settings|(select NIC)|Device Level COnfig ## Virtualization mode, Link speed, stuff that likely seldom need changing.
</PRE>
<H5>usb</H5>
connect a usb (micro) from a laptop, it will establish an IP network over this connection.
<BR>
<BR>
details at:
<BR>
https://www.dell.com/support/manuals/en-us/idrac9-lifecycle-controller-v3.1-series/idrac_3.15.15.15_ug/accessing-idrac-interface-over-direct-usb-connection?guid=guid-2c43a175-3dd3-4a12-b747-d17c08e5bef6&lang=en-us
<BR>
<BR>
<PRE>
essentially, server/laptop would need to be configured with
169.254.0.3
169.254.0.4
</PRE>
<H3>HT - Hyperthreading/Logical Processor</H3>
<PRE>
/opt/dell/srvadmin/sbin/racadm get BIOS.ProcSettings.LogicalProc # query HT status
/opt/dell/srvadmin/sbin/racadm set BIOS.ProcSettings.LogicalProc Disabled # disable HT
/opt/dell/srvadmin/sbin/racadm jobqueue create BIOS.Setup.1-1 # create job to change BIOS setting
/opt/dell/srvadmin/sbin/racadm serveraction powercycle # power cycle to effect change
</PRE>
<H3>SNC - Sub NUMA Cluster on Skylake</H3>
Leave SNC disabled (default).
<BR>
See Fig 2 of:
http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2017/08/01/bios-characterization-for-hpc-with-intel-skylake-processor
<PRE>
Skylake... 14th gen PowerEdge...
"introduce a clustering mode called Sub NUMA clustering (SNC). On CPU models that support SNC, enabling SNC is akin to splitting the single socket into two NUMA domains, each with half the physical cores and half the memory of the socket."
Query Sub NUMA Cluster (SNC) modes
- numactl -H
- /opt/dell/srvadmin/sbin/racadm get BIOS.ProcSettings.SubNumaCluster
- lstopo
- SNC=Disabled (default). ie, two socket show up as two NUMA domains.
- SNC=Enabled. within socket has split NUMA domains. two socket system would then have FOUR NUMA domains. application who does not optimize for memory locality would be worse off. Thus, not recommended unless benchmark and tuning has been done that benefit from this setting.
SNC disabled is baseline. If enable SNC:
- STREAM, WRF, Fluent, which are highly localized memory optimized, have slight gain of 1-2%
- HPL was ~1% worse off.
- /opt/dell/srvadmin/sbin/racadm get BIOS.MemSettings.NodeInterleave
- Disabled (default) = system support NUMA (asymmetric) mem config
- Enabled = memory interleave is supported IF symetric memory config is installed. (SNC will not work when memory configured this way?)
</PRE>
See <A HREF="script/hpc/racadmSetBios.sh">racadmSetBios.sh</A> script that set this and many below values. <BR>
<H3>Numa on Epyc Rome 7xx2</H3>
<PRE>
BIOS.ProcSettings.NumaNodesPerSocket # default to 4, and numactl -H will show node with 0 RAM :/
BIOS.ProcSettings.ProcVirtualization
</PRE>
<H3>Memory Operating Mode</H3>
<PRE>
racadm help BIOS.MemSettings.MemOpMode
OptimizerMode - Optimizer Mode (default)
SingleRankSpareMode - Single Rank Spare Mode;
MirrorMode - Mirror Mode;
FaultResilientMode - Fault Resilient Mode;
OppSrefEn=Disabled (default)
</PRE>
<H3>DAPC & GFLOPS/watt</H3>
DAPC profile consume less power than Performance profile, yet said to produce essentially same chrunching power. [my actual HPL test lost 7% performance]
<BR>
See Fig 3 + 4 of:
http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2017/08/01/bios-characterization-for-hpc-with-intel-skylake-processor
<PRE>
Skylake... 14th gen PowerEdge...
- In idle state, Performance Profile consumes ~28% more power than DAPC.
- Peak power consumption in DAPC Profile is ~16% less than in Performance Profile.
- STREAM, WRF have essentialy same performance in either Performance or DAPC mode.
- HPL is 1.05% better in Performance than DAPC mode.
- Fluent is ~1.01% better.
/opt/dell/srvadmin/sbin/racadm get BIOS.SysProfileSettings.SysProfile
SysProfile=PerfOptimized (default)
/opt/dell/srvadmin/sbin/racadm set BIOS.SysProfileSettings.SysProfile PerfPerWattOptimizedOs
Using PerfPerWattOptimizedDapc:
- TBA
Using PerfPerWattOptimizedOs:
- Reduce HPL performance by ~7% on Skylake 6130 2.1 GHz 32cores 96G on Dell C6420
- Exposes HPL clock calculation bug in random manner (cuz it messes with CPU clock?)
Using LowLatencyOptimizedProfile:
- This was undocumented profile in 12th Gen PowerEdge
- said to make a world of difference for latency , but has significant draw on power and will keep all fans blowing on full speed all the time
racadm help BIOS.SysProfileSettings.SysProfile # list available modes (and dependencies):
- PerfPerWattOptimizedDapc - Performance Per Watt (DAPC)
- PerfPerWattOptimizedOs - Performance Per Watt (OS)
- PerfOptimized - Performance
- PerfWorkStationOptimized - Workstation Performance;Custom - Custom;
</PRE>
<H3>Other BIOS param</H3>
see PSG/script/hpc/record_bios_settings.sh
<BR>
<BR>
<!--
## ***** really >>>
## ***** better to use PSG/script/hpc/record_bios_settings.sh
x singularity exec -B /var/run /global/home/users/tin/sn-gh/dell_idracadm/dell_idracadm.img /opt/dell/srvadmin/sbin/racadm \
x singularity exec -B /var/run /global/scratch/tin/singularity-repo/dell_idracadm.img /opt/dell/srvadmin/sbin/racadm \
x singularity exec -B /var/run /global/scratch/tin/singularity-repo/dirac1_dell_idracadm.img /opt/dell/srvadmin/sbin/racadm \
singularity exec -B /var/run /global/home/groups/scs/tin/singularity-repo/tin6150-dell_idracadm-master-latest.simg /opt/dell/srvadmin/sbin/racadm \
export SIFracadm=/global/home/groups/scs/tin/singularity-repo/dirac1_dell_idracadm.img
singularity exec -B /var/run $SIFracadm /opt/dell/srvadmin/sbin/racadm \
get BIOS.ProcSettings.LogicalProc
#### just use pdsh -w n0[218-229].savio3 /global/home/users/tin/PSG/script/hpc/record_bios_settings.sh dell # 2021.0920
# singularity-hub build probably won't work cuz build machine don't have an iDRAC :(
racadm commands, maybe more useful ones
rollback - Rollback firmware to its previous version.
frontpanelerror - hide LCD errors - color amber to blue
recover - Recover firmware to its previous version.
remoteimage - make a remote ISO image available to the server
rollback - Rollback firmware to its previous version.
hwinventory - Monitoring and Inventory of H/W NICs connected to the server.
update - Platform Update of the devices on the server
jobqueue - Jobqueue of of the jobs currently scheduled
sensorsettings - Set the sensor threshold levels.
getsensorinfo - got psu of chassis info.
-->
When CPU doesn't perform, check these params (racadm):
<PRE>
set BIOS.ProcSettings.LogicalProc Disabled # non default, but HPC don't want HT on
set BIOS.MemSettings.MemOpMode OptimizerMode # default, should not need to change this for HPC
set BIOS.MemSettings.NodeInterleave Enabled # non default, not compatible with SubNumaCluster, not typically recommended
set BIOS.ProcSettings.SubNumaCluster Enabled # non default, only useful if app can better utlize localized mem
set BIOS.SysProfileSettings.SysProfile PerfOptimized # default
set BIOS.SysProfileSettings.SysProfile PerfPerWattOptimizedDapc # said to save energy
set BIOS.SysProfileSettings.SysProfile PerfPerWattOptimizedOs # may save energy, seems to introduce clock/timing bug
set BIOS.ProcSettings.ControlledTurbo Enabled # allow for external control of when to engage turbo? Def: Disabled.
get BIOS.ProcSettings.ProcTurboMode # def Enabled, changeable only in Custom SysProfile
get BIOS.SysInformation.SystemBiosVersion # BIOS version
set BIOS.SysProfileSettings.SysProfile Custom # need custom mode to set the next two option:
set BIOS.SysProfileSettings.ProcCStates Autonomous # def: Disabled. alt: Enabled # to allow proc to operate in all avail power state
set BIOS.SysProfileSettings.UncoreFrequency DynamicUFS # def: MaxUFS
# https://software.intel.com/en-us/forums/software-tuning-performance-optimization-platform-monitoring/topic/543513
# Perhaps also check:
get BIOS.ProcSettings.UpiPrefetch # def: Enabled
get BIOS.MemSettings.OppSrefEn # def: Disabled
get BIOS.SysProfileSettings.MemFrequency # def: MaxPerf. Could choose diff speed such as 2666 MHz, 2400, 1866.
get BIOS.SysProfileSettings.ProcC1E # def: Disabled # but enabled when switch to Custom SysProfile !!
# Enable = processor is allowed to switch to minimum performance state when idle.
get BIOS.SysProfileSettings.EnergyPerformanceBias # def: MaxPower (ie Performance).
# alt: BalancedPerformance, BalancedEfficiency, LowPower
</PRE>
Note: Lost of redundant power supply may cause system to clock down significantly, eg turbostat may report 800 MHz on a 2.1 GHz CascadeLake 6230.
Checking the following may help to see if there is not enough power supplied to the system:
<PRE>
# enable IPMI/iDRAC:
racadm set iDRAC.IPMILan.Enable 1
# [Key=iDRAC.Embedded.1#IPMILan.1]
racadm get System.ChassisInfo
other get options: System.Power System.Power.RedundancyPolicy System.PowerHistorical System.ServerPwr iDRAC.Info iDRAC.WebServer iDRAC.VNCServer
omreport may have data, if can get it to work
dmidecde -t chassis
</PRE>
<!--
singularity exec -B /var/run /global/home/users/tin/sn-gh/dell_idracadm/dell_idracadm.img /opt/dell/srvadmin/sbin/racadm jobqueue create BIOS.Setup.1-1
singularity exec -B /var/run /global/home/users/tin/sn-gh/dell_idracadm/dell_idracadm.img /opt/dell/srvadmin/sbin/racadm serveraction powercycle
-->
<H5>Undocumented HPL parameter</H5>
<PRE>
# set env var for HPL to properly use AVX
export HPL_HOST_ARCH=3 # AVX2, eg Hashwell
export HPL_HOST_ARCH=9 # AVX512 eg Skylake
</PRE>
<H3>KNL BIOS settings</H3>
KNL specific cpu/memory config:
<PRE>
racadm help BIOS.MemSettings...
MemThrottlingMode
Cltt (def)
Oltt
SystemMemoryModel
All2All
SNC-2
SNC-4
Hemisphere
Quadrant (def) # 2x2
ProcEmbMemMode # KNL Memory Mode. affect "free -h", "numactl -H"
Cache (def)
Memory # ie flat mode, memory seen in "free -h" and malloc-able.
Hybrid
BIOS.ProcSettings
DynamicCoreAllocation=Disabled -- This field enables or disables the OS capability to put logical processors in the idling state in order to reduce power consumption.
ProcConfigTdp=Nominal -- This field allows reconfiguration of TDP (Thermal Design Power) to lower levels.
AMD can't call Hyperthreading, they call it SMT, disabling it need to go to some obscure meny in BIOS, accept some waiver stuff, before can set thread per core.
(at least in Asus)
2021.11 bios has this under Advanced, AMD CBS, CPU Common options, Performance, SMT control. no more accepting waiver.
Can leave the SVM setting (under CPU config), that's for Virtual machine virtualization instruction.
CBS = ??
</PRE>
<H3>Retrieve Serial stored in BIOS</H3>
<PRE>
racadm get System.ChassisInfo # service tag of chassis, eg Dell C6420 chassis
racadm get System... # service tag of blade/sledge
ipmitool fru | grep Serial # list all fru and filter for serial of sledge/blade
ipmitool raw 0x30 0xc8 0x01 0x00 0x0b 0x00 0x00 0x00 | xxd -r # get Dell C6320 chassis service tag vintage 2017
</PRE>
More info, eg fetching Dell C6220 II chassis serial, see: <A HREF="ipmi.html#hexcode">ipmi</A>
<H3>Enter F10 LifeCycle Controller</H3>
https://serverfault.com/questions/866657/racadm-reboot-to-lifecycle-controller
says that one time boot setting to do equiv of pressing F10 is to do:
<PRE>
# racadm set iDRAC.ServerBoot.BootOnce Enabled
# racadm set iDRAC.ServerBoot.FirstBootDevice F10
to get back to normal:
# racadm set iDRAC.ServerBoot.FirstBootDevice Normal
# racadm set iDRAC.ServerBoot.BootOnce Disabled
</PRE>
<H3>Misc</H3>
<PRE>
racadm techsupreport collect # start a support collection
racadm jobqueue view
racadm techsupreport export -f tsr_report.zip # didn't work via singularity img (cuz idrac version diff?)
racadm supportassist exportlastcollection -f tsr.zip # via sl7-tools vnfs
racadm racreset # restart rac, like power cycling the service processor, but not the wipe config and restore to factory default kind of reset, which does have a separate cmd for
</PRE>
<H5>ipmi</H5>
Dell maintain their <A HREF="https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0992n">own version of ipmi</A> that handles complex situation that RHEL7 ipmitool didn't.
eg: <BR>
<BR>
<PRE>
./ipmitool delloem lan set shared with lom1
singularity exec -B /var/run /global/scratch/users/tin/singularity-repo/dirac1_dell_idracadm.img /opt/dell/srvadmin/sbin/racadm get iDRAC.NIC
# disable SNMP in idrac
sudo apptainer exec -B /var/run /local/home/tin//dirac1_dell_idracadm.img /opt/dell/srvadmin/sbin/racadm set iDRAC.SNMP.AgentEnable Disabled
sudo apptainer exec -B /var/run /local/home/tin//dirac1_dell_idracadm.img /opt/dell/srvadmin/sbin/racadm get iDRAC.SNMP.AgentEnable
</PRE>
<!--
~tin/sw/dell_ipmi/ipmitool delloem lan set shared with lom1
c00: /local/tin/dell_tool/iDRACTools/ipmitool/usr/bin/ipmitool delloem lan set shared with lom1
-->
<BR>
<PRE>
racadm set iDRAC.NIC.Selection 2
with the selections as follows
1 - Dedicated
2 - LOM1
3 - LOM2
4 - LOM3
5 - LOM4
</PRE>
<!--
# singularity exec -B /var/run /global/scratch/tin/singularity-repo/dell_idracadm+idracFw.img /opt/dell/srvadmin/sbin/racadm \
# update iDRAC firmware
not worked out yet.
iDRAC: *Extract the firmimg.d7 file*
iDRAC update: https://downloads.dell.com/FOLDER04457642M/1/iDRAC8_2.50.50.50_A00.exe
unzip iDRAC8_2.50.50.50_A00.exe # produce firmimg.d7 etc
racadm fwupdate -p -u -d ./firmimg.d7 # upload firmware, start update. this didn't work.
racadm update -f ./firmimg.d7 # upload firmware, start update. this works.
### error
### sh: bmcfwu.so: command not found
### maybe looking for /opt/dell/toolkit/bin/bmcfwu.so
### and so may have to install the .BIN in the singularity container...
### *sigh* building new container to try this...
racadm fwupdate -r # rollback to standby fw
racadm fwupdate -s # query update process
# http://www.dell.com/support/home/vi/en/vibsdt1/drivers/driversdetails?driverId=278FC
wget https://downloads.dell.com/FOLDER04457684M/1/iDRAC-with-Lifecycle-Controller_Firmware_278FC_LN_2.50.50.50_A00.BIN
-->
<!--
bios update (not full lifecycle/idrac stuff??)
BIOS_YP88M_WN64_2.9.3_01 as .bin .efi (uEFI uses DOS exec) .exe just run on host as root??
cp -pR /global/scratch/tin/Downloads/dell/ /local
bash BIOS_YP88M_LN_2.9.3_01.BIN
Collecting inventory...
Running validation...
PowerEdge BIOS
The version of this Update Package is newer than the currently installed version.
Software application name: BIOS
Package version: 2.9.3
Installed version: 2.5.4
Continue? Y/N:
-->
<!--
install dell OMSA programatically?
wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash
yum install srvadmin-all
ref https://serverfault.com/questions/716467/cant-get-omsa-installed-on-dell-r610-w-centos-6
-->
<H5>Ref</H5>
<A HREF="https://www.intel.com/content/dam/support/us/en/documents/network/omni-adptr/sb/Intel_OP_Performance_Tuning_UG_H93143_v1_0.pdf">Intel E5 v3 (Hashwell) bios param tuning for Fabric</A>(Omnipath, Infiniband).
<H3>Dell Ref</H3>
Ref:
<UL>
<LI><A HREF="http://www.dell.com/support/manuals/us/en/19/dell-opnmang-dplymnt-toolkit-v4.4/dtk_cli-v3/syscfg-for-bios-configuration?guid=guid-928037f2-1410-4547-b2f7-d9b07d702060&lang=en-us">List of many racadm settable params (vs syscfg)</A>
<LI><A HREF="http://www.dell.com/support/manuals/vi/en/vidhs1/dell-opnmang-dplymnt-toolkit-v5.0.1/dtk_cli-v5/syscfg-for-bios-configuration?guid=guid-928037f2-1410-4547-b2f7-d9b07d702060&lang=en-us">v5.0.1 of above</A>
<LI><A HREF="http://topics-cdn.dell.com/pdf/poweredge-c6420_owner%27s%20manual_en-us.pdf">C6420 Spec and Config</A>BIOS/Memory settings in page 40</A>
<LI><A HREF="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwinlpT0wvHYAhUI0mMKHRpJAlkQFggpMAA&url=http%3A%2F%2Fen.community.dell.com%2Fcfs-file%2F__key%2Ftelligent-evolution-components-attachments%2F13-4491-00-00-20-44-05-27%2FBIOS-Setup-User-Guide.pdf%3Fforcedownload%3Dtrue&usg=AOvVaw0ZWFfEd1FlvrKGCYkXhXtS">Dell 14th Gen server (skylake) BIOS settings</A>
<LI><A HREF="http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2017/08/01/bios-characterization-for-hpc-with-intel-skylake-processor">Dell skylake Performance Benchmark results</A> (HPL, Stream, SpecInt, Ansys, etc)
<LI><A HREF="http://www.dell.com/support/manuals/vi/en/vidhs1/dell-opnmang-dplymnt-toolkit-v5.0.1/dtk_cli-v5/syscfg-for-bios-configuration?guid=guid-928037f2-1410-4547-b2f7-d9b07d702060&lang=en-us"></A>
<LI><A HREF="https://downloads.dell.com/manuals/common/poweredge_perf_amdepyc7002series.pdf">Dell Epyc Rome Becnharmk results</A>
<LI><A HREF="https://www.dell.com/support/article/en-us/sln311501/high-performance-computing?lang=en">Dell HPC results</A>
<LI><A HREF=""></A>
<LI>
</UL>
<BR>
<A NAME="sum"></A>
<A NAME="SUM"></A>
<A NAME="sm"></A>
<A NAME="SM"></A>
<H3>SuperMicro (bios) update manager</H3>
<!-- /global/scratch/tin/sw/sum/sum_2.0.0_Linux_x86_64/sum -->
SUM = supermicro update manager, not the checksum command from the OS!
<BR>
<BR>
<LI> simple binary to extract out of tgz file, then it just run.
<LI> write to current dir for a sum.log file.
<LI> root access required
<LI> Note that older bios output is simple key-value pair flat file, whereas newer bios is in XML format. (it has nothing to do with SUM version (eg 2.0 vs 2.2, older version with newer bios still produce XML output).
<BR>
<PRE>
untar gz, no real need to install
./sum -h # not the checksum command in linux default path...
sum -i 10.10.... -u ADMIN -p ADMIN -c GetBIOSInfo # use IPMI interface to get info
sum -c GetBIOSInfo # this find out firmware version info, etc. run on local machine
sum -c UpdateBios --file BIOS.rom # this update bios, not change config
sum -c GetCurrentBiosCfg --file smBiosCf.txt # write output of BIOS settings (HT on or off, etc) to file
sum -c GetCurrentBiosCfg > smBiosCf.txt # should be same, but couple of places have * (default) next to diff entries. also has SM(c) header, should avoid
vi smBiosCf.txt # make desired changes to BIOS use hex code for values. eg turn off Hyperthreading
# remove the first two lines of cfg that had the SM copyright info
# in KNL, file was simple text file
# in skylake, file was xml (UEFI bios?)
sum -c ChangeBiosCfg --file smBiosCf.txt --reboot # update bios config, rebooting host
# --reboot would actually be an soft reboot telling OS to shutdown (via ACPI?)
sum -h -c ChangeBiosCfg # get help on how to make changes to bios
sum -i ipmi-n0260.savio3 -U admin -P $PASS -c UpdateBIOS --file BIOS_X12DPG-OA6-1B88_20220823_1.4_STDsp.bin ## --reboot, just do manual ipmi cycle afterwards
## --preserve-config option wasn't operation for SUM 2.8, so have to redo bios changes like logical processor after the update :-\
## very annoyingly, the xml from the GetBiosConfig has mac and IP in it
## there are no commands to change specific setting
## so have to manually edit the xml file for each machine before uploading them to bios to apply the changes.
## SMC, please fix this anoying tedious mess!