-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathipmi.html
531 lines (374 loc) · 20.3 KB
/
ipmi.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>Pocket Survival Guide - IPMI</title>
<link rel="stylesheet" href="psg.css" type="text/css">
<LINK REL="SHORTCUT ICON" HREF="favicon.ico" type="image/x-icon"/>
<META NAME="description" content="System Administrator Pocket Survival Guide - A series of notes for Sys Admin"/>
<META NAME="keyword" content="Sys Admin, System Administrator, Solaris, HP-UX, AIX, Linux, Note, Notes, Pocket, Survival, Guide, psg, data center, power, electrical, plug, LYS, LKS, LAPPLAPP"/>
<META NAME="Robots" CONTENT="all"/>
<META NAME="Author" CONTENT="Tin Ho"/>
</head>
<body>
<div class="navheader">
<table summary="Navigation header" width="100%">
<tbody>
<tr>
<th colspan="4" align="center">
<A HREF="http://tin6150.github.io/psg/">Sys Admin Pocket Survival Guide - IPMI</A>
</th>
</tr>
<tr>
<td align="left"><a accesskey="h" href="psg.html">Home</a></td>
<td align="center"><a accesskey="f" href="firewall.html">Linux Firewall</a></td>
<td align="center"><a accesskey="n" href="net.html">Network</a></td>
<td align="center"><a accesskey="s" href="sol.html">Solaris</a></td>
<td align="center"><a accesskey="p" href="hpux.html">HP-UX</a></td>
<td align="right"><a accesskey="a" href="aix.html">AIX</a></td>
</tr>
</tbody>
</table>
<hr></div>
<div class="chapter" lang="en">
<div class="titlepage">
</div>
</div>
<div align="CENTER">
<A HREF="http://www.explainxkcd.com/wiki/index.php/1854"><IMG SRC="fig/xkcd_refresh_types.png" title="The hardest refresh requires both a Mac keyboard and a Windows keyboard as a security measure, like how missile launch systems require two keys to be turned at once." alt="Refresh Types"></A><BR>
</div>
<H1>IPMI</H1>
<H2>101</H2>
<PRE>
There are two modes.
- "ipmitool cmd" on the machine itself, which doesn't go thru the network, and always work.
- "ipmitool -H bmc.ip.addr cmd" to use the "lanplus" network interface, which will require working IP stack, and require for example for the bmc ip to be pingable and reachable via the IPMI protocol. (port ??)
If ipmitool doesn't work, returning with "Error in open session response message : no matching cipher suite"
try
ipmitool -C 3 ...
to specify cipher to use.
</PRE>
<H2>102</H2>
<PRE>
Dell DRAC and HP iLo are implementation of IPMI.
Could use embeded NIC (shared with OS)?
Dedicated NIC provides more functionality.
when enabled, can ssh in (eg into the iDRAC port).
The standard iDRAC supports IPMI and basic web interface to look at temperature, fru status, reboot machine, etc.
The Enterprise iDRAC adds remote (GUI) console, AD-based user auth.
IPMI sol (serial over lan), isol, etc is avail from iDRAC, presumably even in standard version.
</PRE>
<PRE>
For Dell, IPMI don't seems to have way to change hyperthreading settings, not even in the delloem commands.
Disabling hyperthreading seems to req interactive BIOS changes.
Linux kernel param noht doesn't work (or was it only supposed to work at install time?)
~~
sky4-5 has decent config, bio2 don't.
There is likely an IPMI-only w/o iDRAC on the machine as well, eg the low end 1U box
</PRE>
<H2>CMD</H2>
<PRE>
ipmitool lan print 1 # iDRAC interface (could be shared if set so). 1 refer to channel 1, which typically only channel.
ipmitool lan set 1 ipaddr 10.10.58.140 # set iDRAC interface IP (should be, need to verify it works)
ipmitool lan set 1 defgw ipaddr 10.10.58.1 # set default gateway
ipmitool lan set 1 vlan id 20 # if in network require vlan id tagging, set to use the correct id (eg: 20)
ipmitool sdr # print environment statistics
ipmitool fru
ipmitool sol info # supposed to provide serial over lan info
ipmitool shell # get to ipmi> shell so that don't have to invoke the tool each time. it is NOT gettingo inside iDRAC shell.
ipmitool delloem lan # dell specific command. lan section set whether to use dedicated NIC or shared.
# but seems like iDRAC is configured yet this info can still be not set.
# so iDRAC can probably speak IPMI and interfaces with it, but it is not an instance of IPMI.
# if work, just tell whether dedicated or shared nic
ipmitool delloem lan set dedicated # disable use of shared IPMI. set once iDRAC enterprise module is installed.
ipmitool delloem powermonitor # display power usage info
ipmi tag to vlan 20 (default is untagged):
ipmitool lan set 1 vlan id 20
ipmitool user list 1 # list users configured to use ipmi
ipmitool chassis identify # blink location becon (typically blue) to help find machine
ipmitool chassis bootdev pxe options=persistent # change to use pxe boot by default on every boot.
ipmitool -I lan -U root -P calvin -H 10.8.1.76 chassis bootdev pxe options=persistent
# issue the ipmi command remotely to a remote host using the remote iDRAC interface
ipmitool fru # list all fru (eg 4 node in 2U chassis would list service tag and board serial of each node)
ipmitool dcmi set_asset_tag # set asset tag, presumably for the whole chassis ...
ipmitool dcmi asset_tag # retrieve asset tag info
ipmitool dcmi get_temp_reading # cpu temperature
ipmitool sensor # many sensor info, may include inlet temperature
ipmitool sel # system event log, overview
ipmitool sel list # get list of events and dates. eg, PSU AC lost. additional example output:
3 | 08/12/2021 | 16:49:18 | Unknown #0x2e | | Asserted
4 | 08/12/2021 | 16:49:18 | Memory #0x45 | Redundancy Degraded from Fully Redundant | Asserted
1 | 10/22/2020 | 00:44:54 | Event Logging Disabled #0x72 | Log area reset/cleared | Asserted
2 | 01/25/2021 | 19:00:24 | Power Supply #0x18 | Fully Redundant | Asserted
3 | 09/11/2021 | 22:23:05 | Memory #0x1b | Monitor | Asserted
4 | 10/27/2021 | 17:32:09 | Memory #0x1b | Monitor | Asserted
ipmitool sensor | grep Current # on Dell C6400 see output like this, and 1 PSU likely dont have AC power:
Current 1 | 0.000 | Amps | ok | na | na | na | na | na | na
Current 2 | 2.400 | Amps | ok | na | na | na | na | na | na
# if IPMI isn't working, in RHEL 6, it is a kernel module that need to be started
# for machines with module:
module load shared
module load ipmitool
service ipmi start
</PRE>
<A ID="hexcode"></A>
<H5>Finding Dell Chassis ServiceTag from a node using hexcode</H5>
For dell C6320 and the like, the chassis has its own service tag. <BR>
hex code via ipmi "raw" command can retrieve them. <BR>
conversion would be needed. eg <BR>
xxd -r reverse hex conversion to ascii, avail from vim-common rpm <BR>
<PRE>
# Dell C6320 vintage 2017
ipmitool raw 0x30 0xc8 0x01 0x00 0x0b 0x00 0x00 0x00 | xxd -r
# additional ipmi raw command to try to get chassis code
# The C6220II should use the same commands as the C6320, but if the system is too old the original version of this unit won’t work.
# The C6100 and C6220 do not have a unique chassis service tag to query.
# To query the Set FCB value run ALL of the following commands in this order
# (It generate and execute some sort of reservation, each command should produce some output,
# till the last one should yield a serial number of the chassis)
ipmitool raw 0x30 0xC8 0x01 0x00 0x02 0x00 0x00 0x00
ipmitool raw 0x30 0xC8 0x01 0x00 0x02 0x00 0x00 0x00 0x00
ipmitool raw 0x30 0xC8 0x01 0x00 0x02 0x00 0x00 0x00 0x01
ipmitool raw 0x30 0xC8 0x01 0x00 0x0c 0x00 0x02 0x00 0x00 0x00 | xxd -r
</PRE>
<H5>Programming Dell Chassis from a node using IPMI RAW commands</H5>
While some chassis have svc tag programmed out of factory, others are not.
For inventory purpose, if one wish to program in the service tag so that they can be queried.
Here is an example of steps to program svc tag of "bjhkqd2" to the chassis via one of its node. <BR>
<TT>bjhkqd2 in hex is 62 6a 68 6b 71 64 32</TT> <BR>
<PRE>
b j ...
vvvv vvvv
ipmitool raw 0x30 0xC8 0x00 0x00 0x0B 0x00 0x00 0x00 0x0B 0x00 0x11 0x0A 0x62 0x6a 0x68 0x6b 0x71 0x64 0x32
## commit
ipmitool raw 0x30 0xC8 0x01 0x00 0x02 0x00 0x00 0x00
</PRE>
<H5>IPMI RAW</H5>
<A HREF="http://www.dell.com/downloads/global/power/ps4q07-20070387-Babu.pdf">
Dell "Using IPMITOOL RAW commands for remote management"</A>.
It is pdf from dell. includes IPMI, BIOS config for SOL, escape seq keys, etc.
<PRE>
# raw ipmi command can potentially set what temp server shutdown when overheat
# raw cuz it is hex code being fed to the service processor.
ipmitool -I open sel list
# list raw events... maybe plaform specific.
# Over temperature raw 0x04 0x02 0x04 0x01 0x01 0x01 0x09 00 00
# Over voltage raw 0x04 0x02 0x04 0x02 0x11 0x01 0x09 00 00
</PRE>
<A ID="asus"></A>
<H5>Obscure IPMI cmd for Asus</H5>
May only be relevant to ESC4000 G4
<PRE>
ipmitool raw 0x32 0x71 0x00 0x01 0x01 # set ipmi/bmc to use shared nic
ipmitool lan set 8 cipher_privs XaaaaaaaaaaaXXX # reset cipher suite allowed for channel 8
</PRE>
<H5>Obscure IPMI cmd for Penguin Relion 2940 w/ Gigabyte mboard</H5>
<PRE>
ipmitool raw 0x2e 0xcc 0x5e 0x2b 0 0x0C 0x01 0x02 # set ipmi/bmc to use share nic
# there maybe a bios settings to put the NIC in shared mode, but some notes has it that it wasn't very reliable,
# so support provided the ipmitool raw command.
# this was for the 4 nodes in 2U w/ intel E5-2640 v4 CPU
</PRE>
<H5>Dell ipmitool </H5>
Dell maintain their <A HREF="https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0992n">own version of ipmi</A> that handles complex situation that RHEL7 ipmitool didn't.
eg: <BR>
<BR>
eg, this command is needed by some new Dell eg PE R7525 :
<BR>
<BR>
/local/tin/dell_tool/iDRACTools/ipmitool/usr/bin/ipmitool delloem lan set shared with lom1
<!-- c00.greta.local -->
<H5>GNU freeipmi / ipmi-oem</H5>
Ref:
<A HREF="https://www.gnu.org/software/freeipmi/manpages/man8/ipmi-oem.8.html">
gnu freeipmi</A> aka ipmi-oem.
<PRE>
In CentOS 7 has freeipmi* rpm
may provide oem commands to set ipmi
eg appro, which is/uses Intel S2600JF
oem get-bmc-services
eg with Dell
/usr/sbin/ipmi-sensors
ipmi-oem Dell get-power-capacity
ipmi-oem Dell get-active-lom-status
/usr/sbin/bmc-config ??
</PRE>
<H5>Serial over IPMI SoL </H5>
This doesn't quite redirect grub login to serial yet.
But don't really want to redirect VGA to serial.
Only want to spawn a serial login session so that <TT>ipmi sol</TT>
can gain a login.
<BR>
inittab and getty no longer works in RHEL7, below config is for systemd.
<BR>
<PRE class="cf">
## /etc/systemd/system/mgetty.service ::
[Unit]
Description=Smart Modem Getty(mgetty)
Documentation=man:mgetty(8)
Requires=systemd-udev-settle.service
After=systemd-udev-settle.service
[Service]
Type=simple
ExecStart=/sbin/mgetty -r -s 115200 /dev/ttyS1
# -r is for direct connection (ie null modem)
Restart=always
PIDFile=/var/run/mgetty.pid.ttyS1
[Install]
WantedBy=basic.target multi-user.target
</PRE>
<PRE>
ls -Z /usr/sbin/mgetty
ls -Z /var/log/mgetty.*
chcon --type getty_exec_t /var/log/mgetty.*.log
### still need more tweaking, but don't really prevent mgetty from creating a login session.
systemctl start mgetty.service
systemctl enable mgetty.service
# enable will create sym links in /etc/systemd/system/basic.target.wants and multi-user.target.wants
# create ipmi user #3 to have access
ipmitool user list
ipmitool user set password 3 NewIpmiPwForUserNum3 # if need to change password
ipmitool user enable 3
ipmitool channel setaccess 1 3 ipmi=on privilege=4
ipmitool user priv 3 4 1
</PRE>
Generic usage:
<PRE>
# using serial over lan (sol, ipmi 2.0)
ipmitool -H hostname -U username -P pw list # see list of user
ipmitool -H hostname -U username -P pw sol activate # initial sol connection
ipmitool -I lan -U $username -P $pass -H $nodename sol activate # activte serial over lan
# should drop into console. escape seq to control IPMI session:
# ~. = exit
# ~ ^Z = suspend
# ~B = send break
# ~? = help
# Esc R Esc r Esc R = ctrl-alt-del
# Esc 1 = F1
ipmitool -I lan -U $username -P $pass -H $nodename sol deactivate # disconnect prev sol, clean up measure
</PRE>
Specific usage example:
<PRE>
csh
ipmitool -I lanplus -U root -P ipmipw -H 172.10.0.128 sol info # 115.2 should be default speed
ipmitool -I lanplus -U root -P ipmipw -H 172.10.0.128 sol activate instance=1 # maps to /dev/ttyS1
# ~? = help (but note that ssh also use ~ as escape, so may need ~~?
# ~. = terminate session
ipmitool -I lanplus -U root -P ipmipw -H 172.10.0.128 sol deactivate # if some old session was left running
exit
rm ~/.history
# most ipmitool over lan uses 623.
# dell idrac says use UDP 623
# but on SMC, can use telnet to port 623 and get connection established, so it is over TCP??!!
# test connectivity eg:
telnet 172.10.0.128 623
possible test for connectivity:
traceroute --port=623 172.10.0.128
traceroute --port=33434 is the default, it is Destination UDP port that this pokes at.
</PRE>
ref:
<A HREF="http://snarkybrill.blogspot.com/2015/08/mgetty-in-systemd-for-modem-dial-in.html">
http://snarkybrill.blogspot.com/2015/08/mgetty-in-systemd-for-modem-dial-in.html</A>
<BR>
# http://thenetworksherpa.com/using-ipmi-serial-lan-consoles/
<BR>
# ref: (for windows, but mostly applicable to linux)
<BR>
# https://www.symantec.com/connect/articles/how-do-sol-serial-over-lan-aka-console-redirection-dell-servers
<BR>
https://topics-cdn.dell.com/pdf/idrac9-lifecycle-controller-v3212121_users-guide_en-us.pdf
page 31,32 discuss firewall req for iDRAC. states UDP 623.
Remote console need VNC on port TCP/5900 and 5901
<H5>Remote Power Cycle using IPMI</H5>
<PRE>
linux machine with ipmitool (provides ipmi cli) installed, so can also do:
ipmitool -H hostname -U username -P pass power status
credentials are IDRAC credentials
*should* be blade specific when hostname is iDRAC of blade even when description say chassis. Have not been able to get it to work.
ipmitool -I lan -U $username -P $pass -H $nodename mc info # ipmi ver info
ipmitool -I lan -U $username -P $pass -H $nodename chassis power status
ipmitool -I lanplus -U $username -P $pass -H $nodename chassis power status
ipmitool -I lanplus -U $username -P $pass -H $nodename chassis power down
ipmitool -I lanplus -U $username -P $pass -H $nodename chassis power on
ipmitool -I lanplus -U $username -P $pass -H $nodename chassis power cycle
# nodename is the DNS name of the ipmi address of a host, eg ipmi-HOSTNAME
# most of time use lanplus interface
# lan may work for some basic commands, but more sophisticated command req it to be run over lanplus
# dell def username pass is root calvin
ipmitool mc reset cold # reset the service processor on the node itself, cold? booting it?
# dell C6320 need the following to configure IPMI:
ipmitool delloem lan set shared with lom1
ipmitool -H
sol payload enable 1 2
# these supposed to work, but haven't been able to get them to work :(
# may need to configure grub level serial redirect
# but realistically haven't been able to connect to serial, not just a blank screen...
</PRE>
<H5>Verify status</H5>
<PRE>
ipmitool lan print 1
ipmitool lan print | grep 'IP Address'
</PRE>
<BR>
<H2>Finding power usage</H2>
<PRE>
# Dell C6320, 32 skylake cores in use. drawing 336W
# Merger reads busy server drawing 3.7A x 2 feed x 202V / 4 nodes per chassis = 373W.
# ipmitool sensor | grep 'Pwr Consumption'
Pwr Consumption | 336.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
# on, but idle skylake server, 96 GB RAM
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
# on, but idle skylake server, 192 GB RAM
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
# when machine is ipmi off:
Pwr Consumption | 0.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Power on power consumption remotely monitored by ipmitool :
Pwr Consumption | 0.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 56.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 112.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 126.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 140.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 154.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 154.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 154.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 252.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 210.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 196.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 182.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 182.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
Pwr Consumption | 168.000 | Watts | ok | na | na | na | 1106.000 | 1218.000 | na
</PRE>
<BR><HR><BR>
<H2>Ref</H2>
<A HREF=sol.html#ipmi>Solaris#IPMI</A>
<BR>
<BR>
<BR><HR>
<div align="CENTER">
[Doc URL: <A HREF="http://tin6150.github.io/psg/psg/ipmi.html">
http://tin6150.github.io/psg/psg/ipmi.html</A>]
<BR>
(cc) Tin Ho. See
<A HREF=psg.html#cc>main page</A>
for copyright info. <BR>
<HR>
<A HREF="http://www.taos.com"><IMG SRC="banner/taos_banner1.gif" width="728" height="98"></A>
</div>
<div class="sig"><BR>
hoti1<BR>
bofh1</div>
</body>
<!-- Google analytics new tracking code ga.js. Will actually need to add this code to every page for full tracking! (still the case in 2011?) Using my gmail login 2011.0617 updated with code for http://tin6150.github.io/psg/psg.html --> <script type="text/javascript"> var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-4515095-4']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script>
</html>