NVME pass through

[jorgefm]

Hi all,

Is there any possibility to have NVME pass through support implemented like for the SATA devices?

I've been trying to access to the OPAL info from a M.2 device connected through an ASMedia ASM2362 USB to NVMe bridge enclosure with no luck. After changing a little the 'DtaDevLinuxNvme::identify' method I've been able to access the information but I don't have more information to pass the NVME_SECURITY_RECV|SEND data in 'DtaDevLinuxNvme::sendCmd' embedded in a cdb structure.

To identify the device I've follow the 'https://www.smartmontools.org/ticket/1221' info.

`
void DtaDevLinuxNvme::identify(OPAL_DiskInfo& disk_info)
{
LOG(D4) << "Entering DtaDevLinuxNvme::identify()";

uint8_t ctrl[4096];
sg_io_hdr_t sg;
uint8_t sense[32]; // how big should this be??
uint8_t cdb[16];
memset(&cdb, 0, sizeof (cdb));
memset(&sense, 0, sizeof (sense));
memset(&sg, 0, sizeof (sg));

// https://www.smartmontools.org/ticket/1221
cdb[0] = 0xe6; // asmedia pass through
cdb[1] = NVME_IDENTIFY;
cdb[3] = 1;    // CDW10.byte0

/*
 * Set up the SCSI Generic structure
 * see the SG HOWTO for the best info I could find
 */
sg.interface_id = 'S';
sg.dxfer_direction = SG_DXFER_FROM_DEV;
sg.cmd_len = sizeof (cdb);
sg.mx_sb_len = sizeof (sense);
sg.iovec_count = 0;
sg.dxfer_len = 4096;
sg.dxferp = ctrl;
sg.cmdp = cdb;
sg.sbp = sense;
sg.timeout = 60000;
sg.flags = 0;
sg.pack_id = 0;
sg.usr_ptr = NULL;
//    LOG(D4) << "cdb before ";
//    IFLOG(D4) hexDump(cdb, sizeof (cdb));
//    LOG(D4) << "sg before ";
//    IFLOG(D4) hexDump(&sg, sizeof (sg));
/*
 * Do the IO
 */
if (ioctl(fd, SG_IO, &sg) < 0) {
    LOG(D4) << "cdb after ";
    IFLOG(D4) DtaHexDump(cdb, sizeof (cdb));
    LOG(D4) << "sense after ";
    IFLOG(D4) DtaHexDump(sense, sizeof (sense));
    disk_info.devType = DEVICE_TYPE_OTHER;
    return;
}

disk_info.devType = DEVICE_TYPE_NVME;
uint8_t *results = ctrl;
results += 4;
memcpy(disk_info.serialNum, results, sizeof(disk_info.serialNum));
results += sizeof(disk_info.serialNum);
memcpy(disk_info.modelNum, results, sizeof(disk_info.modelNum));
results += sizeof(disk_info.modelNum);
memcpy(disk_info.firmwareRev, results, sizeof(disk_info.firmwareRev));

return;

}`

[hashkool]

I am not sure if this is applicable to your situation, but this sounds like your drive works as an "external drive"?

From arch:

Using sedutil with external drives, even if sedutil claims the drive supports OPAL, has been reported as bricking them. For example, WD My Book external hard drives owners should set up encryption with the WD Security utility in Windows, and then use sg_raw for unlocking it. There is also a utility that supports setting up and unlocking such drives here: wdpassport-utils

On the other hand, it seems like people got things working on a PI (ASMedia 1153 family USB to SATA bridge) with a Samsung: https://forums.raspberrypi.com/viewtopic.php?t=306817.

[jorgefm]

Hi, thanks for the info!

Using the sedutil-cli against the M.2 device attached to the mainboard it works ok, but it's not very comfortable when you have to initialize several disks. Then, I was looking for a external drive to do that. I've tested several devices based in the Realtek RTL9210 and in the ASMedia ASM2362, but both are not supported in sedutil to be able to detect OPAL info and set it up.

The USB to SATA is working ok, but the USB to NVMe is a little behind. Maybe is needed some reverse engineering to capture the OPAL data when is working ok attached on mainboard but the hard point is how to map this info in a ASMedia pass through command without info about the ASM2362 device.

[hashkool]

Please let me know if I only add noise (I know little about this stuff). I see there is a patch for linux-arm which moves opal stuff from the pci layer to core: https://lore.kernel.org/linux-arm-kernel/[email protected]/#Z31drivers:nvme:host:pci.c

If you have an arm pc you could maybe see if you get different results? I am not sure if these patches have been mainlined already btw.

[jorgefm]

Yes, the OPAL support for NVMe is already in the linux kernel and sedutil use it nicely... the problem is when you try to access this functionality through an intermediate USB to NVMe bridge. You must know how to map your request onto an ioctl the bridge (ASM2362 chip) is able to recognize and send it to the NVMe device attached... But I don't expect a lot from ASMedia to get the ASM2362 datasheet...

[hashkool]

I was under the impression that an unpatched kernel assumes that opal is tied to pci or sata, and that these code paths wouldn't be reached if using an usb bridge, as the kernel assumes it is talking to an usb disk. I thought the kernel patch would help in this situation, as it lifts the opal stuff from interface level to disk level. Then the interface wouldn't matter anymore (usb/pcie)

I think you know more about this stuff then me however.

[jorgefm]

Well, I've not analized the problem to the kernel level because I've got sedutil working when the NVME device is connected on the mainboard (the https://github.com/Drive-Trust-Alliance/sedutil/blob/master/linux/DtaDevLinuxNvme.cpp code path is used).

When the device is attached to an enclosure with ASM2362 is detected has an SATA device and no info is showed about the device. Forcing to identify it using the code from the first message (extracted from smarttool tool) I was able to access to that information, then I guess that to get access to the rest of functionality I've to go the same way but I've not found information about it...

[hashkool]

EDIT: the fact that crystal disk had to tell ASM2362 to pass through nvme commands, makes me wondering how brittle this solution would be. You would rather have the kernel enumerate the disk transparantly as an nvme disk, but I guess it doesnt show up as /dev/nvmex when attached to enclosure?

---

Hmm, I am out of my depth here.

From this thread, it seems a bit of a hit and miss with adapters: https://forum.level1techs.com/t/nvme-to-usb-3-1-enclosure-buggy-in-linux-rtl9210b-chipset/199752

There are lots of drive specific quirks addressed on the pci level: torvalds/linux@107b4e0.
I wonder

1. if the ioctl will take drivers/nvme/host/pci.c into account, and
2. wether your adapter will faithfully pass through all info the kernel is poking for.

[hashkool]

Edit 2: Some adapters seem to be working, from a quick glance: #115

[Blacklands]

Has anyone been able to test an adapter using PCIe Tunneling (whether it's Thunderbolt 3 or fully-featured USB4/TB4)?
I've been hoping that those would be able to pass through the OPAL commands similar to a direct internal NVME -> PCIe connection.

I just got an enclosure with the Asmedia ASM2464PD just to then find out that my only Thunderbolt host device doesn't play nicely with it, and just falls back to 10 Gb/s USB Legacy Mode (and there the passthrough does not work, sedutil doesn't detect the drive as an SED). :(
So I still haven't been able to answer this.

Apart from that, I've only seen SATA -> USB adapters working, but there might be NVME -> USB ones out there that work too (the ones I have don't).

I also wonder if there's a difference between Windows and Linux. I personally can only test Windows (11) right now.

[JaBoMa]

I did not have problem with initialization of my Samsung 970 EVO Plus (M.2 NVMe) 1TB disk on my 12 Years old HP EliteBook 8540w, I was operating using DTA Sedutil Rescue System v.1.20 i.e. RESCUE32.img.gz from exec-1.20.0.zip or from exec-1.20.0.tar.gz that you can find under https://github.com/Drive-Trust-Alliance/exec/tags, and the SSD was connected via the AXAGON EEM2-UG2 USB-C to NVMe M.2 SSD interface. It was working well together. The rescue system was booted from USB memory Stick.
I was able to initialize that SSD as bootable TCG OPAL encrypted disk. The commands sedutil-cli --scan, sedutil-cli --query, and sedutil-cli --listLockingRanges were also working well,
The sedutil-cli.exe v. 1.20.0, that was started from Windows 7 on the same laptop, can't see that disk - connected the same way - at all.
I didn't try to reach that disk from the same laptop using sedutil-cli v.1.20.0 under Ubuntu.
I hope I did not add any noise to this thread. I was just checking possibilities to reach data from new SSD on other PC in the case the hosting PC fails.
ATB

[jorgefm]

Edit 2: Some adapters seem to be working, from a quick glance: #115

Great! This issue has a lot of interesting info!! I'll try to get a RTL9210b adapter to test...

Thank you!

[youk]

Other people already provided a lot of useful info. From my experience with SATA/USB I can only confirm that this heavily depends on the bridge used in the enclosure. Most vendors don't implement in firmware anything beyond the basic use cases. So the wisest thing is to use an enclosure/bridge already reported as working. For SATA/USB I was once advised to look at Sabrent enclosures and that worked out.