My best theoretical guess is that the bug may lie somewhere here in the detection code.

It may need to be updated to detect if the drive is actually APA-Jailed and act accordingly. It doesn't seem like it expects both APA and MBR/GPT on the same drive.

After editing that function to detect APA Jail and just go regardless, it becomes very inconsistent and throws an error: pfs: Error: Disk error partition 0, block 4296, err -48 when attempting to save the config.

I'm not even sure if this executes at this point because I don't see it in the UDPTTY log yet (or maybe it's not caught early enough?)

EDIT: Yeah, it only starts together with the HDD device once the module is started, I just noticed. Then that means this (probably) has nothing to do with the config errors I've seen. In any case, I'll leave the edited function below. APA-Jail leaves a very helpful mark at 0xE0 which can be used to identify it.

Here's the changed function:

// Returns 1 for MBR/GPT, 0 for APA, and -1 if an error occured
int hddDetectNonSonyFileSystem()
{
    int result = -1;
    int is_apaj = 0;
    // Allocate memory for storing data for the first two sectors.
    u8 *pSectorData = (u8 *)malloc(512 * 2);
    if (pSectorData == NULL) {
        LOG("hddDetectNonSonyFileSystem: failed to allocate scratch memory\n");
        return -1;
    }

    // Trying to load the APA/PFS irx modules when a non-sony formatted HDD is connected (ie: MBR/GPT  w/ exFAT) runs
    // the risk of corrupting the HDD. To avoid that get the first two sectors and perform some sanity checks. If
    // we reasonably suspect the disk is not APA formatted bail out from loading the sony fs irx modules.
    result = fileXioDevctl("xhdd0:", ATA_DEVCTL_READ_PARTITION_SECTOR, NULL, 0, pSectorData, 512 * 2);
    if (result < 0) {
        LOG("hddDetectNonSonyFileSystem: failed to read data from hdd %d\n", result);
        free(pSectorData);
        return -1;
    }

    // Check for APA-Jail signature
    if (strncmp((const char*)&pSectorData[0xE0], "APAJ", 4) == 0) {
        // Found APA-Jail.
        LOG("hddDetectNonSonyFileSystem: found APA-Jail\n");
        is_apaj = 1;
    }

    // Check for MBR signature.
    if (pSectorData[0x1FE] == 0x55 && pSectorData[0x1FF] == 0xAA) {
        // Found MBR partition type.
        LOG("hddDetectNonSonyFileSystem: found MBR partition data\n");
        if (is_apaj == 0)
            result = 1;
        else
            result = 0;
    } else if (strncmp((const char *)&pSectorData[0x200], "EFI PART", 8) == 0) {
        // Found GPT partition type.
        LOG("hddDetectNonSonyFileSystem: found GPT partition data\n");
        if (is_apaj == 0)
            result = 1;
        else
            result = 0;
    } else if (strncmp((const char *)&pSectorData[4], "APA", 3) == 0) {
        // Found APA partition type.
        LOG("hddDetectNonSonyFileSystem: found APA partition data\n");
        result = 0;
    } else {
        // Even though we didn't find evidence of non-APA partition data, if we load the APA irx module
        // it will write to the drive and potentially corrupt any data that might be there.
        LOG("hddDetectNonSonyFileSystem: partition data not recognized\n");
        result = -1;
    }

    // Cleanup and return.
    free(pSectorData);
    return result;
}

EDIT2: Yeah, I was right the first time -- it is not caught early enough in the UDPTTY. Reverting the change drops this:

HDDSUPPORT LoadModules 2
HDDSUPPORT LoadModules done
Unknown device 'hdd'
Known devices are  tty:(TTY via SMAP UDP)  rom:(ROM/Flash)  cdrom:(CD-ROM )  mc:(Memory Card)  mass:(FATFS driver)  iso:(ISOFS)  genvmc:(genvmc)  dev9x:(DEV9)  xhdd:(XHDD)  host:(fsys driver)
HDD: Status is -19

And when attempting to start the APA/HDD module:

HDDSUPPORT Init
HDDSUPPORT LoadModules 3
ETHSUPPORT ApplyConfig
HDDSUPPORT LoadModules done
HDDSUPPORT LoadSupportModules
hddDetectNonSonyFileSystem: found MBR partition data
HDDSUPPORT LoadSupportModules bailing out early...
Unknown device 'pfs'
THEMES Trying to set again theme: <OPL>
Unknown device 'pfs'
Unknown device 'pfs'
Unknown device 'pfs'
Unknown device 'pfs'
Unknown device 'pfs'
Unknown device 'pfs'
Unknown device 'pfs'
Unknown device 'pfs'
Unknown device 'hdd'
HDDSUPPORT LoadModules 4
HDDSUPPORT LoadModules done
Unknown device 'hdd'
HDD: Status is -19

EDIT3: And now it thinks it's unformatted... It's very inconsistent. I'm not sure if it is actually corrupting something.

HDDSUPPORT LoadModules 2
HDDSUPPORT LoadModules done
HDD: Status is 1

I've JUST noticed that the Debug builds cause issues with writing to PFS partitions.

Release build writes the config to the PFS partition fine with this modification, provided that you DON'T start the APA mode before writing the config.

Now, the issue remains - it doesn't want to read the config from the PFS partition. I can see the contents just fine in the __common partition via LaunchELF though (since that's where I have it configured), so it knows at least where to look for the conf_hdd.cfg and can access that.

I forgot to also mention that it happened to read the config off the HDD randomly during my previous testing, but I can't tell you the exact details because it really was just random. I hadn't changed anything other than trying different debug builds at that time.

EDIT: I've just edited the config file via wLaunchELF, deleted a whole bunch of directories in __common and then launched OPL via wLaunchELF, and then the config was read?!

But, after power cycling the console and then launching through OSDSYS, it doesn't read the config.

Modules, perhaps? Partition mounting?

EDIT2: I can't replicate it anymore... It's super inconsistent. I've tried to create a +OPL partition now (with the config applied) and... it wrote the config into it just fine, but it STILL can't read.

EDIT3: Even after redoing EVERYTHING from scratch, it still misbehaves. But - there is one key common thing I've noticed.

Every time it worked, the APA HDD start mode was set to "Auto" even if the mode was disabled in the config file.

Setting it manually via code did nothing.

As I'm unfamiliar with the codebase, I assume that there is something that sets it automatically and whenever that happens, it works. I'll take a deeper look.

Can you try this build and see if it fixes your issue?
https://github.com/pcm720/Open-PS2-Loader/releases/tag/latest

It just limits BDM ATA mode to UDMA 4.

So, I've decided to create the test environment in PCSX2 in hopes to be able to debug something.

Unfortunately, over there, it all works properly. Config is read/written to the PFS partition just fine (but ONLY after disable BDM/mass config writers).

But, during testing, I've also noticed that it wrote the config onto the exFAT partition on the HDD as well. I did notice it said "saved to mass0" when I saved the config.

Could it be that the BDM driver was writing to the exFAT partition just fine and is simply unable to read the config?

@pcm720 I was writing this just as you posted your comment. I'll give it a shot.

but ONLY after disable BDM/mass config writers

Currently, BDM backend in upstream OPL always forces maximum UDMA mode it gets from your drive's response to ATA_IDENTIFY command without taking into account that some adapters can't actually work on these modes.
These can be unstable at UDMA 5/6, so this explains why it works when BDM is disabled, but doesn't when BDM is enabled.

As far I as understand, using APA Jail with OPL can cause the issue you have here simply because OPL treats your HDD as two separate devices, hdd0: and mass0:. However, since UDMA mode is set on the lower level, it obviously applies to both APA and BDM backends.

So, your build, as-is, still has the issue of reading the config from the exFAT partition. However, the config is written to it just fine.

The games work fine for me in both builds now (with your fork or otherwise) after I've transferred over the PCSX2 HDD image to a real HDD.

This entire time I've been working under the assumption that exFAT is read-only on the HDD and that's the reason why it doesn't work. (I didn't think about VMC and other stuff)

As long as the config can be read from the HDD (PFS or exFAT) I think that'll be fine enough.

Ideally there would be an option where to save (hdd0: or mass0:) but honestly, the more we move away from PFS the better IMO.

So the crux of the issue is - configs simply aren't read from BDM HDD devices at bootup.

The fix would probably be something related to checkLoadConfigHDD. Adding an APA-Jail detect there and then initializing BDM in case a valid GPT/MBR is detected should theoretically fix it, right? (Or just simply detecting a GPT/MBR is enough, maybe)

So, on PCSX2, making this change fixes the config bootup read:

static int checkLoadConfigHDD(int types)
{
    int value;
    char path[64];

    hddLoadModules();

    if (hddDetectNonSonyFileSystem() == 1)
        return checkLoadConfigBDM(types);

    hddLoadSupportModules();

    snprintf(path, sizeof(path), "%sconf_opl.cfg", gHDDPrefix);
    value = open(path, O_RDONLY);
    if (value >= 0) {
        close(value);
        configEnd();
        configInit(gHDDPrefix);
        value = configReadMulti(types);
        config_set_t *configOPL = configGetByType(CONFIG_OPL);
        configSetInt(configOPL, CONFIG_OPL_HDD_MODE, START_MODE_AUTO);
        return value;
    }

    return 0;
}

I still have to figure out why it doesn't work on real hardware (I'll try the same thing on your fork and observe).

EDIT: UDMA4 doesn't seem to affect anything in this regard, sadly...

I'd love to get some info in the TTY at this point of execution so I can see why it fails exactly. (PCSX2 working doesn't help much right now lol)

EDIT2: Oh I see, I'm a dummy - this should happen if it fails anyway...

    // First, try the device that OPL booted from.
    if (!strncmp(pwd, "mass", 4) && (pwd[4] == ':' || pwd[5] == ':')) {
        if ((value = checkLoadConfigBDM(types)) != 0)
            return value;
    } else if (!strncmp(pwd, "hdd", 3) && (pwd[3] == ':' || pwd[4] == ':')) {
        if ((value = checkLoadConfigHDD(types)) != 0)
            return value;
    }

    // Config was not found on the boot device. Check all supported devices.
    //  Check USB device
    if ((value = checkLoadConfigBDM(types)) != 0)
        return value;
    // Check HDD
    if ((value = checkLoadConfigHDD(types)) != 0)
        return value;

Interesting, I've added a few modifications (added a gEnableBdmHDD = 1; and bdmLoadModules(); during checkLoadConfigHDD, and some logs in checkLoadConfigBDM), here's the interesting part of the PCSX2 log:

CONFIG No file mc?:OPL/conf_opl.cfg.
CONFIG No file mc?:OPL/conf_last.cfg.
CONFIG No file mc?:OPL/conf_apps.cfg.
CONFIG No file mc?:OPL/conf_network.cfg.
CONFIG No file mc?:OPL/conf_game.cfg.
checkLoadConfigBDM: starting...
checkLoadConfigBDM: failed to find anything!
HDDSUPPORT LoadModules 0

Notice how the HDD support is loaded after checkLoadConfigBDM. That's not very good for this issue.

Later, when checkLoadConfigHDD happens with the modification, this is the log:

BDM: connecting device ata0p0 id=0x0
atad: Driver loaded.
Found MBR disk
Found partition type 0x17
BDM: connecting device ata0p1 id=0x17
Found partition type 0x07
BDM: connecting device ata0p2 id=0x7
BDM: ata0p0 mounted to MBR
BDM: ata0p2 mounted to fatfs
	-- ID=44, ret=0
[XHDD]:
	-- ID=45, ret=0
HDDSUPPORT LoadModules done
hddDetectNonSonyFileSystem: found MBR partition data
BDMSUPPORT LoadModules
[BDM]:
MODULE ALREADY LOADED (7)
[BDMFS_FATFS]:
MODULE ALREADY LOADED (8)
[USBD]:
MODULE ALREADY LOADED (9)
[USBMASS_BD]:
MODULE ALREADY LOADED (10)
bdmLoadBlockDeviceModules loading hdd drivers...
[BDMEVENT]:
HDDSUPPORT LoadModules 1
MODULE ALREADY LOADED (11)
HDDSUPPORT LoadModules done
BDMSUPPORT Modules loaded
checkLoadConfigBDM: starting...
checkLoadConfigBDM: path is: mass0:
CONFIG No file mass0:/conf_last.cfg.
CONFIG No file mass0:/conf_apps.cfg.

So, I believe that BDM needs to be initialized with HDD support earlier to try to fix this issue. I'll look into that next...

Yep, that's it! I've fixed it simply by doing this:

static int checkLoadConfigBDM(int types)
{
    char path[64];
    int value;

    hddLoadModules();

    // check USB
    if (bdmFindPartition(path, "conf_opl.cfg", 0)) {
        LOG("checkLoadConfigBDM: path is: %s\n", path);
        configEnd();
        configInit(path);
        value = configReadMulti(types);
        config_set_t *configOPL = configGetByType(CONFIG_OPL);
        configSetInt(configOPL, CONFIG_OPL_BDM_MODE, START_MODE_AUTO);
        return value;
    }

    return 0;
}

I've added a call to hddLoadModules() in here and it works! The only thing I need to check now is if it breaks anything with the classic APA configuration.

EDIT: Okay, I might've spoken too soon, loading from LaunchELF it still breaks lol, but, it did affect things in a positive way.

EDIT2: I've also tried UDMA4 but still nothing. It's inconsistent on a real PS2 now. I'll have to test this with someone who has a better SATA adapter to just double check if that's an issue on my part.

After testing with a friend of mine (who has a 1TB PNY SSD and a Bitfunx SATA adapter (the exact one I have too)) it reads the configs consistently.

It appears to be an issue on my end, then! The BDM inconsistency issue (#1437) is causing inconsistencies with the config reading as well, so, I think that will resolve this issue once it is fully figured out!

@xan1242 I also have a bitfunx sata adapter in a original network adapter (SCPH-10281) and a Toshiba MQ01ABF050 500GB HDD, with the same issues as you. I will get your version and do some tests and put the results here.
Also I think it's good to see how NHDDL does, since I don't have issues loading the configs, the neutrino.elf is recognized in APA, and the games in exFAT are correctly shown.

@ramonesfm The only change I've made is to fix the OPL config loading/saving onto the exFAT partition on the internal HDD.

This, in combination with pcm720's fix, should resolve most of the issues.

My build alone should only affect config loading.

So, until both of our changes get merged into upstream, you'll have to wait (or ask someone to build it for you, which I can do as well).