Pre-tokenization step outdated

[NahuelCostaCortez]

Dear ETHOS Team,

First, congratulations on the project, it’s truly interesting work, and thank you for making it available to the community.

Following the README instructions, I’m trying to convert MIMIC-IV to the MEDS format. However, I’m unsure which instructions I should follow: those in the root folder’s README or those in scripts/meds/mimic/README.

In both cases, running either run_mimic.sh (from the root instructions) or run.sh (from the scripts folder) results in an error originating from the pre_MEDS.py script. The issue seems to come from these import lines:

from MEDS_transforms.extract.utils import get_supported_fp
from MEDS_transforms.utils import get_shard_prefix, write_lazyframe 

These functions no longer appear to exist in the current MEDS_transforms package. It looks like they were part of earlier releases.

Given this, I’m not sure what the intended workflow is for converting the data to the MEDS format with the current version of the code. Could you clarify which steps or scripts should be used?

Additionally, I wanted to ask why the pipeline does not rely on the official conversion pipeline at:
https://github.com/Medical-Event-Data-Standard/MIMIC_IV_MEDS
What are the differences between using the MEDS format produced by your pipeline versus this one?

Any guidance would be greatly appreciated. I’m looking forward to reproducing your work.

Thank you in advance, and happy holidays!

Nahuel

[prenc]

Hi Nahuel, thank you very much for your interest in ETHOS and for the kind words about our work. We appreciate it!

I’m unsure which instructions I should follow: those in the root folder’s README or those in scripts/meds/mimic/README.

To clarify, the intended workflow is the one described in the main README at the root of the repository. The scripts under scripts/meds/mimic/ are implementation details and may be less stable or less up to date as standalone entry points.
For reference, I personally downloaded the repository, installed the deps with uv pip install -e . and ran the preprocessing successfully using the script from the scripts/meds folder with the following command:

N_WORKERS=7 bash run_mimic.sh ../../data/mimic-2.2 ../../data/mimic-2.2-premeds ../../data/mimic-2.2-meds-ed ed

These functions no longer appear to exist in the current MEDS_transforms package. It looks like they were part of earlier releases.

You are absolutely right. The preprocessing pipeline is confirmed to work only when using the package versions specified in pyproject.toml. In particular, MEDS_transforms is pinned to version 0.1.1, which still includes the referenced utilities. Installing newer versions will result in the import errors you observed. Please make sure to follow the dependency installation steps exactly as described in the main README.

I’m not sure what the intended workflow is for converting the data to the MEDS format with the current version of the code.

With the dependencies installed according to pyproject.toml, the pipeline should run end-to-end as documented.

Why the pipeline does not rely on the official conversion pipeline at:
https://github.com/Medical-Event-Data-Standard/MIMIC_IV_MEDS

This is mainly for historical reasons. Our preprocessing pipeline was developed before the official MEDS MIMIC-IV conversion pipeline was released, so we implemented our own conversion logic at the time. Conceptually, both pipelines aim to produce MEDS-compatible outputs, but the implementations differ due to independent development timelines.

That said, we are absolutely open to community contributions that would help align ETHOS more closely with the official MEDS pipeline. If this is something you would like to explore, I would be happy to accept your PR.

Thanks again for your thoughtful questions and for taking the time to dig into the code. We hope this clarifies things, and please feel free to follow up if anything remains unclear.

[NahuelCostaCortez]

Thanks a lot for the prompt and detailed reply, really appreciate it! I followed your advice, stuck to the versions pinned in pyproject.toml, and I was able to run the script successfully. I was getting that error because I had updated the MEDS_transforms package to the latest version.

I don't wanna be annoying but I wanted to ask you about something I’m running into now. When running the script with:

suffix=""

bash run_mimic.sh
"$MIMIC_IV_DIR"
"$MIMIC_IV_DIR/mimic-premeds"
"$MIMIC_IV_DIR/mimic-meds"
"$suffix"

it finishes without errors, but it only generates 14 parquet files (0–13) in the train folder and 2 files in the test folder. That seems much smaller than expected, so I suspect something is off in my setup.

For reference, MIMIC_IV_DIR points to a directory containing:

hosp/
icu/
CHANGELOG.txt
LICENSE.txt
SHA256SUMS.txt

that is, the files here. I’m not sure whether additional files or subdirectories are required for the pipeline to produce the full dataset. One thing worth mentioning is that the files are in plain .csv format rather than .csv.gz. Could this be an issue?

I assume the problem is probably on my end, but I wanted to hear your thoughts. If you have any guidance on this, I’d really appreciate it.

Thanks again for all your help!

[prenc]

Hi Nahuel, thanks for the update, glad to hear the dependency issue is resolved.

Regarding the output size:

it only generates 14 parquet files (0–13) in the train folder and 2 files in the test folder

This is exactly the same number of files I obtain on my side, so this output is expected. Could you clarify why you were expecting a larger number of files?

Also, using .csv instead of .csv.gz is not an issue, both formats are supported.

As a side note, when suffix is left empty, the pipeline generates the core MIMIC-IV MEDS dataset only. If you instead set:

suffix="ed"

the pipeline assumes the presence of the MIMIC-IV-ED extension from PhysioNet in MIMIC_IV_DIR (ed subfolder). This extended dataset is the one used in the paper and while it results in a larger dataset, the same number of output files remains the same.

Let me know if you have any questions.