Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance is Slow with Docker + Wine; Migrate to wsl2 for converter #20

Open
jmdelahanty opened this issue Jun 8, 2022 · 11 comments
Open

Performance is Slow with Docker + Wine; Migrate to wsl2 for converter #20

jmdelahanty opened this issue Jun 8, 2022 · 11 comments

Comments

@jmdelahanty
Copy link
Contributor

According to a chat I had online in the zarr developer gitter (need to find that conversation in particular) things might slow down due to the use of glob to determine how many tiffs are present in the output directory. They suggested using os.scandir instead. Another friend I was seeking advice from mentioned a similar piece of advice as well for finding the source of slowdowns. It's worth a shot to try this out and, even if it doesn't work, it's probably better to use this anyways for this use case.
@bjnielsen
Copy link

This is because of stat() which is often the cause of the I/O wait I was showing your a while back. stat() pulls in all the metadata, which you often don't need, like mtime, ctime, atime, etc. In fact we turn off atime on the fileserver since it's always different and quickly can cause an I/O bottleneck since it never stays in cache (always new) and it's never really used.

$ man 2 stat # shows you the structure of all information in an inode. All this is collected with any stat call.
$ stat # will show you all the stat info for any given file, directory, etc.

Noting that directories are just files with tables of what's in them. The tables contain the list of filename/inodes. An inode with a table of inodes. Including a parent inode of ".." This is why a "mv" in UNIX is virtually instantaneous since you're only updating the tables in two inodes. No data actually moves.

Take home session; with any code, avoid the stat unless you need it. If you only need filenames, all your doing is reading a table in a series of inodes.

@jmdelahanty
Copy link
Contributor Author

jmdelahanty commented Jun 13, 2022 •
edited
Loading

Thank you for this tip Bryan! I had no idea that's how these things work and that it was pulling all that information whenever you called stat. Also very cool to learn that's why mv is so fast! I had known it was related to just reassigning some kind of metadata, but I didn't know it had to do with the inodes.

In this case we don't even need the filenames! We literally just need how many files exist!

@bjnielsen
Copy link

bjnielsen commented Jun 13, 2022 •
edited
Loading

No problem. Here's a real-world test I just did to demonstrate. Create a 1000 files, and iteration of just filenames is 68.6 times faster than iterating both filenames and stats for each of the 1000 files.

Let us know if you're seeing performance issues in general, we may have some insight into why and help out.

$ mkdir tmp1; cd tmp1; time for i in $(seq -w 1 1000); do touch $i; done; cd ..
real    0m6.529s
user    0m1.416s
sys     0m1.384s
$ time find ./tmp1/. -maxdepth 1 -type f -exec ls -l {} \; > /dev/null # force a stat on every file
real    0m2.264s
user    0m1.194s
sys     0m0.658s
# rm and recreate to avoid caching
$ rm -rf tmp1
$ mkdir tmp1; cd tmp1; time for i in $(seq -w 1 1000); do touch $i; done; cd ..
real    0m6.346s
user    0m1.454s
sys     0m1.352s
$ time find ./tmp1/. -type f -print > /dev/null 2>&1 # no stat, just path/filename
real    0m0.033s
user    0m0.000s
sys     0m0.008s

@bjnielsen
Copy link

In this case we don't even need the filenames! We literally just need how many files exist!

Do you need a file count? Or just if file(s) exist or not.

@jmdelahanty jmdelahanty mentioned this issue Jun 13, 2022
Closed
@jmdelahanty
Copy link
Contributor Author

We just need the file count in the directory. The converter is killed only when the specified number of images has been collected or if a very long timeout period is hit. As far as I'm aware, the converter doesn't output anything if it finishes successfully or crashes, so doing something like subprocess.communicate for collecting its outputs doesn't help us here.

So what happens instead is the file system is polled every 10 seconds to see if the number of found images matches the number of expected images from the recording. See here for the conditional it's looking for.

@jmdelahanty
Copy link
Contributor Author

jmdelahanty commented Jun 15, 2022 •
edited
Loading

Here's some very rough/not standard kinds of benchmarking with different use of the ripper. There's a summary at the bottom.

Note:
All "Total to tiff time" includes the amount of time it takes the ripper to convert the voltage recording binary into a csv file + transferring from server to compute node when applicable. Directory locations are noted as well as where to things are being written. This assumes that the raw binaries/metadata files have been written to the server/are located there for the REMOTE conversions. For LOCAL directories, the binaries are stored on the local machine's SSD.

For the Avg Tifs/sec metric, the number is likely a fair bit higher than what's written for the LOCAL ON NATIVE WINDOWS measures because, again, the time it takes to perform the conversion to csv happens first and is included in the total time. It would be somewhat higher for the the REMOTE TO ones. Memory profiling as well as disk write speeds during conversion have yet to be profiled for this. I'm also not sure how to actually do that without just staring at the screen (which is what I'm currently doing).

LOCAL ON NATIVE WINDOWS
Using: C:\Users\jdelahanty\Desktop\20211105_CSE020_plane1_-587.325_raw-013
Writing to: C:\Users\jdelahanty\Desktop\local_out
Ripper: 5.5.64.500

AMD Ryzen 9

Copying FROM server TO local machine special-k.snl.ad.salk.edu:
22 min

Transfer topped out close to 100MB/s most of the time.

Total Tiffs: 45510

Total Conversion Time: ~ 6 min (!)

Avg Tifs/sec: ~ 126/s (!)

Total to tiff time: ~ 28 min

Writing to disk this way looks like it has a pretty sustained rate of creation of 323MB/s during processing and it's memory usage remains at just under 0.5GB throughout the whole conversion....

If there's a way to speed up transfers to the windows machine upstairs that would be very cool. A better solution would be if there was a dedicated windows machine that's linked up over some fast ethernet to move things to local SSDs, but as far as I'm aware, one is not available in the cluster at the moment. The other thing would be to have a script invoke subprocessing on that windows machine so conversions can happen in parallel. I don't know how to tell a specific Windows machine to do stuff from a script, though...

REMOTE TO NATIVE WINDOWS
Using: X:\specialk\jeremy_testing\ripping_tests\20211105_CSE020_plane1_-587.325_raw-013
Ripper: ibid

AMD Ryzen 9

Total Tiffs: 45510

Total Conversion Time: ~ 19 min

Avg Tifs/sec: ~ 40

Total to tiff time: ~ 19 min

Reading over the 1Gbps line slows conversions down quite a bit, but isn't that much shorter than the Windows local total amount of time due to the amount of time it took to transfer to Austin's windows upstairs.

This uses approximately the same amount of memory (just under .5GB) consistently but is writing to disk at a consistently slow rate of about 22MB/s! It's transferring data from the network at a rate of near .5Gbps, so half the bandwidth of the available 1Gbps line...

REMOTE TO COMPUTE NODE SCRATCH, DOCKER WINE
Using: X:\specialk\jeremy_testing\ripping_tests\20211105_CSE020_plane1_-587.325_raw-013
Ripper: ibid

Total Tiffs: 45510

Total Conversion Time: ~ 28 min

Avg Tifs/sec: ~ 27

Total to tiff time: ~ 28 min

This takes slightly longer maybe due to the fact that it's reading data into the machine over LAN before processing it, but is much more likely to be related to Docker + Wine doing things slowly as Annie has mentioned.

Given the slow speed of writing tiffs to disk over the network shown in the REMOTE TO NATIVE WINDOWS section, it seems like this is also a candidate for things being slow, but the fact that using the local scratch space for reading/writing data is similarly slow overall, Docker + Wine are the more likely culprits it seems...

LOCAL COMPUTE NODE SCRATCH TO LOCAL SCRATCH, DOCKER WINE
Location on cheetos.snl.salk.edu

cp /snlkt/data/specialk/jeremy_testing/ripping_tests/20211105_CSE020_plane1_-587.325_raw-013 /scratch/snlkt2p_format_testing/
1m17.496s

Using: /scratch/snlkt2p_format_testing/20211105_CSE020_plane1_-587.325_raw-013
Ripper: ibid

Total Tiffs: 45510

Total Conversion Time: ~ 23 min

Avg Tifs/sec: ~ 33

As expected, moving data to/from nodes and storage in the datacenter is very fast. However, there was only a slight speedup by having data available on the machine's SSD (disappointingly).

While staring at the memory usage through docker stats, it has the same memory profile as the native windows machine while reading data basically, under .5GB most of the time

CONTAINER ID        NAME                                                 CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
848d3eb9a764        scratch-plane1-snlkt2p_format_testing-SNLKT-ripper   102.46%             377.9MiB / 10GiB    3.69%               0B / 0B             147kB / 22.8GB      61

However, the disk read/write speeds are very slow.

While doing this:

$ iostat /dev/sdd -y 3

I see this at the very beginning once tiffs have started being generated:

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdd              79.67         0.00    32018.33          0      64960

Over time this slowly decreases down further and further until it looks like it hits a floor of about 10000 kB_wrtn/s.

How to actually tell if this is a hard drive specific thing vs actually something going on in the code is something I don't think we can do that well since we have no idea what the converter is actually doing... The only thing that remains to be tested is the subject of this issue really.

I feel like most likely it's a Wine thing as Annie said, but it looks like at the very beginning of conversion it starts out doing at least okay. Why would it slow down so badly the longer it's been running? And why doesn't it ever reach the fast 300+MB/s generation of tiffs as on a windows machine?

Not sure how to properly monitor file IO/writing on linux still, but I output iostat by piping it to a txt file. Not sure how helpful it is...

local_scratch_iostat.txt

Summary:

As we've known, performing the ripping on a windows machine with data locally stored on the computer's SSD performs much much faster. I'm not aware of a dedicated windows machine that we can perform this procedure on through shell scripting as Annie has suggested doing which would be really cool. Another thing is that sending the raw binaries over the network and then performing the conversion upon incurs a performance penalty, which is compounded by the slowdowns experienced by the docker container (probably because Wine isn't managing the conversions to unix calls very well for this case, or for the other reasons mentioned above/performance penalties due to virtualizaiton through docker). Making copies of files from the server to local scratch spaces in the data center is crazy fast (just over 1 min for 75GB of stuff!) especially when compared to Austin's special-k computer (as expected since it's on a slower connection), but the unnecessary copying of data, even temporarily, isn't ideal and I would imagine it's something we'd like to avoid.

There's definitely better ways of doing benchmarking, and there are logs available for the Docker implementation about tif generation, but I'm not sure how to properly log things like memory usage, disk write speeds, and cpu usage to files yet. No real idea about how to log them on Linux properly...

@bjnielsen
Copy link

bjnielsen commented Jun 16, 2022 via email

@jmdelahanty
Copy link
Contributor Author

Sent along an email a little bit ago that has that kind of information! Can repost here if we'd like.

You can boot a linux kernel in windows, use wsl v2. This is pure
linux, as opposed to an emulator like wine.

I remember now that you mentioned this! I completely forgot about it. Do we have any machines that are running wsl v2 that we can try running the converter on?

@bjnielsen
Copy link

bjnielsen commented Jun 16, 2022 via email

@jmdelahanty
Copy link
Contributor Author

The machine that I'm working on at my desk is called busdriver (so busdriver.snl.ad.salk.edu). On the cluster I've been using Cheetos for the docker version.

So to be clear here, the converter (you can find the executables here) was written for Windows. If I'm understanding your suggestion correctly, the idea would be to have a Windows machine with wsl2 installed in the cluster that can run this converter. That machine can be told to execute the converter from a different linux machine running a shell script or something.

If we ran it this way, we would avoid the use of Docker/Wine altogether and get the super quick speed performances of a native windows machine with the flexibility of Linux.

@jmdelahanty jmdelahanty changed the title Use os.scandir instead of path.glob for polling directory during tiff creation Performance is Slow with Docker + Wine; Migrate to wsl2 for converter Jun 16, 2022
@jmdelahanty
Copy link
Contributor Author

One thing that was discussed at our 2P meeting yesterday was to just have a Windows machine next to the Bruker computer that does nothing but ripping, which will be nice and fast, and then runs the conversion to H5 before it's sent to the server. Writing tiffs to H5 using this takes 2-3 minutes, but there are other implementations we can use too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bjnielsen @jmdelahanty