Concept 'magic': Think about implementing a OSXFS strategy architecture for 2 way sync

[EugenMayer]

Mentioned here https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/130?u=eugenmayer

Using OSXFS to sync from host to sync container "tmp folder" and then sync from this tmp folder to sync-container app-folder "locally".

This has some pros:

• fswatch can be removed on host side, saving ressources and "sync delays" (still we need fswatch on the sync container though )
• fsevents of the host are bubbling to the VM ( no need for custom implementations, OSXFS implements that )
• it would be probably easier to implement 2 way sync this way

Cons:
a) still needs 2 fswatch processes : One the the sync container to watch the changes on the tmp folder, one on the app-mount folder in the sync container to get app-container changes on the share
b) docker for mac only and probably OSX only

Comments on the cons:
a) well, thats just the same number as we would have without OSXFS, so its rather on par, but, fswatch is more effective on linux due to inotify evens, so it might gain performance, sind we reduce the fswatch on OSX, shifting it into linux
b) Well we do support OSX with docker-sync yet only ( i guess, not tested on windows ). Secondly, since it would end up being a strategy only, we could say that some strategies are not available for some docker-types like docker-machine and stuff. So thats sovleable

---

the data flow would look like this
Host --- OSXFS ---> sync-container /tmp[fswatch-trigger]--- local unison sync ---> sync-container /app/code[fswatch-trigger] ----- named volume mount -> app container

[mickaelperrin]

Which tool do you intend to use to to the sync ? rsync ?

I tried this approach not with OSXFS but with NFS in the dinghy project, see codekitchen/dinghy#197. The main issue I got was the lack of support of delete events in the project fsevents_to_vm, using OSXFS mount would indeed help.

[EugenMayer]

I would say, this concept is to achieve true transperency for 2 way sync for now - it yet not really change too much for rsync and, since it is docker for mac only, i would not like to switch the current strategies to it, rather add this on as an extra.

I therefor tend to use unison as the local sync. So wie have

Host --- OSXFS ---> sync-container /tmp[fswatch-trigger]--- local unison sync ---> sync-container /app/code[fswatch-trigger] ----- named volume mount -> app container

[EugenMayer]

After @mickaelperrin implemented unison-dualside we talked to let it "walk" for a while and then try this concept.

A correction, OSXFS shall be replaced by "any native sync" and the limitation on docker-for-mac is also invalid, iw would work with all of those

[mickaelperrin]

In fact, this concept only works with filemounts in docker that supports file events forwarding from the host to the mounted folder in the container. OSXFS supports this, but others solutions like Boot2docker don't, even dinghy has only a partial support with fsevents_to_vm (missing delete events).

[mickaelperrin]

I switched this week-end from dinghy to Docker for mac.

Despite the hard work done on unison:dualside I tested a different approach as you spotted here and as I did previously with lsyncd.

I tested several approaches (fswatch / unison, unison only...). Currently, the setup that performs the better is also the simpler with a pure unison implementation, without fswatch. So far, this is the only solution, that supports (and with very little CPU usage) a full Drupal bootstrap from composer in real-time. However, I encountered some missing delete events from the host to the container. I will investigate this further.

I released my experiments in a separate project docker magic sync that relies on a simple additional container.

Feel free to give your impressions and how it could be used with/in docker-sync.

This additional work with fswatch and unison make me understand how the implementation of unison:dualside could be simplified a lot, also.

[EugenMayer]

Well what shall i say, thats the concept outlined above, so there is nothing to comment more from me. As stated on the project page, i asked people to not re-do projects again and again, i guess, thats what we have this now.

The auto-discovery of mounts is a point is a clever thing, but probably can be done differently.

To use this in docker-sync i would need to understand:

• what do you start on the host? ( nothing i guess, due to osxfs usage )
• you do start one container and map all syncs in there. Does this work for several sync endpoints?

It is pretty easy to use with docker-sync:

• create a new strategy, lets call it magic.
• start your container there
• let the people configure compose the way you outlined
• as watcher strategy set 'disabled'

And thats it.

What is your plan in general, you want to keep docker-magic-sync as a new project or do you want to incorporate the changes / concept into docker-sync?

[mickaelperrin]

To be clear, I think some people will be interested from this "only docker needed" approach, so it could be maintained as a project.

But I would like docker-sync to help it to be even more 'magic' by reducing even more the configuration setup and making it simpler to use for docker newcomers. So yes, I would like to see it implemented in docker-sync as long as it is more simpler to use / configure than what I have done so far.

As you may have noticed, I don't use external volumes. This is something I found useful as there is no additionnal step to clean up docker when removing / restarting a container. However, currently I don't see how to implement this with docker-sync. Maybe docker-sync-stack could inject the container in the compose file dynamically ? Should it be done by hacking the compose file or is there a better way with docker commands ?

To answer your questions:

• yes, there is nothing to be run on the host
• yes it works with any sync endpoints as soon as they are not mapped to the same path even if they are in different containers.

Before starting a ruby implementation, I will test it for a few days to see if the core concepts are working very well or at least with limited drawbacks.

[EugenMayer]

Simplifying docker-sync configuration: full opt in. Not only for magic, in general. But you know what that means, we will try to guess / automate assumptions leading to new issues when we are "wrong". If you also opt in helping with those, lets do it. Because auto-configration is mostly effort for the docker-sync developers and effort in maintainance.

Great, ok so battle plan:

1. With "magic" we do not need external volumes by design, with the default strategys we need those. So we need to make those optional or strategy aware to so this is not needed. This should be already the case - external volumes are not used by docker-sync in general, its just a docker-compose thing and we document it. If we work out magic to be the new default, we swap the examples to be simple ( magic ) and add documentation for the others.
2. docker sync stack:
   docker-compose can ready the compose file into yml, and the write it ( talked with the developer about the write, should be easy. So we could read the yml file, modify it and then add it.
   But i do not really see a reason for that - we could just have this .developer.yml coping with that "non automatically" or at least start with that, add more automation later
   So working together with docker-compose.rb we can get this done "automatically" but i would suggest to start with .development.yml first, then add automatiation.

Does that sound reasonable?

[EugenMayer]

@mickaelperrin probably join https://gitter.im/EugenMayer/docker-sync?utm_source=share-link&utm_medium=link&utm_campaign=share-link to further discuss it

[mickaelperrin]

Mmm, bad news here. It looks like there is an issue with OSXFS. As soon as the files increases, inotify events are no more triggered in the container.

Some other users reports a similar issue on Docker forums. I will investigate this further and try to do a proper issue on Docker forums.

:(

[EugenMayer]

Shall we delay this implementation until we know if it is "fixable" or shall we go forward with this one? Architectural / Conceptual wise things are set, i do not expect any big hurdles

[mickaelperrin]

On my side I won't work on the implementation of this in Docker-sync until the core concept works.

Currently, I will see if the knowledge gained while working on this could benefit to the dualside implementation.

[mickaelperrin]

Some updates here.

I created the issue on the Docker forums, hope we will have some update soon.

I did also some new tests and what I can say it that as exepected the sync with magic is far slower and heavy in CPU than with unox. I think this is not a bid deal when doing simple file editing but when bootstrapping a big project you can feel the difference.

[mickaelperrin]

Just did some tests with the latest release of Docker with magic-sync.

Things got improved, basic sync is performed.

But the composer install test (run on the host and on the container) doesn't perform well, only a few part of files got synced.

[EugenMayer]

ending this discussion for now, should i? I would stall this ticket and remove it if we move on, or do you have other ideas @mickaelperrin

[mickaelperrin]

You can close it I think this strategy will work when OSXFS will improve performance and so... When we won't need syncing :)