Project Continuation

[fynnfluegge]

Hello,

I want to ask what is the roadmap for continuation of the project.
I saw there is a maintainer needed. As I currently need as many websocket implemantations for various exchanges as possible, I am open for pushing the project forward.
Currently 12 exchange streams are implemented, but I guess meanwhile there may be some more exchanges providing a websocket API. Correct me if I'm wrong...

[davidjirovec]

It is sad to see so many pull requests, which are not merged. Maybe people would contribute even more, if they would know their pull requests will get merged. I understand @dozd might not have time for this project, but this project should go on. What is the best solution? Is there anybody to maintain fork of this project?

[Flemingjp]

I don't have the time to maintain a whole fork, but would invest some time in contribution. It's difficult to invest any time when I know this repository is becoming stagnant.

[fynnfluegge]

I may be willing to maintain a fork, but I don't think that I if can do this alone since not that much time...

[Flemingjp]

I also think it will help this repository by reviewing the current PRs. @dozd explained that they would be happy to merge from then on.

[fynnfluegge]

Yeah the primary goal should review all PRs and merge as many as possible, I think it would help if @dozd would give some chosen people write access and finally synch the develop branch with all forks.
For example, this https://github.com/KapitalTrading/xchange-stream fork is far ahead, it's sad to see how the develop branch is out of synch :(

Another point, I am considering to start another project that provides Java Websocket streams for cryptocompare API, anyone interested in this? I think it's good to have various streaming APIs

[Flemingjp]

If we can pull through as many PR as possible, and then look towards releasing a new version of XChange-Streams to be inline with the, quite recent, version 4.3.11 of XChange, I think that'd be a good start.

[badgerwithagun]

I already maintain a separate development branch for my own purposes, which is develop + all my PRs + some other selected PRs that I needed.

The divergence from develop is rapidly getting impossible to manage though. I have no wish to fork someone's project on a permanent basis. I'd much rather see the main project succeed.

For example, #185, #191 and now #225 are all dealing with the same set of problems. I've managed to combine #185 and #191 in a way that works, but all it will take is for someone to merge one and not the other, or combine them in a different way, and I will be in a world of pain.

I build against the latest XChange that I can (which is much more recent than the one in develop), but actually force it at runtime to 4.3.11, which breaks BitMex (which I don't use). It's all a bit of a mess.

I'm happy to help, but the problem is cyclic. I have tried to review PRs, but without the ability to merge them, I can't make simple fixes to their branch pre-merge, leaving me in a position where I have to push back tiny change requests at the contributor, who doesn't bother or is too jaded from not having their PR picked up for months to care.

At the same time, I can't really properly review anything complex because I can't merge complex changes into my own code to test them without heavy conflicts (which kind of defeats the point) and I can't run my code against vanilla develop because it just crashes.

We need at least one person with merge access helping to push these things through for a while.

[badgerwithagun]

Read #160 for a frustrating example.

[fynnfluegge]

I understand, I saw your branch latestandgreatest @badgerwithagun, I'd rather build your fork and use it but I agree with you, that's not the long-term solution. I don't get why nobody else get merge permissions, I mean, the project is on the best way to get burried if we don't merge some PRs asap and sync the develop branch. That's sad for this nice project...

[fynnfluegge]

However, seems like we must deal with a bunch of spreaded forks, after reviewing some current forks I think it's impossible to get all synch

[badgerwithagun]

If we can just get to one authoritative repo with the important PRs cleared, a fresh release and master and develop synced, I think everyone will be a lot more motivated to return to the mainline and re-contribute the things they are missing. I certainly would.

[fynnfluegge]

@badgerwithagun one question, why you force XChange 4.3.11 at runtime or what you mean by that? Does it not work to set the property <xchange.version> to the specific version?

[badgerwithagun]

If I do that the xchange-bitmex project doesn't compile. There's a small change in the XChange BitMex implementation between 4.3.9 and 4.3.10 IIRC which is incompatible.

So I just set a requirement for 4.3.11 on my own code which overrides the version in xchange-stream. As long as I don't use the BitMex implementation it works OK.

[fynnfluegge]

Ah okay, this is a tiny fix.
@dozd we need your opinion please, in order to go on. At least we need a small bunch of selected persons who have permissions to merge PRs to merge the most critical PRs asap.

[fynnfluegge]

@badgerwithagun as long as this stagnates here, am I invited to use and contribute to your fork?

[badgerwithagun]

@oreonengine - I hope you understand why I am a bit nervous about that. I don't want to fork @dozd's project at this time. Feel free to use it, but don't submit PRs on there. Let's keep the project focused on this repo. Submit them on here and @mention me. I'll merge them into my fork if they look good.

Bear in mind that the latestandgreatest branch is not just everything that works, it's also a bunch of things I've patched in for my own purposes which don't all necessarily quite work yet, and they were often cherry-picked in rather than properly merged. If forking properly, I'd want to start with develop and have nice,. clean, ordered merge history.

[pchertalev]

So, what we gonna do? Waiting for @dozd opinion? For instance, I very need Bitfinex Authorized api and fix of netty disconnecting issue. Of course I have own fork with that fixes, but it smells bad to support own fork by myself only. Does anyone has @dozd contacts to ask review and merge permissions?

[badgerwithagun]

It's only fair to give him some time. He's not around much, having already said he doesn't have the time for the project.

[badgerwithagun]

@oreonengine - I've got scared about divergence of my code, so I've started building a clean develop fork which merges all the essential PRs in the way (and order) in which I've tested that they work. IMO, it should make a good basis for a fast-forward pull to develop as a quick way of getting up to somewhere it can be synced to release. At least it feels like I'm doing something useful. Will put up a link when done.

[badgerwithagun]

OK, take a look at https://github.com/badgerwithagun/xchange-stream/tree/develop-fork

That is what I am running in production, minus the stuff I don't think is ready yet, and with everything (relatively) cleanly merged over develop with the PRs clearly referenced. It's not perfect by any means, but it does work, at least for the exchanges I use.

@dozd: my proposal is:

• Do a fast-forward to pull this fork into develop, That can close off several big, scary PRs in one go.
• Sync that to release and do an actual 4.3.9 release. Then at least there's something in Maven Central that doesn't fall over completely.
• Give a few more people merge permissions so we can start wading through the rest of the PRs from a stable position.

In the meantime, @oreonengine - feel free to use the fork.

@pchertalev - should work for you too.

[fynnfluegge]

@badgerwithagun alright thanks for the information, I take a look into develp branch of your fork. I was thinking about using the develop fork of this main repo, but was scared about merging all critical or trivial PRs while some forks already did this.

[badgerwithagun]

@oreonengine it's the develop-fork branch on my repo. I keep develop in line with this one.

[Flemingjp]

@badgerwithagun what exchanges do you use? It maybe useful to get maintainers across all exchanges, depending on the demand

[badgerwithagun]

@Flemingjp: Binance, BitFinex, Coinbase Pro, a little bit of Cryptopia and Bittrex.

[Flemingjp]

@badgerwithagun OK, I'm mostly focused on Binance, and there's maybe some scope for HitBTC in the future.

[pchertalev]

So dear developer guys,
@dozd does not have enough time for getting focus on bitrich and not one pool request will be accepted and merged soon. I propose to move all pool requests to Graham's (@badgerwithagun) fork. One more It will be great to update project root page with information about live fork. But I'm afraid only @dozd can do this.

[badgerwithagun]

@Flemingjp appears to have merge access and is working on getting the key PRs merged (e.g. #191 is in progress). I think we can get this all moving for now without moving development elsewhere.

[Flemingjp]

Unfortunately I don't have merge access. I've been trying to organise and review code - from here I've emailed @dodz to push through reviewed PRs.

I would apply to become a maintainer. But I can't guarantee consistent dedication and time - maybe I could attempt to do so for a short while to rejuvenate this repo.

[badgerwithagun]

Likewise. Happy to help get it back on track.

Could you poke him to merge #160? I can't approve it because he requested changes and hasn't approved the result.

[Flemingjp]

@badgerwithagun and all - I now have write access to the repo. My plan is to commit time temporarily to move the project forward with the plan of producing a new release. Due to limited time I have I will probably not be producing PRs myself - mainly reviewing and trying to coordinate resources on open issues.

The best help would be for people to review others PRs. If I see a PR with a (well written) review, I shall review it myself and merge into the branch.

Any PRs I produce, I would appreciate if someone else would review it for me to ensure the standard of code produced remains high.

I'll expose my email address if anyone needs to contact me.

[badgerwithagun]

Thanks @Flemingjp. I should be able to start reviewing and testing PRs more as soon as develop has caught up a bit. I think #191 is the only one really causing me trouble.

[Flemingjp]

It would be good now to create a check list of items that need to be done before we can consider releasing a new release. My personal opinion would be, in order:

1. Continue to work on and merge Fix gdax disconnect NPE/Bitmex execution reports/bug-fix rollup #191
2. Merge any existing bug-fix related PR that is appropriate
3. Push forward with PRs with known bugs if time permits
4. Clean up open issues that may now be irrelevant due to many overlapping commits

In such case adding any new feature or exchange should be put on the back burner. But that's just my two-cents, so any opinions would be great to hear.

[badgerwithagun]

That's pretty much in line with my thinking.

We should definitely add syncing with XChange 4.3.11, which just needs a Bitmex fix. No PR open for that as yet.

On which note, a sweep of the Issues for reported bugs with no existing fix PR would be a good idea. Mark them up as "needs replication" perhaps?

[dozd]

Hello guys, I'm ready to give other person write access (@Flemingjp got it) to this repo so you can still maintain it further.

I'm also able to release versions when you say so 👍 .

[dozd]

Actually a release of new version should be the priority now. So if anyone else want to contribute, gimme a note. After all major PR are merged I'm happy to upload new version into maven central. I will have some time in following two weeks, so if needed, gimme a ping (email works best ;) ).

If anyone can upgrade the underlying version of Xchange to the latest released, it would be great before the release of xchange-stream.

[fynnfluegge]

Once develop is synch with XChange 4.3.11 I can create Bitmex fix PR

[dozd]

1a645e8 wasn't hard

[Flemingjp]

Outlined the current take of the roadmap to 4.3.11 release.

[dozd]

Cool, assign the PRs into mileston 4.3.11

[Flemingjp]

As of now, the remaining PRs rely on BitMex. In really need of some developers to test these PRs. #234

[Flemingjp]

Due to the slow progress on BitMex and lack of developers using that exchange, that we should push back on BitMex PRs. This would effectively bring the repository into a state close to releasing 4.3.11 (give or take a few bug fixes) given that #191 has been merged.

[badgerwithagun]

There are always bugs! Is there any harm in releasing 4.3.11 now? Xchange 4.3.12 will be along in a matter of weeks.

[Flemingjp]

That's true. @dodz can you set up the next release?

[dozd]

Sure.

[dozd]

4.3.11 released & uploaded into central. Will be available soon.

[Flemingjp]

Currently a lot of open issues are over a year old, and are to do with the dependency on an old version of XChange. I have labelled these as here - with the intention of going through these issues to see if they are still relevant.

There are also several open bug issues, which also falls into the same category.

[Flemingjp]

@fynnfluegge Are you using BitMex?

[fynnfluegge]

@Flemingjp No don't use it, I tried to fix the Bitmex issues though but didn't find a solution...

[wvr]

Observation from my side on the topic. The bitmex implementation from https://github.com/KapitalTrading/xchange-stream works fine for me, so might be a good idea to just copy that across and close all related/old PRs? I'll be using it in prod in the coming weeks so will log any (new) bugs I find here.

[badgerwithagun]

If I remember correctly. @wvr, that's the content of PR #244.

[wvr]

Thanks @badgerwithagun

@jkolobok , @Foat - I see you guys have made most of those additions - ok if I merge all that into [develop] and create a new PR for it? Should be able to assign sufficient time to testing it this week.

Doing this should close at least #163 #183 and make #244 unnecessary (can close).

[badgerwithagun]

Closing - we managed to keep things moving, and the plan is now to clean up and merge with XChange.