Forum rss-feed

Forum

Developers: EigenD 2.0 branch

STICKY

written by: john

Hi

We've now had time to have a decent discussion about how we're going to solve the problem of Agents in the source tree that contain non GPL code. To reiterate, the principal problem is that setups built to use an Agent that we may decide to do a non GPL version of for a while will not load in a system built from the GPL'd code as those Agents will simply not be there. We solved this issue by just pulling the current working branch which is 2.0. There are, as many of you have pointed out, a great number of disadvantages to not having the ability to work on the current code and this is a very crude fix. We don't like it either.

Instead, we've decided to simply remove Agents from the GPL source tree that we want to release non GPL. In order that setups built in the paid for release (that may well use the non GPL'd Agent) can still run in an EigenD that you might build we will be giving instructions as to how all the bits of non GPL Agents can be copied from your paid for binaries into the GPL'd system so that one can run it against your normal setups. This copying process is going to be clumsy (the files that need to be copied or linked to will vary from Agent to Agent and it won't be wholly consistent) to start with, but as I've said before we'll be working on improving this over time - our ambition is that Agents can be easily moved about in the future a well as traded and swapped. It's going to be a
'developers only, not supported' kind of thing for the near future, but Jim reckons it will work fine.

I think this will provide plenty of flexibility in regards to licensing, whilst still making it straightforward to develop and look at EigenD code. It's a little weird as sometimes an Agent might disappear for a while from the source tree if we want to add some groovy new feature that we think is part of the value that paying customers get, but in the absence of some fancy Agent versioning system, which we aren't going to get to implement this side of 2.something, I think it'll do fine. We'll try to announce any Agent removals a way in advance so that anyone who might be coding with one gets to know about it and can discuss it with us beforehand. We'll try not to do it too often.

To give you an idea of why we got in such a pickle over this, the Agent in question for 2.0 is the audio agent. It's getting multiple configurable outputs and inputs in 2.0 and this is one of the nice new features for base level subscribers. This Agent will certainly be dropping fully back into the GPL release, probably for 2.1 or 2.2, but its a great example of an Agent that is important in the system getting a major upgrade that we don't want in the GPL release for a little bit. It's an Agent that is kind of hard to live without in any setup, and one we didn't want to fork for reasons of cruft - we really don't need two kinds of audio in/out in the long run.

So 2.0 will be reappearing on github as soon as we get the necessary rearrangement done.

John

BTW, in terms of an open platform (to NothanUmber), I don't think there is any need to change the EigenD licensing - GPLv3 has excellent protections against Tivoisation, a thing I loathe, and those protections are personally important to me. A binary API coupled with the ability to look at the vast majority of the system source code via the GPL and build a running system from scratch does enable anyone to make and sell Agents whilst still protecting our interests in the broader system - I think that scenario contains the best of all worlds. After two years of considerable change we are now heading towards being able to stabilise and document our Agent interface nicely, and I'd like to see this emerge as part of our collective thinking over the next few months.

written by: NothanUmber

Sun, 20 Nov 2011 22:52:22 +0000 GMT

The EigenD 2 branch seams to be gone from github on first sight, only 1.4 is accessible atm. Does that mean a change in policy or is it just a temporary thing?

*edit* Just had a look, seems to have happened 4 days ago:


jschpmn deleted branch master at jschpmn/EigenD 4 days ago
Deleted branch was at /jschpmn/EigenD/tree/master
jschpmn deleted branch master at Eigenlabs/EigenD 4 days ago
Deleted branch was at /Eigenlabs/EigenD/tree/master


Perhaps Jim wanted to delete the master branch from his fork and deleted the official one accidentally?

*edit2* the v2.0 branch seems still to be there, not called master anymore but "ea66164"...(?)

Greetings,
NothanUmber


written by: barnone

Sun, 20 Nov 2011 23:46:16 +0000 GMT

I noticed that a few days ago when I went to fetch.

-- snip --

I hope it is just a temporary thing. I was enjoying keeping up with the checkins. Was learning from that.


written by: john

Mon, 21 Nov 2011 07:57:47 +0000 GMT

The 2.0 branch contains a number of things that aren't released under the GPL and the technical difficulty of keeping one of those separate from the rest of the code was proving to be too much, so we've decided to remove it from github for now. We'll probably drop 2.0 into the public domain when 2.1 comes out or therabouts, and work like that in future. It's also possible that we'll be able to have github track the releases in some circumstances in the future - the principle difficulty is when we're introducing a major new feature into an existing Agent, not when we are adding an Agent, so if we haven't made significant modifications to something already there its a non issue. We have looked at versioning Agents to avoid this problem, but that is quite a challenging thing to get right and we simply don't have the time for it right now.

John


written by: NothanUmber

Mon, 21 Nov 2011 10:39:14 +0000 GMT

Hello John,

I can understand your reasons why you are doing this. However from my personal point of view it will be crucial for the future of EigenD/Eigenharps/Eigenlabs to establish EigenD as the emerging open platform for next generation controllers that run against the limits in the currently established VST/AU/MIDI land.
Eigenlabs together with the Eigenharp community might have a hard time keeping that milestone project alive, it's scope might be too big and ambiguous for what we can finance with a (perhaps too) small group of people in the long run.
And as I am not sure whether Eigenlabs can support that "beyond Eigenharp" scope I think it is utterly important to have the current platform (that is compatible with the latest Workbench version) open, so people can write extensions for this latest version.
The current open source version will most probably not take off as a side branch by itself - mainly because of the lack of a Workbench like graphical configuration tool in my opinion. (And trying to write the missing things ourselves is presumably not the way to go - it would most probably always lack behind what Eigenlabs accomplishes and more important it would further split the community - which is already small enough when everybody uses the same version...)
It would help a lot to have one latest and greatest version with an open base and commercial additions.
So - although I completely understand that you do not want to release sources for things that you want to sell first, please consider to rethink priorities and keep the open platform that is necessary to develop agents for the at a time current EigenD release up to date. I think selling Workbench and commercial agent licenses to people outside the Eigenharp community (and convincing some of them that the Eigenharp is the optimal instrument for EigenD by the way) will more than pay off for you. (I honestly believe that is the only chance to see this succeed, not knowing which "aces" you still have in your sleeves). Even if it means that you'd have to release one or two features in existing agents up front.

Kind regards,
Ferdinand


written by: john

Mon, 21 Nov 2011 14:12:05 +0000 GMT

Hi Ferdinand

I agree with pretty much everything you have to say there. The decision to keep 2.0 out of the github repo is purely technical at the moment, it's about how we can add new functionality to an existing Agent that we want to delay for a little while in the GPL release. If we had a nice way to version Agents it wouldn't be an issue, but we don't, and we must have the ability to do this, it is not something I'm prepared to lose - an open source strategy is not possible if we cannot stagger the release process so that people who pay money get some additional value beyond the GPL release, at least for a while.

This usually won't be a problem as I think that new things will often be in new Agents, but for 2.0 we have some significant new functionality in an existing Agent that I don't want to release under GPL for a little while. We could make the new version a new Agent with a new name, but we talked about it here and that seemed a bad choice for a number of reasons. In a world with infinite resources we'd have just version the Agents and this would never come up. We just don't have the time to do that well before 2.0 comes out.

You are quite correct that developers need to be able to build their own Agents against 2.0 from the start and not have to wait for it to be GPL. There wouldn't exactly be much point in a developers conference without that ability. Jim is looking at this right now and we should have some news on that front shortly, certainly before you would have wanted to be building against 2.0. We're as keen as you are to make life easy for you to develop against 2.0. This is part of the overall desire to make Agents more standalone - the need to build the whole of EigenD just to make an Agent is something we consider to beunsatisfactory and we'd like to make it possible to exchange (and possibly sell) Agents in the future, outside of the normal EigenD install.

We'll see if we can get back to you by the end of the week with some further news. BTW, if you have any neat ideas (we really tried to think of something clever but failed, but is more than possible we missed something obvious) that let us keep selected bits of the 2.0 branch on github without Agent versioning in the system, I'm sure Jim would love to hear them.

John


written by: barnone

Mon, 21 Nov 2011 17:03:44 +0000 GMT

Hi John,

Some ideas.

You could keep a 2.0 mainline but use an integration branch to do some major surgery. I know that in my day to day dev, everyone uses a private branch and checkins to mainline are always a merge from a development branch. It's also possible to have a broken mainline at times. Not ideal, but it's way better than no mainline at all. I guess we don't understand the agent versioning thing. This is a mismatch between the agent in the mainline GPL and the one in the release for Stage and Workbench?

Keeping Stage and Workbench out of the GPL should be enough to get people to purchase 2.0. Is it not possible to keep them separate?

You could keep the daily mainline private but push updates to a 2.0 open branch at every dot release. Not ideal but far better than the other option. Also, wouldn't that take care of the agent versioning issue?

Another option is to give developers who have paid for 2.0 access to the protected mainline. Github very easily allows for access rights to individual developer accounts. This would make sure that people are not freeloading off you while still giving us access to the latest platform. Maybe this is the easiest for you and the best for us.

I fear that if it's not at least that frequent at a minimum that I wouldn't really consider it a viable Open Source project anymore. Instead it would become a guarantee that we could fall back to 1.4 if Eigenlabs gave up the business. That is valuable but it would not motivate me to contribute to a platform that is a major revision back.

Having an API is interesting but it's not the same as Open Source and I fear that the complexity of this system lends itself better to be able to debug things by being able to look through the entire system. We've already experienced the difference between being inside at Eigenlabs and being outside trying to understand the system through an API - Belcanto(In the early days before Stage). An API has to be VERY well documented to be useful. That's part of the motivation for Open Source to begin with, it's let's us developers figure stuff out for ourselves. I fear a true API would be more work for you than sorting out the Open Source repo and dev model.

I know that workbench will make this easier but I think there is still huge value in really doing Open Source. You can't be half pregnant with this. It's a fragile thing and looking at old code is not motivational in the slightest.

I hope there is a way to work it out.

Best,
Chris


written by: john

Mon, 21 Nov 2011 18:49:16 +0000 GMT

Hi Chris (and Ferdinand)

Firstly I'd like to say that our desire is very strongly that the working branch on githib is actually the same as the branch that is released or is heading for release. None of us here are very enamored of having to pull 2.0 from github, it seems like a very crude fix for a problem that we had and it has a lot of downsides we don't like. We'd like to find a better way so that we could keep 2.0 (and future development) live with all of us including you working on the same main branch. We do however need the ability to add a non GPL feature to an Agent - it's just a thing that we want to be able to do and one we want to do right now in 2.0. I'm not going to debate the merits of this commercially I'm afraid, particularly since I think it probably has a nice technical fix somewhere or other that makes it irrelevant to you.

It all comes down to the issue of what happens when we want to take an existing Agent and add non GPL code to it, perhaps only non GPL for a while, but non GPL all the same. We have exactly that situation right now. Now, Agent by Agent we can choose, GPL and on github, or non GPL and here on our local server and this has almost zero impact on open source developers. So the easiest thing we can do is simply fork the Agent, call it something else and make a new Agent. There are problems with this - we would start to get a lot of cruft in naming and (this is particularly and hopefully uniquely the case for 2.0 where setups are not forward compatible) the new Agent may not be setup compatible with the old Agent, the old Agent may actually not work or even compile in the new version etc. It also makes it much harder to provdie an automatic upgrade script that enables older setups to be loaded (not an issue for 2.0, but it would be in the future).

All of these issues do tend to go away of Agents are able to be seperate, installable things (which was really the point I was making about being able to build Agents separately as well) - we can just pull an Agent out of the open source tree while it has non GPL code in and drop it back in when we want to release the code, and developers can just use the new version as binary in the systems they build themselves. Can't do that right this second though, not enough hours in the day to get that working and we really really want to get 2.0.0 Experimental out there this year. We do probably have a bit of a deadline though if you're all coming to see us in mid January - not much point in a devcon if one can't develop.

We were in a hit of flap last week when we took 2.0 out of github but we'll have a longer chat this week, probably on Wednesday or Thursday when Jim is about and try to make some kind of announcement as to what we're planning to do in the medium term - we're aware of the development issues but didn't think it was much of a concern until 2.0 was actually released.


John


written by: barnone

Mon, 21 Nov 2011 20:28:52 +0000 GMT

Thanks John.

That explains it well and all we wanted to hear.

Much Appreciated,
Chris


written by: NothanUmber

Mon, 21 Nov 2011 23:55:09 +0000 GMT

Hi everybody,

first of all, I don't mind if the EigenD 2 sources or an alternative is not available within the next weeks - imho it won't matter until somehow after the release of 2.0 experimental + Workbench essentially. (After the release we'll presumably nonetheless be busy exploring Workbench and stuff for the next weeks to come :) )
What is very important for me are the mid term plans though.

It will be very interesting with which solution you will come up with!

Had some thoughts about it at the way home, here the current results:

Perhaps it makes sense to cathegorize the parts of EigenD first:

1) Platform: This is everything that is necessary to load setups, thus connect and configure agents, all the belcanto stuff etc.
2) fundamental agents: things that are necessary to get sound out of a "synthesizer agent": kgroup, scaler plus the input and output agents to communicate with the environment: MIDI in+out, OSC stuff etc.
3) Eigenharp specific agents: alpha manager + keyboard, etc.
4) "generators": typical "plugin like" agents like synthesizer components, plugin subhosts etc.

Now you could define what should be available in which form:
* Everything that belongs to the platform group should always be available as full open source in it's latest version
* For fundamental agents first please think twice whether it's really not feasable to release them as open source. If you nonetheless come to the conclusion that some parts shouldn't be open sourced atm. at least the binaries are always made available for free (comparable to parts of the pico manager now)
* the binaries of commercial, not yet open sourced Eigenharp specific parts as well as generator agents could finally be made available to paying customers only (the first presumably available as part of a commercial EigenD base package the latter eventually even sold extra) None of those should be required to start up a minimal EigenD system.

Additional the following aids could be provided:
* partial source code of the commercial agents: E.g. the following blocks could be defined that are interpreted by the merge manager (these sources are not compiled in the open source release, they are just for reference/debugging):
// CONFIDENTAL CODE BEGIN ... END
when merging to the open source release the code inside these blocks is replaced with blanks. Additionally the binaries and debug symbols are added to the open source release
// COMMERCIAL CODE BEGIN ... END
when merging to the open source release the code inside these blocks is replaced with blanks. Binaries are not provided for the open source release, those are sold
// OPEN SOURCE CODE BEGIN ... END
This code is only merged to the open source release (e.g. for conditions in the build scripts, special code to ensure that the open source version does what it should etc.

That way you could perhaps even provide parts of currently unreleased code from Stage or even Workbench so developers can partially debug code but can't build the agent/exe from source themselves. (I know of sources that are prepared that way. When you have the debug symbols you can debug through the visible parts of a file that also contains "blanked out" stuff without problems as long as the line numbers stay correct.
For confidental parts the debug symbols are part of the OS release, for commercial parts they can be downloaded by customers. (Even if you don't have sources at all you at least get a call stack that way)

This would not be "open source" per definition anymore (if you are strict the current OS release with it's binary fragments is also not pure OS though). So some open source enthusiasts will stay away from contributing (this will be the price to pay if you want to keep parts of the code private). But for the rest this should be a viable solution.

Another idea goes into the direction Chris wrote:
All open source stuff is open source as above and additional to getting "just the binaries" for the commercial stuff you get the source code with all stuff you buy under some kind of "shared source" license - you may use it for debugging and understanding purposes (so you can write agents that interact with the commercial one) but you can't distribute it (neither the sources nor the compiled binaries) - so it's essentially "read only" - which is MUCH better than not having the code at all.
This would be more convenient for developers, not sure how much fun the legal part would be though...

This coming from me is all hypothetical of course. As written, it will be really interesting what the Eigenlabs staff will come up with!

Best,
Ferdinand


written by: NothanUmber

Tue, 22 Nov 2011 11:52:09 +0000 GMT

Had another thought about this "open platform" concept:
* It will also be interesting whether/in how far EigenD might establish itself as a platform for third party commercial developers (what would increase the likelyhood that it etablishes itself - most probably it's even a precondition for broader success when you look at ladspa&Co vs VST...)
For that it might perhaps be worthwhile to think about going to a BSD like license (what would probably make the Eigenlabs special agreement unnecessary as a side effect) or offering a free/very affordable(*) commercial license as an alternative. (* at least in the vst world the majority of developers don't earn much from their plugins, so setting on a technology makes only sense when it doesn't eat up large parts/all of the revenue - being free for developers is probably a major reason for the success of the VST standard)
* Unlike VST it's presumably not sufficient to just have a bunch of headers/libs as long as no extensive documentation about the protocols that existing agents use is available - we most probably will have to have a look at the code of the agents that the new one is supposed to communicate with. (At least that was the case with the Midi Illuminator test project - I wouldn't even have had a remote chance to write it just with the infos from headers of midi in and keyboard agents.) So either we'll need more sources (e.g. as suggested above) or a whole bunch of additional documentation will become necessary.


written by: john

Wed, 23 Nov 2011 18:53:12 +0000 GMT

Hi

We've now had time to have a decent discussion about how we're going to solve the problem of Agents in the source tree that contain non GPL code. To reiterate, the principal problem is that setups built to use an Agent that we may decide to do a non GPL version of for a while will not load in a system built from the GPL'd code as those Agents will simply not be there. We solved this issue by just pulling the current working branch which is 2.0. There are, as many of you have pointed out, a great number of disadvantages to not having the ability to work on the current code and this is a very crude fix. We don't like it either.

Instead, we've decided to simply remove Agents from the GPL source tree that we want to release non GPL. In order that setups built in the paid for release (that may well use the non GPL'd Agent) can still run in an EigenD that you might build we will be giving instructions as to how all the bits of non GPL Agents can be copied from your paid for binaries into the GPL'd system so that one can run it against your normal setups. This copying process is going to be clumsy (the files that need to be copied or linked to will vary from Agent to Agent and it won't be wholly consistent) to start with, but as I've said before we'll be working on improving this over time - our ambition is that Agents can be easily moved about in the future a well as traded and swapped. It's going to be a
'developers only, not supported' kind of thing for the near future, but Jim reckons it will work fine.

I think this will provide plenty of flexibility in regards to licensing, whilst still making it straightforward to develop and look at EigenD code. It's a little weird as sometimes an Agent might disappear for a while from the source tree if we want to add some groovy new feature that we think is part of the value that paying customers get, but in the absence of some fancy Agent versioning system, which we aren't going to get to implement this side of 2.something, I think it'll do fine. We'll try to announce any Agent removals a way in advance so that anyone who might be coding with one gets to know about it and can discuss it with us beforehand. We'll try not to do it too often.

To give you an idea of why we got in such a pickle over this, the Agent in question for 2.0 is the audio agent. It's getting multiple configurable outputs and inputs in 2.0 and this is one of the nice new features for base level subscribers. This Agent will certainly be dropping fully back into the GPL release, probably for 2.1 or 2.2, but its a great example of an Agent that is important in the system getting a major upgrade that we don't want in the GPL release for a little bit. It's an Agent that is kind of hard to live without in any setup, and one we didn't want to fork for reasons of cruft - we really don't need two kinds of audio in/out in the long run.

So 2.0 will be reappearing on github as soon as we get the necessary rearrangement done.

John

BTW, in terms of an open platform (to NothanUmber), I don't think there is any need to change the EigenD licensing - GPLv3 has excellent protections against Tivoisation, a thing I loathe, and those protections are personally important to me. A binary API coupled with the ability to look at the vast majority of the system source code via the GPL and build a running system from scratch does enable anyone to make and sell Agents whilst still protecting our interests in the broader system - I think that scenario contains the best of all worlds. After two years of considerable change we are now heading towards being able to stabilise and document our Agent interface nicely, and I'd like to see this emerge as part of our collective thinking over the next few months.


written by: NothanUmber

Wed, 23 Nov 2011 20:17:15 +0000 GMT

A GPL version with source code for most agents (but the ones you explicitly exclude) for open source work and reference plus a binary API for commercial stuff sounds like a good plan, indeed!
So we are always on the safe side when we write new agents and for modifications of existing ones the pre warn system is definitely helpful (And if an agent is made unavailable before a currently worked on feature is ready perhaps there is room for individual case by case solutions - e.g. via code snippets provided under NDAs, final merging and testing on Eigenlabs side based on a patch against the last free base agent (when it's a worthwile addition) etc.)

Best,
Ferdinand

P.S.: you might want to sticky your announcement which is more relevant than my answer :)

P.P.S.: Unfortunate that it's the audio agent that is pulled (which is pretty fundamental). Without at least having the binaries available for free (what seems to be the intention in this case) this renders the OS release mostly useless by itself, so in order to use it you really have to buy EigenD to get the binaries. It will be interesting whether that is "good for Eigenlabs" in the long run as developers can't start to experiment with the system anymore without investing into it first, what could have a severe impact on the likelyhood of becoming popular. Will see...

(P.)^3.S.:
Would it perhaps be possible to just limit the number of audio channels and release this binary together with the OS version? Then you could test everything but surround stuff - what would be better by miles than without an audio agent at all. And without the sources people couldn't easily "undo" your limitation.
That could be a viable approach for agents that are pulled in general: Where viable think about offering binaries that are limited to the functionality they had before. That way the OS version would loose transparency and options to change things when an agent is pulled but at least not the previously available functionality.


written by: john

Wed, 23 Nov 2011 21:36:15 +0000 GMT

Hi Ferdinand

I don't think the missing audio agent will be much of an issue - we plan to drop it back into the GPL release quite quickly and if anyone really wants an audio agent outside of the mainline, they can always fork the one in 1.4, call it something else and build it. If you have EigenD 2.0 from Eigenlabs that's be fairly pointless, but if you truly didn't want to give us money then its there as an option. It's a little inconvenient, but that's kind of the point.

I stickied that post, thanks for the suggestion..

John


written by: NothanUmber

Wed, 23 Nov 2011 22:22:59 +0000 GMT

Yes, presumably most "never seen this, want to try it" developers who did not already buy the commercial EigenD package just because they are Eigenharp users will presumably be alerted at the point when EigenD becomes a VST/AU plugin itself, so it can effectively act as a modular environment inside existing DAWs (what might happen much sooner than the massive rise of expressive controller instruments - which could more be a result of the spread of EigenD than the other way around. So I think that's the first reachable game changing point where specifically commercial devs could be attracted because it would mean an instant increase of potential clients who just use what their DAWs support - MIDI/VST/AU atm - so the henn-egg cycle has to be broken - most probably from the EigenD side).

And this step presumably makes more sense when the experimental phase is over and everything is more tested and proven (the first impression should be...impressive... because large parts of the "audio community" seems to be quite unforgiving regarding once-and-for-all-categorization based on glimpse looks of a few who pre determine the opinion of the rest via forum posts that are often just taken as reference without further questioning them because there is just so much stuff that is constantly released. So nobody can test everything - or even retest whether something earlier not-so-good became better).

So at the latest at this point of time imho it would be helpful when the GPL version does at least offer a complete base set of fundamental agents, so a bigger group of developers can start to experiment with stuff "without obligations".

-Ferdinand


written by: dhjdhj

Thu, 24 Nov 2011 03:33:17 +0000 GMT

I submit that a viable MaxMSP environment could be just as successful. Indeed, if there are any other experienced Max users listening, I'd like to suggest some kind of collaboration to build up a significant environment.

I've only had a few hours to spend on this but I now have single object abstractions that can be used to create layers, splits that directly control external midi devices as well as VSTs. I'm very encouraged by the potential of this.


written by: barnone

Thu, 24 Nov 2011 04:10:15 +0000 GMT

@John
Great solution. Thank you.


written by: carvingCode

Thu, 24 Nov 2011 15:35:04 +0000 GMT

I'm very encouraged by the progress of EigenD, the opening of the source and the interesting early 3rd party explorations. I also understand the need to keep some things private to strengthen the entire platform, i.e. hardware/software/company/people.

The next couple of years should be quite interesting.


written by: jim

Thu, 24 Nov 2011 15:38:42 +0000 GMT

The 2.0 code is now back on github (minus Audio), as the '2.0' branch.

We'll post instructions on how to copy the audio agent binary when we make a binary release.

jim


written by: geert

Fri, 25 Nov 2011 15:02:48 +0000 GMT

The last commits in the 2.0 branch now also remove the reliance on the old GCC 4.0 version and allows the code to be compiled with GCC 4.2, which what we're now going to be using by default also. It should thus be easier to get EigenD to compile on Macs running OSX Lion.


written by: NothanUmber

Fri, 25 Nov 2011 21:39:20 +0000 GMT

That was fast. Nice to have it back! :)

I guess it still compiles on Snow Leopard? (Didn't dare to update my Mac yet)



Please log in to join the discussions