Monday, March 22

Frameworks, Libraries, Namespaces and Distributions

I've been busy not working on the code for my game engine framework Grease recently. I've actually been working on the documentation, but since I can't help myself, I've been thinking a good bit about the code, and specifically what I want to work on next.

The functionality that I plan to work on next is rather general 2D geometry and vector graphics. The important thing is that although Grease will depend on these things, they are not part of the framework per se. They should be useful for folks who don't buy into Grease's framework dogma, or don't need its features. Simply put they will be libraries that could be perfectly happy outside of Grease or any framework.

So, humble denizens of the Python planet, what is the best approach to naming and packaging these libraries for distribution? I see two possible options, both with pros and cons:
  1. Put them under the grease "brand" packaging them as something like: grease.geometry and They would be distributed separately and probably together with grease the framework as well.
  2. Give them their own independent name (I'm leaning toward flatly) and package them separately from grease, though grease would depend on them.
#1 has some advantages:
    • Less "brand" complexity. It's obvious that is part of the grease project, and when used inside of grease there's one less arbitrary brand-name to remember.
    • It's easy to envision these libraries fitting snugly with the rest of the framework in terms of documentation and what not.
    • It's pretty easy to justify folding the release cycle of the libraries in that of the framework.
    And some disadvantages:
    • The independence of these subpackages is unclear, folks may be reluctant to use them if they feel that they carry too much framework baggage or cognitive complexity.
    • I'm not sure the best way to actually make these separate from a distutils perspective, or whatever distribution flavor of the minute is in fashion. I suspect they should just live in their own directory trees and get stitched into the grease top-level package at installation-time. I'm wary of drawbacks and unintended side-affects of that, however.
    • They're nested rather than flat (sorry Tim).
    #2 also has some advantages:
    • The independence of the library package(s) is obvious.
    •  It's rather obvious how to organize them from a distribution perspective.
    • They aren't weighed down by any framework connotations.
    • The namespace is flatter.
    And of course disadvantages:
    • They don't promote the framework "brand" in any way and may be perceived as less well integrated with the framework.
    • They would feel like more independent creatures to me, thus I think I would tend to fuss about them individually more. That's a subjective thing, of course, and not necessarily bad for users.
    • It's not as clear to me how to deal with them as grease dependencies. Bundle them or fetch them at installation time?
    Honestly I'm able to justify both ways to myself, and of course it's not a matter of life-and-death. But it does feel like an important api decision and I value any opinion or ideas you may have on the matter. So please comment if you feel inclined.

    Also I'm interested in people's opinions on bundling vs. fetching dependencies at installation. In the bad old days, bundling was basically a no-brainer with Python, but given things like setuptools/distribute and buildout, the latter has gained appeal. Which do you prefer?


    Blogger chrism said...

    Definitely don't use a namespace package. This has caused us endless pain and confusion in zope and repoze lands because the namespace package invariably ends up being a "garbage barge" that means "stuff created by the same people that created all this other stuff". People are busy and cannot be bothered to make the distinction between a brand and a package.

    9:18 PM  
    Blogger casey said...

    Good advice, I kinda wondered how the whole little-"z" zope namespace conbobulation worked out. Seems like you're a fan ;^)

    It also occurs to me that temptation and/or ignorance could could conspire to sneak in dependencies between the library and the framework later on. This seems far less likely with a totally separate library package and namespace.

    9:51 PM  
    Blogger casey said...

    Oh an by the way, "bfg" would have made a nice name for a game engine, damn you! 8^)

    9:57 PM  
    Anonymous Anonymous said...

    Bundling vs. Fetching: Fetching is preferable, the tools exist to make this easily manageable now. It gives a big advantage in things like sliding in a bug fix release into a dependency under the stack, without needing to reinstall the full stack. If I've got a web app built with a framework that's been chopped into lots of dependencies, and I stumble on a show stopper bug in one of those parts, it's easy to pull just upgrade that part - either from PyPI if there is a newer release, or from upstream source control. If the framework is one big bundle though, I can only pull the whole thing down, and I'm very likley going to also pull in a whole wack of changes which I have no interest in dealing with at that point in time.

    It doesn't have to be an either/or proposition though, there are lots of ways of stitching together dependencies so that an end-user only sees a single install. Tools like z3c.recipe.eggbasket allow you to ship all dependencies in a single compressed file. While Ruby on Rails takes another approach by keeping multiple dependencies in one source tree, with each dependency in a seperate sub-directory, but then releasing them separately (a more brute force method, since there is no metadata to indicate to a packaging tool that you're using a development version of a dependant pacakge).

    11:59 PM  
    Anonymous Anonymous said...

    I always fancied 'flatter' as a name for a 2D vector graphics engine.

    flatter, verb, to represent favorably; to show to advantage.

    6:48 AM  
    Blogger Chris Perkins said...

    I've been struggling with this very issue with TurboGears for a bit now. I now feel that #2 is the best option. The fact is, someone might not find they want to swallow every part of your framework, but find a morsel of it particularly good. If you give that morsel it's own unique identity, it can live on it's own, and you might find yourself with some increased contribution! While I agree, there is some confusion with respect to creating a framework based on parts, the fact is that people actually care what things are named, and if you name everything homogeneously, someone not completely astounded with your framework may not use it. I find that I'd really like to use schemaish and validatish, but I just cant get past the fact that my code will have cues to these packages, leaving my user base wondering if they should throw my stuff away and just move over to restish. It also seems to cause some user confusion in general. TG relies on repoze considerably for it's auth/auth, and while this has been a great partnership in general, I find a lot of users a bit dismayed when easy install downloads zope.interface, for instance.

    In fact, I have begun to split TG's guts in to morsels that can be used independently. is our dispatch mechanism, split off on it's own. The other benefit we are reaping besides increasing our collaboration base is that it's a heck of a lot easier to unit test things when they are split off. Cutting the strings allows for an increased awareness of separation of concerns.


    7:50 AM  
    Blogger casey said...

    @tartley - flatter is also a name I'm considering. flatly has a little bit of an edge to it and it's shorter, but I think if it can paint pretty pictures, flatter is a better name.

    Now the next question is whether the geometry and vector graphics functionality go together in one lib. Certainly the latter depends on the former, but I kinda think they would be clearer as separate things. The hardest thing there is thinking of two good names 8^)

    @Chris P. - I'm definitely sold on independently naming each lib and keeping them separate. You make a good point about "pull anxiety" to coin a phase. I've definitely found myself worried when packages have seemingly crazy dependancies. Like when you install a little command line utility and it pulls down all of KDE ;^)

    One of the challenges to games, compared to say web frameworks, is that folks always want to be able to turn their apps into a standalone executable, or at least create a reasonable installer for the,. That creates additional packaging challenges. Regardless though I think the benefits of package modularity are enough to counterbalance the added costs.

    8:43 AM  
    Blogger grey said...

    Give it its own separate namespace, for a few reasons.

    Branding shouldn't enter into your thought process. It's useful only for advertising and selling things, not for designing them and never for using them. Marketing-centric design rarely works even with consumer products. I can't see it working at all in software design.

    If you want to relate the packages, do it outside the programming scope. That's what web sites, logos, documentation and tricky naming schemes are for. The packages should be focused only on what they do with no allowances to anything else.

    Any nice side effects (like one working well with the other) should be mentioned somewhere else; the fact they're both on your site and both written by you should give most people a good hint they're compatible.

    Making a programming remember and acknowledge that the package they want to use is written by the same person who wrote this other package they're not interested is annoying. It's the same situation when a company lumps all their unrelated applications under their corporate name on Windows' start menu. Your users don't really care who made it; they only care that it works and they can find it. Since they're interested in functionality, the packages should be organized that way. Since your graphics library provides independent functionality, it has its own namespace.

    Keeping the graphics library in a separate namespace can serve as a reminder to you to keep the entanglement between the two packages to a minimum and to maximize compatibility with other libraries.

    It will be less confusing to people searching for your graphics lib, because no matter how many times you mention it doesn't require Grease, if you put it in the Grease namespace a lot of people will still assume that requirement. If they're using Pypi they'll spend no more than a few seconds scanning your single entry in a long list of search results, so you need to be as clear as possible.

    I just dragged my sorry ass out of bed, so apologies if this sounds strident, rambling, insane or all three.

    10:37 AM  
    Blogger casey said...

    @grey - Good point about the start-menu grouping, I hate that. Even worse is when you have a bunch of brand groups with only one thing in each. Pointless nesting does suck.

    10:56 AM  

    Post a Comment

    << Home