Packaging a Drupal site -- possible for under $10,000?

published by tgeller on Mon, 2010-06-07 12:50

I'm currently developing some Drupal 7 instructional videos for lynda.com. The company likes to offer "exercise files" for their courses so folks can jump in in the middle. In other words, someone could start at Chapter 5 by loading the Chapter 5 exercise file, which would make the site appear as though all the exercises from Chapters 1-4 had already been completed. A course would have at least a half-dozen such files (if one per section), and possibly as many as 70 (if one per video, as is preferred).

This has proven very difficult in Drupal. In Drupal Essential Training (2008) we included a .sql file at every step, but that didn't put assets in their proper directories. We included graphics wherever they first appeared, but that meant a user would have to go through all previous chapters to find the assets and load them up. That also didn't address the issue of modules and themes. It's been a support nightmare.

An ideal exercise file would:

  • import content and settings into the database;
  • put assets (such as graphics) in the correct directories;
  • install and enable the latest versions of specific modules (preferably by grabbing them live from drupal.org);
  • configure it all; and
  • be installed through a process that beginners could understand from a well-made three-minute video.

Oh, and the proposed solution must be available soon for Drupal 7 -- for instance, modules that carry the D7CX pledge. And because people will be watching these courses until Drupal 8 comes out, it also has to take into account future versions of Drupal 7. That's why I hope I can find something that grabs "live" modules and themes, rather than packaging them in the exercise files themselves. (That's not essential, though, as users could just use Drupal 7's new Update features to bring the software up to date.)

I posted this question to a mailing list for Drupal consultants and got some good advice. But none of the recommendations seems to really solve the problem. The options -- and their drawbacks -- appear to be:

  • Demo. Only manages the database. Doesn't do anything with site assets, modules, etc.
  • Drush, perhaps with a custom command. Hard to use, but perhaps promising? I only use it rarely and am unclear on how it would work.
  • Features. Not promised for Drupal 7. I watched this video about Features by Mustardseed Media and think it's only a partial solution since it doesn't appear to save content. But maybe we could include a Feature module for each chapter and a database.
  • Patterns. Not promised for Drupal 7; in fact the Drupal 6 version is still only "-dev".
  • Installation profiles. I like this because it could contain everything: files, assets, and database settings. The user would have to reinstall a complete copy of Drupal each time, but that's fine.
  • A hosted service such as WebEnabled. (That's how TopNotchThemes lets users try out their themes before buying.) Presently too complicated for lynda.com's audience, but that could change. And would the user pay for this external service, or would lynda.com? Currently, people who buy lynda.com's exercise files download and own them, so a hosted service would be a big business-model change.
  • A version-control system such as GitHub. Same problems as other hosted services, and currently far too complicated for lynda.com's audience.

I doubt many people will have read this post down to here. If you're one of them, here's a secret opportunity... shh.... ;)

Any answer we find will need custom development. I'm not a programmer; I can't do it. The ideal solution would be a module that takes a snapshot that meets all our criteria and packages it up into a downloadable file. I'm looking into funding such a module, which would then be released back into the community under the GPL.

I talked to one developer I admire a great deal; he believed it would take 9-12 weeks and cost $10,000, which is WAY beyond what I could raise. I suspect he missed some easier options (such as modifying one of the solutions above). If you think you can do it -- or can make any suggestions -- I'd love to hear them!

Blog category: 
Drupal Planet

Comments

Anonymous (not verified)

Mon, 2010-06-07 15:53

I had exactly the same problem with my Fiat Panda car. They couldn't custom build a new V8 engine it for me under 10K either. Darnit!

tgeller

Mon, 2010-06-07 16:03

You believe that the task is irreducibly complex, and that the price is fair. You might be right; I posted to find out. Extracting meaning from aggressive snottiness: Just one more service we offer.

adrinux (not verified)

Mon, 2010-06-07 15:55

Unless I'm misunderstanding the problem, the answer is drush with drush_make. You write a drush_make 'make' or 'build' file (different people use different terminology) which essentially describes all the modules/themes/features/patches/3rd party libraries etc you need and where they should be installed. And then a single command will pull everything together from drupal.org, public code repositories or your private code repositories. You can also integrate install profiles into this workflow but you don't have too. drush_make is currently used to assemble Drupal distributions like Open Atrium, and will be/is used to package distributions on drupal.org. See http://asides.bmannconsulting.com/?tag=drush_make drush/drush_make are essentially independent of drupal version. You can already build a drupal-7 site with them. It sounds like you might need to do some customization (I'm not entirely sure of your needs), but extending drush_make would be faster and cheaper than an entirely custom codebase. mig5 talks a lot about drush_make in combination with aegir, ignore the aegir stuff in your case but the drush_make stuff may be useful: http://drupal.lu/mig5net-building-drupal-applications-drushmake-provisioning-sites-aegir http://www.mig5.net/content/drupal-deployments-workflows-version-control-drushmake-and-aegir One other link: http://robshouse.net/blog-post/playing-drush-make If you have other questions drop me an email or pop into #aegir or #drush on irc.freenode.net to talk things over.

tgeller

Mon, 2010-06-07 16:04

This is VERY helpful -- thanks! I'll send you a private note as well to explore further.

Jamie Tomlinson (not verified)

Mon, 2010-06-07 16:51

I also recommend drush make especially coupled with something like features. Have you had a look at the demonstration project at http://drupal.org/project/demo you may be able to couple that with drush make and some custom file retrieval as well. I have used drush make a bit let me know if you have any qu's. Regards Jamie

EclipseGc (not verified)

Mon, 2010-06-07 17:37

As others have already mentioned, Drush make to the rescue. That being said, you shouldn't completely discount what a good install profile might buy you. For example: a Chapter 1 install profile could set EVERYTHING up specific to that chapter (custom data placement + features will get you 99% here). Chapter 2 profile will do the same thing, and probably be most of the same code, except will utilize new (or different) features. This can all peacefully co-exist in a single drupal install IF you do the following: /profiles/chapter1/modules/ /profiles/chapter2/modules/ /profiles/chapter3/modules/ etc... In each modules director include the modules specific to that install profile. Even if you use the same named modules from one profile to the next that will still be ok because the profile will install from it's own modules instead of another profile's. I've not tried to use drush make to pull the same module multiple times, but... you can always try :-) Be one heck of a make file, would love to see it.

Lee (not verified)

Mon, 2010-06-07 17:45

As a happy client of an www.omega8.cc managed Drupal/Aegir install, it seems trivial to me. Just use Aegir to make site backups whenever you want. You'll end up with a tarball of the site's Files directories plus the database. End of story. Unless I'm missing something, this seems very do-able, and not necessarily expensive with a short-term VPS for the project.

tgeller

Tue, 2010-06-08 01:04

Thanks for the idea. Could Aegir also package a site's modules and themes, or (better) create pointers to pull them from drupal.org?

tgeller

Tue, 2010-06-08 01:09

Oh, wait... if I'm reading the overview right, Aegir requires Unix. So it could only be done via hosted/managed Aegir... was that what you were thinking? That brings us back to issues of commerce, such as: Who pays for the hosting? But it's worth exploring... thanks again.

KarenS (not verified)

Mon, 2010-06-07 19:11

Drush Make is great but won't give you a file directory or a database. It will work great to create your code base. Drush has commands to backup the database and rsync files that you could combine into scripts to create sites on demand. Either Drush or Provision module has a command to tar an entire installation -- code, database, and files -- into a single gzip file, which you can later untar to duplicate the site somewhere else. You might also do this with Aegir -- creating a separate platform for every chapter with the right code, database, and files, then create new sites on demand with the right platform for the chapter at hand. Aegir can be used to create either D6 or D7 sites, regardless of the version of Aegir you're using. Aegir uses the Drush command mentioned above to accomplish this, but wraps in it a UI with lots of other goodies.

tgeller

Tue, 2010-06-08 01:33

Thanks so much for your thoughts. Of the two ideas, I'm leaning toward an unholy alliance of Drush Make, webchick's DB-packaging script, and some kind of PHP scripting. To repeat a question asked in another comment: Could such a thing be module-ized, and made to work on both Windows and *nix (particularly Mac OS X)?

iMadalin (not verified)

Sat, 2010-08-14 00:52

aegir knows drupal 5 and 6, 7 is only experimental. aegir was builted on top of drupal 5 and at 0.3 ported to drupal 6. drupal 7 has many features different then drupal 6. most of us will stick to drupal 6 a long time, you can't change all clients over night to drupal 7... maybe i'll be experimenting with some small new clients, but ecommerce i think it will be on drupal 6 and ubercart log time before big clients will have confidence into something new. i admit that drupal 7 comes as u want from a long time ago, but drupal 6 still handles all u can do with drupal 7 if u know how to handle some modules.

Peter (not verified)

Mon, 2010-06-07 20:25

I've done something similar... but I didnt care about "up-to-date" code. I would just create a dump after each chapter and do a "5 step drupal distribution" http://www.lullabot.com/articles/5-step-drupal-distributions . If they have 5 chapters... you have 5 drupal distributions. If their are 25 chapters, their are 25 distributions all packaged separately. Easy as pie. This may cause some download redundancy, but who cares, it keeps the environment very controlled. If your teaching people your going to want a controlled environment. Pulling code from drupal "live" could create issues, change menus, etc... I think using features/patterns/etc would just complicate things and be more confusing to the new drupalee. Just some thoughts.

tgeller

Tue, 2010-06-08 01:28

Hi, Peter. Angie's DB script and instructions at http://www.lullabot.com/articles/5-step-drupal-distributions are spot-on; together with drush_make, I think we may have a solution! (I note Damien McKenna's critique of her approach, but think it's fine for my purposes.) Now... tying it all together. Excuse my ignorance, but would it be possible to trigger drush_make commands via a Drupal interface, and make it work on both Windows and *nix, particularly Mac OS X? If so, couldn't the whole shebang could be turned into a module? I realize that might require running Drupal in an unsafe mode, in order to give the module access that Drupal doesn't usually enjoy.

Anonymous (not verified)

Mon, 2010-06-07 21:23

I don't know if there is an open alternative as consumer friendly but maybe something like mokafive with managed Virtual Machines. You setup a base VM and create deltas or snapshots at the end of each lesson. If you can setup virtual layers correctly; you can update the stack, drupal itself, keep a separate layer for the site sample data and maybe server tools (like drush) that students use in the lesson. You can use Mokafive to stream the VM to the user where the VM will be cache locally. You can package a set of VMs on a WD passport HardDrive as a deliverable (it runs across both Macs or Windows. So you can quickly demo any of the Drupal distributions like Open Atrium with sample data on the demo sites. If the student makes a mistakes they can roll the lesson back to beginning of the lesson regardless of the damage they do to the site by rolling back the VM to the snapshot. http://blog.mokafive.com/2010/03/mac-windowscom-mokafive-virtual-layers.html http://blog.mokafive.com/2010/04/free-player-is-back-free-player-is-back.html http://drupal.org/node/678498 Just an idea, I haven't seen anyone do this seamlessly yet.

Marc Caballero (not verified)

Wed, 2010-06-09 05:10

Man, I hear ya. File-management with Drupal is somehow a hidden pain-in-the-neck, at least of you're not anticipating it early. But in many Drupal roll-outs that I've built, I've depended on a very simple work-around. That is: use a----t-- con----- d---rt-y. (just teasing, read further). My simple solution? I create an assets directory inside ~DrupalRoot/sites How it works: When dealing with multi-site installations, Drupal automagically assigns a file-system path for you based on the location of the site's settings.php. So for me, I configure my sites to throw its junk on /sites/assets/{mySiteName}/files or simply /sites/assets/files -- the reason being is that most Drupal deployments require the /sites directory plus you can create custom directories there for whatever purpose you have in mind. And using *features* and *strongarm* modules, you can specify mandatory locations for your file-system path and what not. It's an ultimate fix. File-system config issues mostly matter if you intend to embed/upload files using the upload module. But brother, if you need to specify a more fine-grained spot for your files, then use filefield -- which allows you to configure the path of your files anytime, anywhere (fyi there are other contrib modules that complements this too). If you need help on this, I can build a WebEnabled Drupal app for you which we can push to the WebEnabled marketplace as a free app that users can play with. I'm hoping that this will save you from your misery sooner than expected. ^--- and you don't even need to break the bank :)

tgeller

Wed, 2010-06-09 11:31

Very clever! That sounds useful for many purposes, but I'm afraid it won't fit mine because neither Strongarm nor Features is promised for Drupal 7. :( But if nothing else, I learned about the Strongarm module (which I was unfamiliar with). :) Thanks!

Marc Caballero (not verified)

Thu, 2010-06-10 17:31

I've stumbled on a podcast/article that promises a port of these modules to Drupal 7 when a more stable development release is available. When I find it, I'll post it here.

Jerod (not verified)

Mon, 2010-06-21 12:54

A basic multi-site setup with each chapter its own subsite is the way to go. To get the one step install for your users I would create a variant of XAAMP that auto installs the entire setup and serves up the learning suite from localhost. Feel free to contact me: Jerod Fritz ( jerod@centogram.com ) http://www.centogram.com

tgeller

Mon, 2010-06-21 13:04

Hi, Jerod -- thanks for the idea, which is both unusual and intriguing. It strikes me as extremely heavyweight, though, as it would require multiple copies of everything. Also, it would lack the auto-update mechanism I'm still hoping for. It's quite original, though! I'm pursuing another avenue right now, but will get let you know if that, and some other things, don't pan out.

bserem (not verified)

Fri, 2010-07-23 03:16

I was thinking: backup and migrate with backup and migrate files (submodule) could do the job in almost no time and with not much hassle. Furthermore it is quite easy for all audiences since all you have to do is load the backup on an existing D7 installation. If you decide to use this path all you need to do is develop a D7 version for backup_migrate_files :)

tgeller

Sat, 2010-07-24 14:16

Thanks for the idea! Although he didn't take the D7CX pledge, I wouldn't be surprised if ronan decides to migrate the module himself. Eventually. :) It'd still only be a partial solution though. The backup_migrate_files module currently appears to support only the files directory, which means the user would still have to download and install all the suporting themes and modules manually. That actually wouldn't be a big deal for the Drupal Essential Training course -- it requires that students install very few contributed modules. But it wouldn't be as useful for other projects... so I'm still hopeful for a more-complete solution. Still, I might pursue this in some way. Thanks!

chrisco (not verified)

Mon, 2011-04-04 15:37

So the best a mailing list of Drupal consultants could come up with was a list thta didn't help you solve your "problem" and a $10,000 quote? That reinforces the "overpriced for what you get and vs. the alternatives" smell in the air when prospective new Drupal users start sniffing around. It also seems counter to the pay it forward, non-greedy, give-to-the-community philosophy that open-source and community are about. What does it mean? Does it mean one of these two (of multiple other) possibilities? A) Drupal is "too" complicated and it really does take $10,000 to develop solution to the "problem" you described. If so, is that sustainable? I mean for less than $10,000 you could just get an intern or virtual assistant to do it manually for you. And you'd still have $9,000 left over for whatever you wanted. B) The community has too many people who quote Fortune 500 prices to small businesses and/or startup projects, thus driving prospective new community members, users, etc. into the arms of the alternatives that Drupal has on both sides, such as WordPress on one side and frameworks, such as RoR, Django, and the PHP and other frameworks on the other. It makes me think of those "hybrid" bicycles that are positioned as half road bike / half mountain bike. Sure, if you buy one of those, then you only need one bike, but it's worse than a road bike on the road and it's worse than a mountain bike on the trail. And if it costs more, and has beauty and other drawbacks, too... WTF?! That's a recipe for buyer's remorse and a product that fails to thrive, at least unless there's some major compensating factor, such as the price is really cheap or the community is blazing trails, growing, building momentum, attracting people, solving problems, etc. I mean, to play devil's advocate and ask the question that prospective new users have, what is Drupal's value proposition, how is it positioned relative to the alternatives, and do the answers to those questions hold up to scrutiny. I know Drupal must be the best (or at least a great) tool for certain job(s), but what job(s)? Not blogging. Not forums. Not frameworks. Not beauty. Not speed/performance. Not usability. Not UI/UX design patterns. Not affordability, apparently. Where does it lead? Why should people considering the alternatives use Drupal? What specific job(s) is it the best tool for? Speaking of affordability, as part of the research I'm doing right now as a prospective new Drupal (see "Fresh (and Frank) 1st Impressions From a Drupal Newbie ": http://groups.drupal.org/node/136294#comment-455844 ) I found my way to this page http://www.lullabot.com/contact/work which is linked to from this page http://drupal.org/user/24967 which belongs to webchick, who is the #2 code committer for Drupal core, from what I can tell. That form suggests that, at least with Lullabot, you can't do anything on Drupal for under $15,000 to $50,000 and that $50,000 to $100,000+ is more like it. Of course that's small change to large companies, but it's a deal breaker for the other 90%+ of us. I mean, are a few large companies what's going to enable Drupal to grow, thrive, attract people, users, etc? I don't think so. Ok, before I close, here's one more analogy that pops to mind: AOL and what happened to it when it was disrupted by better, prettier, cheaper, more nimble, and more active alternatives. AOL failed to adapt and turn itself into something people wanted. Instead, it tried to raise switching costs, such as by not making their address book easily exportable, not offering a forwarding function, etc, etc. Greedy. Stupid. Dying. Classic. Could something like that happen to Drupal? Is it at risk of getting (further?) marginalized, as more and more existing Drupalers leave and fewer and fewer new ones join? I hope it doesn't happen, but it seems like a risk, at least from this newbie/prospective Drupaler perspective. I hope my comments have at least some useful nugget or perspective for someone. They represent a real-life usability test of sorts. Cheers, Chris

tgeller

Mon, 2011-04-04 21:03

I think that your comments -- particularly about responses to my original project -- are astute. Because they're stated aggressively, they'll undoubtedly rouse a defensive attitude among potential respondents. I hope they'll put that away before committing fingers to keyboard. (Sorry for the flowery writing. I've been listening to a lot of Mark Twain.) Anyway -- to your two points. 1) I agree that respondents should have come up with a solution that was less expensive and that cost less. But looking backwards, I can see that my specs were too imprecise and inflexible. The solution I finally settled on -- adjusting settings in phpMyAdmin to make the process more foolproof -- didn't deliver on all my specs, but was good enough. How could respondents have done better? If any had worked harder to engage me, together we could have figured out what "good enough" was. I don't blame them for not working the connection too hard. I was a potential client with uncertain funding, trying to do something new and different. That's a big risk for any consultant. Having said that: The failure of this project *does* point out that Opportunties Are Being Missed. Maybe the problem is that there aren't enough low-end Drupal developers out there. I'm still waiting for someone to emerge as a standard solution to packaged, low-price-barrier solutions. I believe that the tools are there: The culture just hasn't caught up. 2) For the record, I've ridden the same hybrid (a 1998 Kona Lava Dome) for twelve years, mostly as an aggressive urban road bike, and have never found one I liked better. :) 3) I used AOL as an example in a recent column for Drupal Watchdog. In that case I said that AOL missed the boat because it thought its valuable product was the "walled garden" whose rights it wholly owned. There may be a relevant parallel to your analogy. Drupal developers may be focusing too much at the software rather than its audience. That's a mark of Innovators and Early Adopters, and is completely understandable. Will Drupal cross the chasm by focusing more on customers? I believe the software has made motion in that direction on the basis of Dries' fiat regarding usability in Drupal 7. But the software is not the community; as the saying goes, Internet technology is wonderful, but it's all done with people.

chrisco (not verified)

Fri, 2011-04-08 14:32

Good comments and cool bike. Yes, I was a little "aggressive" in my comments and I apologize for that. My intent was to provide an unedited look at what some prospective new Drupal users and customers are thinking in the hopes that it will motivate or inspire the way that some "aggressive" comments at halftime to great team that is down by a few points against a tough and fast competitor. I'm in Drupal's corner. Cheers, Chris PS: Cool bike :)

Add new comment