Tom Geller's nuggets o' joy

I'm currently developing some Drupal 7 instructional videos for lynda.com. The company likes to offer "exercise files" for their courses so folks can jump in in the middle. In other words, someone could start at Chapter 5 by loading the Chapter 5 exercise file, which would make the site appear as though all the exercises from Chapters 1-4 had already been completed. A course would have at least a half-dozen such files (if one per section), and possibly as many as 70 (if one per video, as is preferred).

This has proven very difficult in Drupal. In Drupal Essential Training (2008) we included a .sql file at every step, but that didn't put assets in their proper directories. We included graphics wherever they first appeared, but that meant a user would have to go through all previous chapters to find the assets and load them up. That also didn't address the issue of modules and themes. It's been a support nightmare.

An ideal exercise file would:

• import content and settings into the database;
• put assets (such as graphics) in the correct directories;
• install and enable the latest versions of specific modules (preferably by grabbing them live from drupal.org);
• configure it all; and
• be installed through a process that beginners could understand from a well-made three-minute video.

Oh, and the proposed solution must be available soon for Drupal 7 -- for instance, modules that carry the D7CX pledge. And because people will be watching these courses until Drupal 8 comes out, it also has to take into account future versions of Drupal 7. That's why I hope I can find something that grabs "live" modules and themes, rather than packaging them in the exercise files themselves. (That's not essential, though, as users could just use Drupal 7's new Update features to bring the software up to date.)

I posted this question to a mailing list for Drupal consultants and got some good advice. But none of the recommendations seems to really solve the problem. The options -- and their drawbacks -- appear to be:

• Demo. Only manages the database. Doesn't do anything with site assets, modules, etc.
• Drush, perhaps with a custom command. Hard to use, but perhaps promising? I only use it rarely and am unclear on how it would work.
• Features. Not promised for Drupal 7. I watched this video about Features by Mustardseed Media and think it's only a partial solution since it doesn't appear to save content. But maybe we could include a Feature module for each chapter and a database.
• Patterns. Not promised for Drupal 7; in fact the Drupal 6 version is still only "-dev".
• Installation profiles. I like this because it could contain everything: files, assets, and database settings. The user would have to reinstall a complete copy of Drupal each time, but that's fine.
• A hosted service such as WebEnabled. (That's how TopNotchThemes lets users try out their themes before buying.) Presently too complicated for lynda.com's audience, but that could change. And would the user pay for this external service, or would lynda.com? Currently, people who buy lynda.com's exercise files download and own them, so a hosted service would be a big business-model change.
• A version-control system such as GitHub. Same problems as other hosted services, and currently far too complicated for lynda.com's audience.

I doubt many people will have read this post down to here. If you're one of them, here's a secret opportunity... shh.... ;)

Any answer we find will need custom development. I'm not a programmer; I can't do it. The ideal solution would be a module that takes a snapshot that meets all our criteria and packages it up into a downloadable file. I'm looking into funding such a module, which would then be released back into the community under the GPL.

I talked to one developer I admire a great deal; he believed it would take 9-12 weeks and cost $10,000, which is WAY beyond what I could raise. I suspect he missed some easier options (such as modifying one of the solutions above). If you think you can do it -- or can make any suggestions -- I'd love to hear them!

In preparing a "Drupal 7: New Features" video series for lynda.com, I decided to go waaaay back to when we were just starting to plan Drupal 7. Lo and behold, Dries wrote a blog post in February 2008, a few weeks before Drupal 6 was released, with a list of features he wanted to see in Drupal 7.

Now we're pretty darn close to Drupal 7's release. How're we doing? I think we can break down his 11 points into three categories: Done, Sort Of Done, and Not Done. (Special thanks to chx (Károly Négyesi) for providing some details via IRC.)

Done

• Usability improvements comprise the most-visible changes. The intense work of many people resulted in great advances, with Acquia's funding of Mark Boulton Design providing strong direction.
• Custom content types in core: Win! Most of Content Construction Kit (CCK) is now part of Drupal 7, although that popular module will continue to be available for some functions that didn't get into core.
• Automatic upgrade functionality: This is a HUGE win, making Drupal more accessible to people without the knowledge or access to make direct changes on their web server.
• Better internal APIs: I'm not a qualified to talk about these from first-hand knowledge, but chx and Heine gave two convincing examples: the new database layer, which allows Drupal to connect to any database, with the appropriate driver; and stream wrappers, which let Drupal treat remote files as though they're local.

Sort Of Done

• Better media handling: Media handling in Drupal 6 stank, and there's no denying that it's a lot better in Drupal 7. Before, you had to add several contributed modules -- CCK, ImageField, ImageAPI, FileField, and ImageCache -- to do the most mundane tasks. Those have all been incorporated into Drupal 7 to some extent, with intelligent defaults. Further, the built-in Article content type includes space for a single graphic. However, it's a far cry from the sort of media handling that would be possible with a true WYSIWYG editor that allows you to place as much media as you want, wherever you want it.
• Better external APIs (import/export, web services): This is another area that I'm unqualified to talk about directly. chx says this "deserves a tick, too" because of such matters as delivery callback. But I haven't heard much else about this point, and welcome clarifications.
• Better tools to structure/organize content show up in the improved user interface. For example, clicking "Content" in the toolbar takes you straight to a list of content on your site, whereas that took at least two clicks in Drupal 6. Other changes, such as fields in core, affect content management. But there's nothing really groundbreaking here.
• Better performance: As Dries stated at DrupalCon San Francisco (and was reported elsewhere), Drupal 7 is actually slower than Drupal 6. On the other hand, changes to caching and storage make it far more scalable, so perhaps for large sites performance is "better".

Not Done

• Basic Views like module: Not in core. I'm sure it'll be in Acquia Drupal, and a SimpleViews-like module will be in Drupal Gardens.

???

• Improved node access system. I haven't seen any evidence of this. Could anyone comment?

"Scratch your own itch," they say. When I couldn't find a version of the DrupalCon session schedule in iCal format, I converted the Excel schedule created by Lullabot's Kent Bye. How? Using Drupal, of course!

First, the good stuff: here's the iCal file; here's the raw text file.

As with most migrations, this was an iterative process. I tried it once, failed, tried again, failed. Rinse, lather, repeat. For the curious, here's how it went down:

1. Prepare the Excel file for import. Many changes were necessary. For example, the original Excel file gives the date in one row, then lists all the sessions for that day below it. Since each row would become a node, the date information had to be in every row.
2. Save it as a tab-delimited file. Munge that file further. Excel put a quotation mark around every field that contained certain punctuation. If not removed, those extraneous quotation marks would be visible in the final product.
3. Create a custom content type and calendar (with iCal feed) to hold the information in Drupal. Karen Stevenson's excellent Date and Calendar modules made this easy, particularly by using the "Date Tools" module that comes with the former. (Both are part of Acquia Drupal.)
4. Import via the Node Import module. This module required a patch to correctly import dates.
5. Enable and edit the Calendar view as needed.
6. View the calendar and test exports in iCal. We're really close now! This is where flaws in the process became obvious: For example, I had exported early morning times as "9:45" instead of "09:45", causing such events to fail. That meant going back to Step 1 to change how the data was stored in the Excel file.
7. When ready, edit the iCal export with a text editor. The export "HTMLified" title text, so that "&" came out like "& amp;".
8. Publish, accept the adulation of a grateful nation. ;)

Note that this calendar, like the Excel file it came from, only includes session starting times: It doesn't say how long sessions last. I take no responsibility for errors or omissions. Caveat lector. And alas, it doesn't include many of the non-session events, such as exhibit hall hours or (sob!) parties. Just follow the throngs of excited Drupalistas for those. See you there!

View Good cheapish eats near DrupalCon SF 2010 in a larger map

I once heard that San Francisco has so many restaurants that the entire population could eat out and there would still be empty seats. That's probably apocryphal, but it does have an embarrassment of gastronomic riches. The other truth about San Francisco is that it's expensive, so eating lunch for under $8 can be a challenge -- especially near a touristy area such as Moscone Convention Center.

But I lived there for 17 years before moving to Ohio last April and, being a cheap-ass, have collected a few favorites. They're in an annotated short list in this Google Map, "Good cheapish eats near DrupalCon SF 2010". In brief:

Jollibee (4th Street at Howard) is the fastest and cheapest.

Sushi Club is surprisingly fast and cheap, and has take-out.

Tu Lan is the best for the money, but a bit far.

To add your own favorites, log into your Google account, click "My Maps", then click the Edit button. (Thanks to Shawn DeArmond for the tip.) Please (a) give your own marks a distinctive icon, (b) keep it limited to under-$8 lunch places within a 15-minute walk of the convention center, and (c) don't touch anyone else's marks.

Need a suggestion at the con? Text me at 415-317-1805 and I'll do my best to help.

This con feels like it's shaping up to be a real ground-breaker. I can't wait!

If you're thinking of taking my online seminar, "Setting Up, Customizing Drupal" on 29 March, use the discount code "DRUPALTOMG10" to get 10 percent off! Here's a direct link to get the discount.

The discount code also made me realize: You can't spell Tom Geller without OMG!

I'm pleased to announce that I'll be teaching my first live telecourse on Monday, 29 March, "Setting Up, Customizing Drupal", for Environments for Humans. My three-hour course is in the morning, with Sheena Donnelly teaching "Drupal Theme Building" in the afternoon.

Most Drupal people know me for my Lynda.com video courses. I'm no stranger to live teaching: One of my first jobs was teaching secretaries how to use Radio Shack TRS-80 (!) computers when I was 16. (Yes, I'm that old. ;) ) I presented at a lot of tech conferences during the boom, and later taught real estate courses for City College of San Francisco and a private company.

But I'd put off teaching Drupal courses live. I fell into a trap a lot of folks in the Drupal community are in: Because so many of us are highly technical developers and sysadmins, I figured the market for beginner's courses wasn't that big. The success of Drupal Essential Training gave me an inkling that it's bigger than I thought; the entry of such companies as Lullabot into the field, and live training by such excellent video providers as Sean Effel (DrupalTherapy) convinced me. So when Environments for Humans' Christopher Schmitt approached me with a specific proposal, it was easy to say "yes".

Watch for another blog post before the date -- including a way to win two free tickets. And I'll try to report on how it went afterwards. Spread the word!

A recent CNet article notes that such shops as AF83 have been turning away business because they can't keep up with demand. That's a familiar story to many of us, including me: Drupal is just growing and growing, and we're reaping the benefits and challenges.

But consider the other side of that coin, expressed by the article's title: "Need a job? Learn Drupal." If the message gets through, the Drupal community will experience a wave of people driven by practical matters of employment. A few minutes in Drupaldom's current hangouts -- IRC, drupal.org, mailing lists -- predicts how such an influx will clash with the existing culture.

Not that the the Drupal world isn't already commercial and entrepreneurial -- it is, in large part thanks to pioneering companies like Chapter Three* and Dries' own openness to commerce. But the three badges of Drupal honor today are that you (1) you code, (2) you work on GPL projects, and (3) you're active in "the community". Few people responding to the call of this article -- or of the business community at large -- will meet these criteria. Let's look at each separately.

• New immigrants will not be coders. Coders were necessary to Drupal as hunter/trappers were to U.S. expansion. And, like trappers, they're not as important as they used to be. After food sources are secure, people need banking, commerce, clothing, entertainment. These are institutions that pioneers are not equipped to provide.
  
  Obviously, trapping isn't as important a skill now as it was in 1800. We still need food and warm clothing, so the functions formerly served by trappers are now served by others. Trappers can be angry at how they -- the people who built this country! -- have been pushed aside. (Old westerns are full of such grizzled characters.) The smart ones will get off their laurels and adapt to inevitable change.
  
  New Drupal workers will be in public relations, finance, advertising, distribution, sales, business relations, and content. They'll think inheritancy and encapsulation are about wills and pills. They'll fail to recognize coding intelligence, because they're not optimized for such wisdom. In my experience, coders repay that ignorance with a vengeance, failing to recognize the intelligence of those "soft" skills even more. But they'll make Drupal's banks, markets, stores, and bars run -- regardless of how you feel about them.
  
  
• New immigrants will not work on -- or care about -- GPL projects. They're here for a job, not a philosophy. The first ones will become educated about open source because they'll have to be in order to get along with the community. But peer pressure will shrink as the pool of Drupal users grows. We've already seen this in (for example) the Linux world: How many users understand their operating system's origin or license? 1%? How many contribute back to the project? 0.001%, maybe.
  
  Which leads to a hard question: Does Open Source Matter? On some level, yes, and I spent a good part of the late '90s expounding the position that it does. But for the people building a career based on using Drupal? No, it doesn't. They hope that Drupal remains strong, and perhaps have a vague idea that volunteers are behind it. Because they're not coders, they won't have any connection to those volunteers -- unless the Drupal project changes in ways that make it easier for them to get involved.
  
  That, of course, has been a topic of much discussion. The Drupal.org redesign will help, but it's only a drop in the bucket. Ultimately no amount effort will entice the majority of new Drupal users to get involved.
  
  
• New immigrants will not be in "the community". This will be the hardest blow to Drupalistas who have been with the project since Dries was a jongetje. It strikes at the real reason that people contribute their efforts to Drupal, or any cooperative project: Because they like the people as much as the subject.
  
  When a group is small, members feel they can know everybody, and problems can be solved via IM. Even if they don't know everybody, they feel they can at least trust others, and that they'll share common beliefs.
  
  But growth engenders diversity. I remember being part of a pretty insular bisexual activist community in the early '90s, all working together for the recognition and dignity of Our Sort. We started interacting with some counterparts from another city and found them... tacky. Suspicious. Poor representatives of what we thought We were. We had gone beyond our tribe, and not liked what we saw.
  
  So it will be with Drupal. One thing that's surprised me is how little we hear of Drupal being used for right-wing political sites. Will our community, with such left-wing support businesses as Development Seed and Chapter Three prominent among us, trumpet their success as well?
  
  

So -- the contrast between "old guard" and "new school" may sound harsh, but it's actually cause for celebration. If Drupal does in fact attract such people -- who don't code, who aren't GPL-savvy, and who aren't community members -- it'll be a sure sign that it's escaped its corral into a larger world. And as we can take advantage of their skills, the circle will continue to be ever-widening.

* Say, who were the first Drupal service companies? I'm assuming Chapter Three was one of them because of its leadership's involvement in Deanspace.

Here's a statistic I haven't seen bandied about much. Drupal runs three times as many of the Alexa 10,000 top sites as the next CMS, according to backendbattles.com.

This figure was extracted using Wappalyzer. It's not perfect: The Onion doesn't show up as a Drupal site, for example, probably because it runs an old or heavily hacked version.

It's nice to have this statistic at hand when Drupal supporters feel disheartened by popularity comparisons to Joomla. There are many ways to be successful; capturing the attention of the world's biggest sites is a pretty good one.

Having said that: WordPress is found on nearly three times as many big sites as Drupal. (Backendbattles.com categorizes it as "blog software" rather than a CMS, so it doesn't show up in the comparison.)

Peachpit just published my article, "Top Ten Changes That Make Drupal 7 the Best Version". Peachpit is a publisher of a wide range of technology books, so this article will reach many readers who are savvy enough to use Drupal well, but might not know much about it yet. Comments welcome!

Just want to point people at a terrific new resource maintained by a "Seattle Drupal Beginner" group: a list of Drupal-related books, with bonus links to the authors' Drupal.org profiles! Nicely done, folks.