Carton's Content Blog

MoDevDC 2/1/2012

2012-02-07T09:05:00.003-05:00

I went to MoDevDC meetup last Wednesday night - two subjects were presented and both were (for me) very interesting. http://www.meetup.com/modevdc/events/48923902/?a=ea1_grp&rv=ea1

First was a presentation on PhoneGap by Andrew Glover of App47.
I had been fooling with PhoneGap for a while now - mostly to see just how easy it would be to create a multi-phone app. I had not moved beyond the "HelloWorld" stage, but had gotten that deployed on both a Droid and a Blackberry. I figure that the iXX wouldn't be any harder. So I was very interested in this presentation. Here were my take-aways:

PhoneGap is a framework for creating multi-platform apps.
It is designed for creating apps using javascript, HTML5 and CSS3.
It provides a way to create a "shim" (my word) for leveraging a webkit browser on any mobile device. In the iXX world, this happens fairly automajically. In the Droid and other mobile worlds, I still have some file organization and a little coding to do.
PhoneGap provides a seamless javascript API to some of the phone's features, such as the GPS, camera, accelerometer, and so on, so that my app will work across devices without having to change the way I access those phone functions.
In addition to PhoneGap, I need a decent mobile javascript framework for my app. There are several such frameworks:
- jQueryMobile
- Sencha Touch
- Appcelerator

Second was on HTML5 Caching by Ryan McGeary of BusyConf.
This was a very interesting discussion of the caching capabilities of HTML5 and how to use them to create an HTML app that could be downloaded and then used when offline. Most of the discussion was on the contents and syntax of the cache file and some of the "gotchas" that come from that. The material Ryan presented is from http://diveintohtml5.info/

Verizon, Thunderbolt, Network Connectivity

2012-02-02T11:10:00.003-05:00

I have an HTC Thunderbolt and I've been having problems lately maintaining a connection to Verizon's broadband network. After reading a few comments on this, it would seem that the issue has to do with 3G/4G network availability (someone please correct me if I'm wrong). I found this suggestion:

Enter the following from your dial pad- *#*#4636#*#*

Then open the phone information- scroll down and change from CDMA + LTE/EvDo auto to CDMA auto (PRL)

Well - this seems to have done the trick for me.

Windows 7 Symlinks

2011-12-12T12:11:00.003-05:00

I'm finally realizing (maybe the last person to cotton on to this), that starting with Windows Vista, we now have the capability to create Unix-style symlinks between Windows files and folders. YIPPEE!

It's done with the mklink command (from the cmd window):

mklink
Creates a symbolic link.

MKLINK [[/D] | [/H] | [/J]] Link Target

        /D      Creates a directory symbolic link.  Default is a file
                symbolic link.
        /H      Creates a hard link instead of a symbolic link.
        /J      Creates a Directory Junction.
        Link    specifies the new symbolic link name.
        Target  specifies the path (relative or absolute) that the new link
                refers to.

In my particular case, I'm experimenting with the Sencha Touch mobile application development kit. Creating the simple "Hello World" app as a place to start. This requires that Sencha SDK. I downloaded that and would prefer not to create a copy of it in my app server. Symlinks to the rescue. Since my Hello World app makes reference to the SDK as <script src="lib/touch/sencha-touch.js" type="text/javascript"/>, I simply created a lib directory in my app folder and then made a symlink to the Sencha Touch folder:

mkdir lib
cd lib
mklink /D touch ..\..\sencha

And voila - no copying.

Thanks Steve Jobs

2011-10-06T11:03:00.002-04:00

Even though I was expecting it, I was saddened to hear that Steve Jobs had passed away last night. I'm not going to repeat all of the fantastic things he accomplished in his short lifetime. Nor the incredible vision he brought and left as an indelible mark on so many areas of technology.

I just want to say "Thank you Steve, you will be missed".

Ubuntu/VirtualBox/Java Team Development Stack - Part II

2011-10-04T16:28:00.000-04:00

Part II - Install the Development Stack.
Continued from Part I

Having a virtual machine with Ubuntu installed, we now need the developer tools in our stack.

My target developer's stack has:

Java JDK
Eclipse
Ant
jEdit
Firefox (or Chrome)
Tomcat
MySQL
MySQL Client
DropBox

It's worth installing Ubuntu Tweak. At the least, this gives us the window icons on the upper-right corner. Open a browser and go to http://ubuntu-tweak.com/. Install that.From within that, I also found the Chrome browser install and did that.

The approach Ubuntu uses for typical software installation is through the "Ubuntu Software Center" which is invoked from the bottom of the Applications menu. Easiest is to use the search box to find the software we want and then choose the version (if there are options) to install. I simply searched for and installed each package. Some notes:

Java JDK: I chose the "OpenJDK Development Kit".
Tomcat is installed on port 8080. Something else must be running on port 80 so I wasn't able to change the tomcat port.
Tomcat installs as a service. As such, it can be stopped and started: service tomcat6 stop|start|restart
I added a manager role and user ID/Password so I can access the tomcat manager and use the Eclipse/Ant deploy tasks.
I created a new folder in home: /home/projects and made it read/write/execute accessible to the users group.
When I installed dropbox (by going to the www.dropbox.com website and installing from there), I pointed Dropbox at the projects directory so that those files would be under /projects.

Ubuntu/VirtualBox/Java Team Development Stack - Part I

2011-10-04T10:27:00.004-04:00

Part I - Setting up a Virtual Machine and Ubuntu

A controlled java web development stack for a distributed development team. I did this a few times before but never documented it, and (of course) versions of the tools have all moved on. But I have to resurrect it, so this time I think I'll write it up. No where near as easy as Dave Winer's EC2 for Poets (http://poets.scripting.com/), but he's a much better writer than am I!

So, the idea is this: several developers; some co-located and others not; working on a web project being developed in Java. This project is also using MySQL for an underlying database. Code is managed in a remote CVS repository. What I want to create is a controlled environment in which each developer has the same set of tools and test data against which to work. The answer: a virtual machine with all the tools installed and configured so that I can simply deliver a copy of the VM disk image to each developer. I've played with this before in VMWare and some other virtual machine kits that I don't recall, but most recently in Oracle's VirtualBox. So I'm repeating that experience here. It bears noting that I am doing this stack creation on a Windows Vista 32-bit laptop (no flaming please). But I believe the overall approach here would work on pretty much any host machine. Also of note: since I don't want to deal with licenses (and fees) for virtual machines that I send out to developers, I am using Linux Ubuntu as the virtual client. This being a java development project, what we develop is (or certainly should be) portable across different operating systems, but the production target for this is also Linux, though not Ubuntu.

My target developer's stack has:

Java JDK
Eclipse
Ant
jEdit
Firefox (or Chrome)
Tomcat
MySQL
A VPN connection for CVS

In addition, there needs to be a way to share files from the host machine to the virtual machine. Here are the rough steps I followed:

A. Create an ISO Image of Ubuntu:

Download Ubuntu (http://www.ubuntu.com/download/ubuntu/download).
The ISO image will need to be available to VirtualBox for installation, either on a CD or a USB drive. I use a USB drive. Just copy the ISO image to that.

B. Set up a Virtual Machine with Ubuntu:

VirtualBox: https://www.virtualbox.org/wiki/Downloads. This is simple enough to install - just follow the steps and accept the defaults. Once it is installed, start it and create a new virtual machine.
New Virtual Machine: I chose Linux/Ubuntu and then set memory to 1Gb.
New "Hard Disk": I set it to dynamically expanding.
At this point, I'm done creating the virtual machine - I need to install Ubuntu. By clicking "start" on the newly created machine, I get the "first run" wizard. From there, point to the USB drive from A. above.
Follow the setup for Ubuntu.

C. Create a root Password.

By default, Ubuntu is installed without a root login. I'm guessing this is for security reasons. But these developer VMs are not really subject to security issues and having the ability to run as root makes a lot of things easier. So I add a root password, enabling me to su and run as root when I need.

sudo passwd root

First, you'll have to enter your password. Then the password you want for root.

D. Add my ID to the "users" group.
I found that my user ID was in its own group. I added "users" as my group so that I can access common folders more readily.

E. Install Guest Additions

This is a big/important key step. It makes many of the hardware and device features of the host computer pass through to the virtual machine. Like the screen size/resolution. And others.

There is a virtual machine menu bar with a "devices" menu. This has a "Install Guest Additions..." item. Clicking this will *not* actually install the guest additions; it only mounts the guest additions as a virtual drive. So I did this, but then still had to run the actual install.
After the mount, open a terminal window and cd to the mount for the additions. These seem to be under /media.
Run the autorun.sh (sh ./autorun.sh). I switch to root before doing this (su).
Restart the server.

F. Ensure Networking Works.
Since we need to access CVS over a VPN, we need the VPN access to work. For our purposes, each developer has VPN access from the host machine (my laptop). So I want this to "pass through" to the virtual machine. I'm not a networking guy, so I don't know much about this, but it seems that by setting the network adapter in the virtual machine to "NAT" (which seems to be the default anyway), I go the pass-through I wanted. Perhaps someone more knowledgeable than me can elucidate? Anyway, I simply tested this by connecting to the VPN and pinging a machine on our internal network.

G. Set up Sharing.
We'll want to be able to share files from the host file system to the virtual machine file system. This means setting up a Shared folder.

On the host machine, create a folder (I called it "Sharing" and located it in the root).
In the Virtual Machine, create a permanent share - from the "Devices" menu, select "Shared Folders..." and create a new share. I created a "permanent", "Auto Mounted" share and named it Sharing.
After restarting the VM, this share is now found in /media/sf_Sharing.
I had to change my user ID and add "vboxsf" as a group to be able to access this.

Continued: Part II - Install the Development Stack.

Publishing and Mobile Devices

2011-09-29T12:00:00.001-04:00

I've been interested for a long time in writing applications that will provide some useful service on a mobile phone, and with the advent of smart-phones I get to explore this a little more. This got me to thinking about apps that run on smartphones and that are possibly useful to information customers - apps that a publisher might develop and deploy to their subscribers. And since I have a new Droid phone, I started doing some programming for that. But it begs an interesting question - what phone(s) should a publisher target in developing a new app?

Of course, this is heavily informed by the particular publishing sector and by the kind of information being published. Phones, with a small-ish form factor, will favor small units of data; pads would more readily support larger. In fact, the form factor on a pad (like the iPad) is such that simply using a browser in the pad to access the same publisher web resource that one would access via a computer is usually an acceptable user experience. So, if I were a publisher, trying to decide what phone to develop an app for first, what factors might influence my decision? Here's what's floating around in my head:

Hard to ignore the overwhelming presence of the iPhone. Huge market penetration. But small form-factor. And it just plain pisses me off that I have to go through the iTunes marketplace and pay Apple a pound of flesh. It's my data after all! And I suspect most publishers are not really selling the app; they're selling content.
Hard also to ignore the skyrocketing increase in iPads. Seems like everyone has one these days! Wish I did. And, given that the iPad runs the Safari browser (which is Webkit compliant), I can develop a variation of my website using an HTML5/CSS3 approach, host it on my own servers and not have to go through iTunes/Apple. More on Sencha/jQueryMobile/PhoneGap in another post... But if I need to create something that runs on the iPad, I run into the same iTunes/Apple question.
Some of the stuff I'm reading indicates that the various Droid phones are increasing market share considerably. And I can develop an app for a Droid phone and distribute it myself without going through the Google market or, if I choose, I can go that route.
It seems that in certain market segments Blackberry is ubiquitous. Especially in the legal and lobbying worlds. As with the Droids, I like the fact that I can deploy an App frmo my own sites and am not forced to go through a marketplace. The small form-factor makes the BB somewhat limited for larger units of data and the differences between versions of BBs and the attendant differences in API support make BB develop somewhat challenging, although a cross-platform approach such as HTML5/CSS3/PhoneGap may limit those difficulties. OTOH, RIM is trying to shift to QNX and allow Droid apps to run on Blackberries. But when? And how long will it take for that upgrade to penetrate the marketplace. And a lot of the stuff I read indicates that RIM is losing market share.
And then there Microsoft. I sometimes think that if MS creates a phone that integrates seamlessly with their back-office servers (Exchange, Sharepoint, etc.), those would become the darlings of IT departments worldwide. Enough so that they could drive RIM out of that spot. But that hasn't happened and MS phones are distant cousins in terms of market share.
Kindle Fire? I love the idea and the price, but it means limiting app development to Android 2.2. We'll see!

So, I think I still have more questions than answers. And I know that some of this is a religious argument - I hear fans of the droids espousing their virtues over the iPhones and vice-versa.

Android Thunderbolt and Verizon

2011-09-28T11:26:00.000-04:00

So, my ancient cell phone having started death throws, I went to my Verizon store (with whom I've had service for years) and started looking at "smart" phones. I was particularly interested in a droid phone as opposed to an iPhone or a blackberry. I'm suspicious that if RIM can't make a big splash with QNX next year, they're history. And I really want to do some app development but the idea of being forced into the Apple iTunes marketplace just pisses me off. So I went with a Droid phone. It helps that App development for Droids is in Java and although I like to talk about developers being language agnostic, my most extensive and recent experience is in Java.And as I looked at them, I ended up buying the Thunderbolt. Mostly because the screen size is bigger than the other non-pad phones.

So now I have a Thunderbold and a brand-new 2-year contract for voice and data from Verizon. I have some minor gripes about the phone. I think the touchpad is too sensitive for my fat fingers and I'm always mis-dialing. The blue-tooth voice recognition is a continual source of entertainment and not accurate for me at all. And the battery life just plain sucks (though the salesman did warn me about that). But my biggest gripes are a) that the phone comes preloaded with a pile of app that I have no interest in, and b) that connecting the phone to my laptop causes a pile of messages about installing the Verizon media application on my laptop.

The bloated list of installed apps is particularly troublesome. I can't un-install them in any normal way. I'm told by my (many) nerd friends that I need to "root" my phone and then, as super-user, I can remove the apps I don't want. Well, it may come to that. But I don't feel like I should have to hack my phone to make it into the device I want. And the idea that Verizon is intentionally burning my data plan so that all these apps can "phone home" regularly is seriously insulting and may in the end cause me to switch carriers.

The Verizon Media application is also irksome, though a problem that I could solve. After getting warnings and requests to install this every time I plugged my droid into my laptop to charge it, I went ahead and let it install. After that, it inconveniently launches when I log in. I have enough crap on my laptop that starts when I log in - it already takes too long! So after it installed, I went out on the web and found a Windows (yes, my laptop runs Windows, Vista no less) NO-OP program (http://joenord.com/apps/nop/index.html). Essentially a program that does nothing. I then copied that into the media center install directory, renamed the media center (verizon.exe) and then renamed the nop.exe to verizon.exe. Voila, it doesn't launch anymore.

Next up - rooting the phone.

Starting Again

2011-09-28T11:06:00.001-04:00

It's been a while since I posted anything - I went for a long period without feeling like I had much to say. Then I realized that part of the problem is that I was feeling constrained to write specifically about content management. And about that, I really don't have much to say that other aren't saying more eloquently.

So I'm starting anew, with a more general technology focus. I'm heavily influenced by reading other folks blogs so, in true copy-cat style, I'm plagiarizing their concepts and opening this up to just about anything that hits me as worthy of jotting down. Since almost all of my tech work is in the publishing world and since my interests tend to span not only content harvesting, management and delivery but also technologies in the "active" world (as in personal sports), I'll probably offer up misguided observations across all of those boundaries.

And since I'm doing this more for personal catharsis than for either marketing or networking reasons, I can safely say that I don't really care who reads this or if they like or agree with what I write. But if I'm wrong (often) or if there are better ways (oftener), please comment and let me know.

Kindle, eBooks, Others

2010-12-16T11:59:00.004-05:00

I think the Kindle is cool. Easy to use and very easy to publish to. Lots of options in this arena though. Sony has a player as do several of the major book sellers. Of course, formats are different and we're all looking to see who will win the new betamax/VHS war here. And there are some organizations that will adapt content for any/all of these platforms. But when we look at all the options, it seems like the EPUB (http://wiki.mobileread.com/wiki/EPub) format has the best coverage of all.

http://labs.oreilly.com/2009/03/epub-resources-and-guides.html

Delivering Integrated Information from Data Silos Using MODS

2010-01-14T14:52:00.004-05:00

Lyn Robinson of the Burton Group wrote an excellent paper describing a "Methodology for Overcoming Data Silos" (MODS).; a "groundbreaking project structure" for bridging silos and delivering integrated information from decentralized, disparate information systems.

Interesting concept. A lot of thought, time and money has gone into breaking down information silo's over the past few years. Robinson recognizes that this is frequently unsuccessful or, at best temporary, but that the need for integrated information remains very important - even mission critical. So instead of re-engineering the underlying information systems into a single integrated replacement, MODS proposes to implement an information bridge that fills the need for integrated information from across these silos.

So how does it work? "Each MODS project results in the definition of a few relevant and important enterprise data types, as well as a registry that tracks and reconciles the instances of those data types in the disparate silos throughout the enterprise. This registry functions somewhat like the old card catalogs for books in a library. Using a MODS registry, businesspeople can discover where to find particular types as well as instances of relevant data within the disparate silos scattered all over the enterprise." And, it's supposed to be cheap (well, cheap in comparison to development projects).

Wow. Cataloging data - maybe a librarian should be called in! Robinson describes a typical MODS pattern as "establishing a registry or system of record for a particular set of closely related enterprise data types. The scope of these data types should be determined by a conceptual data model, which defines the needs the business has for information regarding a particular business topic. The best MODS projects are narrow in scope, so the more specific the business topic, the better". Here are the steps in the pattern:

Model the relevant data.
Clean and Understand the Relevant Data
Classify
Convey
Control

As I read this approach, I got to thinking, this is a possible application best served by a TopicMap. After all, TopicMaps were originally developed for indexes or catalogs. It seems like a natural fit. TopicMaps can implement many of the features MODS calls for: data typing, data relationships and styled queries. And I also began to think about social networking features like folksonomies and how these might apply to MODS.

A lot of my personal focus is related to content management and publishing. As I consider this, Content Silos are just as prevalent in publishing companies as other information type silos are in other organizations. So MODS may well have a place in content management.

So, in the next several blog posts I'm going to explore TopicsMaps and perhaps Folksonomies as technology solutions for implementing a MODS project for content in publishing.

Stay tuned...

RSuite User Conference - 10/6/2009

2009-10-06T08:58:00.004-04:00

I'm attending the 3rd RSuite users conference in Philly today, by invitation of Really Strategies' Lisa Bos. I've been tracking RSuite since its inception a few years ago - the idea of a decent CMS based on Marklogic always hit me as a good niche product and by all indicators, RSuite is a success, especially in light of how many CMS offerings there are in the world.

I'm here for the technical track - to learn more about how to write customizations for RSuite - I'm a developer afterall. But the Keynote from Howard Ratner (CTO, Nature) is an excellent intro to what comes after CMS.

Nature is using RSuite as an integration tool between various pub tools and the Marklogic content repository. In the Keynote, Howard makes excellent points about the need for content classification, which (of course) is why I'm writing about it! Nature is using a mixture of automated content recognition tools and editorial tools. Also mentioned open search technology - search widgets that create a viral access to the Nature content. One very interesting thing Howard talks about it the "Work Arounds" that are developed because either the content or the technologies don't do the exact job needed. What is the cost of these?

Lisa Bos then gave us a demo of the current version of RSuite. Nice clean and simple to use browser-based GUI. New features include Digital Asset Management, Workflow Configuration, editing (in oXygen and XOpus - one of my fav XML editors). But, IMHO, where's the beef? What does this have that free/open-source solutions (like Alfresco and/or eXist) don't? Or, for that matter, our own Tractare? Other new features - search enhancements, reporting, GUI customizations, better API and the ever needed performance improvements.

Semantic-Content Management

2008-09-19T16:09:00.007-04:00

We've been developing a tool set, or framework, or whatever for several years. We call it "Tractare" which is Latin for "to handle, manage, perform". I'm a sucker for that kind of naming.

Anyway, what's it all about? Well, as the Internet becomes more saturated with raw information, keyword search engines really aren't enough to locate that needle in the haystack. We (content providers) need to describe our content in a way that users "get" -- we need to describe the "aboutness" of our content.

An example I like to use when speaking on this subject is this: If you were to use google to search for "retarded", you would get a gazillion hits. But few, if any of those hits would have come up using the politically correct phrase "intellectually challenged". This is because a keyword engine like google depends on the actual presence of the keyword, either as text in the content or as metadata. Now, you could encode both forms of this concept as meta-data on your web-page and it would be found. Now, if you are a psychologist or someone working in mental health, you'd probably be getting the results you want. But if you are a firefighter, the word "retarded" has a whole different meaning. How do we express that? The answer is in several parts of course. But first, we need to capture the meaning of the content; the "aboutness". We need to associate the "firefighter" concept with the content that pertains to fighting fires. This is what Tractare permits us to do.

Tracare is a framework. It's not an off-the-shelf product. It is built on the idea of topic maps -- organizing content around indexes and concepts. It's true power lies in a combination of searching and navigation tools that allow the user to narrow the scope of their work to a set of concepts. We build custom CMS and delivery solutions on top of it.

The CMS systems we build usually include features found in social networking, including folksonomies (as well as traditional taxonomy and classification support) and ranking/commenting. These features allow content providers to apply semantics to content in a number of new and different ways.

The delivery systems we build often include a number of search and navigation interfaces that web users have come to love, including mashups, classification searching, semantic browsing and so on.

Advertising-Based Publishing

2008-09-09T10:34:00.008-04:00

In a previous post, I was exploring new trends in electronic publishing (April 30, 2008). One of the key points was that traditional subscription and publishing revenues are shifting away from subscription fees towards "free" information supported by advertising revenues.

"One well known revenue shift is towards advertising revenues. This is an old model, of course. Newspapers have been at it forever, and online search engines for years. But more recently, we're seeing a larger shift of revenue away from subscription sales for information resources and towards advertising."

I had this reinforced recently when I realized that even traditional reference book publishers such as dictionaries and encyclopedias were increasingly garnering revenue from advertising rather than from subscription sales. Wow -- that was an eye-opener. It makes sense though -- if I'm looking for an informative article on "Chocolate", am I going to pay for access to Britannica or would I go to the free Wikipedia? I guess the answer (for me at least) depends on how authoritative my answer needs to be. But in general, I would go to the free site. And I suspect I'm not alone. So how is a traditional reference publisher to compete in the age of Wiki-whatever? Product quality alone isn't enough. It has to be free too. Enter advertising.

On the surface of it, it seems easy to generate advertising revenues. Especially if your area of publishing is targeted - in fact, the more specialized your content, the more valuable you are to advertisers? I'm not sure that this is true, but it sure looks that way. Virtually anyone can establish a Google AdSense account and tie it into their content publishing operation (as I've done to this blog). But who's actually making money at this? That's the hard question.

It always comes back to the same basic principal - supply and demand. I loosely translate the supply side to "timely, quality content". Timely doesn't necessarily mean frequent, it means "frequent enough". And demand is partly driven by the content and partly by your sales/marketing operation. Demand is partly the number of visitors to your web site and partly by the number of visitors who pay attention to your advertising. So, we need quality content and we need to advertise its presence.

These sound like the principals as those by which we've been driven forever.

Gilbane, San Francisco, 2008

2008-07-03T10:00:00.003-04:00

I spoke at the Gilbane Conference in San Francisco in June (http://gilbanesf.com/) on the subject of using folksonomies as a publishers tool for classifying content. The session moderator had warned me that attendance was a little thin and that we shouldn't expect too many in the audience, but my rough estimate was that we had about 45 people there. I was pleased. And there were lots of questions and a lot of discussion.

Here's the abstract from my presentation:

Folksonomies, Just Good Enough For All Kinds of Things
Extending Folksonomies to Describe All Kinds of Content

Folksonomies are gaining popularity in the content delivery world as a "good enough" way to classify content; tag clouds are becoming commonplace on content-driven website. More and more, folksonomies are working their way back into the content creation and management process. Huge volumes of content, increasing and diversifying user requirements are pushing content management operations to look to folksonomies as a Web-2.0 way to describe the "aboutness" their content.

With this simple concept in hand, why not use this technology to describe more than just content subjects? "Aboutness" is just one aspect of your content. What about order, threads, or related content? This talk will explore the extended use of folksonomies as a technology for enhancing content in new and different ways.

I can send the power point to anyone interested, please email me or leave a comment on this blog.

New Trends in Electronic Publishing

2008-04-30T14:36:00.010-04:00

I'm a consultant (sounds like an admission) in the publishing industry and throughout my career I've developed and integrated new technologies to help publishers stay competitive. Everything from back-room content management to user-facing "wow" tools. And, I'm noticing some trends in the publishing business lately. Especially in electronic publishing. These are some things I'm observing from my clients specifically, and from public media (annual reports, news, and so on). I think we're seeing a big shift in the way publishers generate revenue, away from the value on information.

One well known revenue shift is towards advertising revenues. This is an old model, of course. Newspapers have been at it forever, and online search engines for years. But more recently, we're seeing a larger shift of revenue away from subscription sales for information resources and towards advertising. I think this is driven by a couple of factors.

First is the nature of the internet and social networking. These are creating an underlying expectation of free information on the web. No news there! But this is creating new revenue streams in the form of advertising that are replacing traditional subscription revenues. And with tools like Google's AdSense, this is easier than ever to do, making it possible for just about anyone to create an ad-driven information portal.

The second factor is the rise in information portals repurposing and rebranding content for their own purposes. Tools like RSS and aggregators like Feedburner make it easy to brand your own content, creating a demand for and an expectation of specialized information sources. In a sense, this is dividing the traditional publishing industry into two parts -- information creation and information dissemination. Publishers used to integrate both these functions under one roof, but in the repurposing/rebranding scenario lines between these functions become clear. New revenue streams are being created and recognized by selling content for aggregation, repurposing and rebranding.

Related to this is the increasing role public entities are playing in the information dissemination arena. A large market used to exist for private publishing of government information. In the old days, these publishers took government data and added value through indexing or other finding aids and sold the improved access. As technology tools have improved, government agencies are more easily and readily able to perform this publishing themselves. So adding content value has become more important to the ability to create revenue than using technology to improve access.

And this is exacerbated by the economy. With more choices and less budget, customers will opt for the cheapest product that reduces their workload. User expectations are indeed greater. Users now expect to access content with minimum wasted time and maximum accuracy. To stay competitive, publishers need more sophisticated user-facing technology to meet these expectations. In turn, new revenue streams are coming from sales of integrated technology and content or from the sale of the technology itself (software products).

So, do I have a conclusion from this? Not really. I don't think anyone can accurately predict the course of publishing over the next few years. I do think publishers need to keep their tools sharp. My personal belief is that investments in the right technology will win out in the end, but mostly because it will enable the publisher to be ready to respond to changes more rapidly. Of course, I'm a techy saying this, so my view is slanted. I'm reminded of a joke by Emo Phillips: "I used to think the brain was the most important organ in the human body, then I realized what was telling me this!" But I can't help thinking that technology investment, especially in these tough economic times will out.

Getting Some Real Offshore Experience, Redux

2008-04-30T14:24:00.003-04:00

I posted about getting some real experience working with an offshore development group. As I said, we entered into this arrangement with a short-term trial to see how it would work out. We ended that trail today and I guess I still have mixed results to report.

Working with an offshore team in an Agile world is annoying. A lot of time is lost because the offshore developers are not co-located (even within the same time zone). I would say that my initial conclusions about the TCO for code developed still holds -- about he same or even more than the cost of doing it with our regular development team.

We had some problems with the develop initially assigned to work with us. To their credit, the offshore company took care of that, replacing the one with a more mature/professional developer. We lost some time and money in that transition, but the startup cost of this kind of relationship cannot be underestimated.

I guess the conclusion I've come to is this, write good specs and way overestimate the cost. But the skills are there. I will work with this group again (probably). Next project we get.

Getting Some Real Offshore Experience

2008-04-12T20:26:00.010-04:00

In a former post (see "Offshore Development Woes"), I wrote some personal conclusions from reading Martin Fowler on "Using an Agile Software Process with Offshore Development". It was a summary of what I got from Mr. Fowler's experiences. Interestingly, though I have worked with many developers from many Asian countries over the years, I have never actually worked with an offshore software development company performing development for hire under my direction. It's easy to be an armchair critic.

But that's changed, at least a little. Faced with a recent influx of development projects and a lack of staff to get them done in a timely manner, I decided to develop a relationship with a firm based in India. Being the cautious type I am, our initial foray into this arrangement was for a limited time and scope -- I really wanted to see how well it would work out. And I was especially intrigued given Mr. Fowler's experiences because we too are an agile development shop. I had misgivings based on the rapidity of our development and the time and language differences. But the hourly rates proffered were downright seductive. I figured that at the $25 per hour rate we were being charged, we could afford to lose some time to these difficulties and still come out ahead. And I really want it to work.

It's now been 7 weeks into this trial and I guess I have mixed results to report. First, as Mr. Fowler observed, there is no real cost savings in spite of the seductive hourly rate. Between the language barriers, the time differences, things just plain take longer. A lot longer. And that costs. It's not like the developer in India isn't charging me for those differences. We use email and Skype to get our communication done since written English at least removes accents from the equation. And our developers in India work odd hours, so in spite of the 9.5 hour time difference, I can usually talk to them throughout the morning.

But I've now worked with the group on three development tasks. The initial effort was to get the existing project development infrastructure (Eclipse, Tomcat, CVS, MySQL, and the project code) up and running. I have to say, I was impressed with how quickly and smoothly that happened. It showed that the Indian team was well acquainted with these tools. So we moved on to actual development tasks. I made an effort to be more clear and concise than I usually am in providing task writeups. We had several email exchanges asking for clarification and I got my first code delivery in about a week. I was very disappointed. The code was no where near complete, was undocumented, did not follow style guidelines, and was generally unproffesional (a java package name of "jesus.xml"?). But the real problem was the lack of some basic computer science -- a huge switch construct instead of arrays; a single large method instead of encapsulation, and so on. I rewrote the code and sent it back with further instructions on what I was looking for.

To make a long story short(er), this seems to be the pattern -- delivered code is under-par, I rewrite it and return it, and they then follow my pattern. This put the first several tasks way over budget.

We did make a complaint to the management team. Their solution was to add another resource (a more senior programmer) to the team. At their own cost (to their credit). But now, every detail is being questioned and designed out in advance. We've lost our agile approach. And the costs are higher still, since I spend so much time trying figure out the answers to their questions.

So, in all, this effort has proven to be more costly than we had hoped and it pushed our project costs over budget. It may be that a different team from India could perform better, or that if we developed using a traditional "waterfall" life cycle, things would be better. But this has not been a success for us.

Using Folksonomies in Content

2008-03-28T18:11:00.002-04:00

I've been thinking more and more about folksonomies as a replacement for traditional taxonomies. We can create all the tools we need, or work off of existing tools such as del.icio.us. But in the end, size does matter. For a folksonomy to work, a *lot* of people have to look at and tag the content. To me, this means content has to be exposed to readers, and lots of them, to tag the content.

Traditionally, the classification operation has been something done under control and part of back-room content management, by a small, select group of indexers. So first, we have to decide to relinquish some control. While this may seem scary, in reality the volume of tagging makes up for the lack of specific control. And, we can build the tagging tools so that editors and indexers can follow behind the tagging and clean it up. But if we can achieve a large volume of tagging, the volume and repetitive nature of the tagging will create common tags, threads and relationships.

Second, we need to find a large group of taggers. Depending on the nature of the content, this can be accomplished in a couple of ways. First, and perhaps easiest, is to expose content to the web. Perhaps through incentives, taggers can be enticed to tag. And, of course, staff of the publisher should be encouraged to participate as well. Failing that, a publisher could look at a human-automation engine such as Amazon's Mechanical Turk, where large numbers of minuscule tasks that are best done by people are spread out over many people for a small fee.

Jobs (or Maybe Just Job)

2008-03-27T11:53:00.004-04:00

Retrieval Systems Corporation (my employer), needs an experienced developer to serve as a project lead, closing out an ongoing project. The project uses SQL Server, IIS, Exchange/Outlook and VB.NET to track document filings, and enforcement activities through a legal process. We are currently wrapping up the initial data entry and enforcement status components of the application. The next phases will involve integration with Quickbooks and Sharepoint services.

The individual we are looking for will be a professional software developer, with a minimum of three years experience. They will need a solid knowledge of SQL, .net, and developing within the Microsoft web technologies. The job is for a project lead who will be expected to work on all aspects of the project life cycle.

Retrieval Systems Corporation is a developer of custom software. Our developers work in small teams and are involved in most aspects of a project. We are located at 2071 Chain Bridge Rd., Suite 510, Vienna, VA 22182. Work on this project, will be performed on site at our offices.

Agile for Consultant Development Teams

2008-03-20T11:23:00.010-04:00

I'm a consultant (sounds like an admission at the start of a 12-step program). I love the Agile approach to software development but there seems to be a fundamental conundrum I haven't been able to solve -- software project pricing. Seems like my clients want to know, in advance, how much the development of software is going to cost (the nerve of some people!). I just can't see how to get a price together for a project using an approach that essentially leaves out the up-front design effort that is characteristic of the traditional waterfall approach. And the Agile experts with whom I've spoken have no answer to this either.

Now, I don't actually believe that the waterfall approach gives any better or more realistic cost estimate, except perhaps on the most simplistic of projects. If anything, I believe it is a more expensive method that produces a false sense of cost security. But, I'm not getting into a comparison of these approaches. I'm just trying to figure out how to give a customer a price for an Agile project. And I've tried to say things like "what is your budget? We'll do as many of your priorities as possible within that." There is a disconnect between what clients want to know (how much for the whole thing) and the reality of what the "whole" thing needs to be and how much it will really cost.

One approach would be to simply lie. Tall the client what they want to hear. And then manage the consequences in the middle of the project. Doesn't seem like the right way though.

So instead, we've been trying to educate our clients on the tension between cost and features. In essence,
We've been using a variation on Wide-Band-Delphi-Blind to create estimates. We take the full list of features, attempt to designate tasks for implementing them, and then send that around to as many senior developers as we can -- developers with the right skill set to create an estimate for that type of project. Each estimator does his estimate alone and sends the results back to a coordinator and then we go through the estimates collectively to resolve the differences. Using this estimate of all features we then attempt to explain to our client that the estimate is under their control -- choose high priority features to implement first and then work through the list until either features or budget are exhausted.

This is not too different from agile poker (http://www.planningpoker.com/). As I understand that technique, several estimators each make a bid for how long each task or feature will take. The actual bids are not really hours. They are units. After several rounds of bidding and discussion on differences between bids, the winning bids are turned into dollars by applying the development team's velocity to the bid and figuring out real hours. This means that we have to know that velocity -- not an easy problem in its own right.

But we end up in the same place -- with an estimate for the whole project which we then have to "sell" to the client. Is there a better way?

Bill Gates at the NVTC Breakfast Forum

2008-03-13T10:38:00.004-04:00

I got the opportunity to attend the Northern Virginia Technology Council (NVTC) breakfast this morning, in DC at the Capitol Hilton. The speaker was Bill Gates. You know, that Microsoft guy? It was kind of cool to be a couple of hundred feet away from the richest man in the world (or maybe second now that Warren Buffet's star has been rising). Bill spoke at length on the direction computing is headed and mentioned everything from surface computing (your desk working for you?) to intention recognition through visualization. Very cool stuff, though I think I'll make sure I stay dressed in front of the PC-CAM.

And, I am fascinated by the concepts of surface interfaces -- as an active sailor, I'm envisioning a future sailboat that has a surface navigation computer that ties together mapping/charting, GPS, depth sounding (and underwater 3-D profiling), routing and guidance, with timely updates on chart data from the NOAA Notices to Mariners, coupled with a finger-driven interface similar to what Apple now uses on their iPods and iPhones (oh, wait, that's not from Microsoft).

But, I have to say, I was not "wow-ed" by Bill Gates' talk. I've heard him speak before and it's pretty much the same spiel I heard 20 years ago at a Microsoft CD-ROM Multimedia conference -- better human interaction with machines.

There were a couple of questions from the audience after the talk, and one questioner pounced on Microsoft for considering security as an afterthought (hear-hear). But my real disappointment is that Bill Gates' vision did not account for any of the real-world problems we seem to be facing: global warming, rising energy costs, environmental issues and so on. Where is computing really going to be in 20 years if, as Al Gore is suggesting, sea level has risen enough to flood some of our coastal cities? Or if we can no longer tolerate the environmental impact of hardware disposal?

So, I throw out a challenge: "What will computing look like in an age where we have higher sea levels, unaffordable energy, and intolerable environmental issues". Bill?

Folksonomies Applied

2008-03-12T15:10:00.006-04:00

In a previous blog entry, I was starting to think about folksonomies as they might apply to content management. Having lots of experience in topical classification, therauri, and related tools, I figured this would be a simple discussion, but it isn't.

What is a Folksonomy? "Folksonomy (also known as collaborative tagging, social classification, social indexing, and social tagging) is the practice and method of collaboratively creating and managing tags to annotate and categorize content. In contrast to traditional subject indexing, metadata is not only generated by experts but also by creators and consumers of the content. Usually, freely chosen keywords are used instead of a controlled vocabulary" (Wikipedia).

By their nature, folksonomies are created by the people. I think the key to a successful folksonomy is participation by many people -- we make up for the lack of a controlled, standardized vocabulary and it's application to content by sheer volume and enthusiasm from a wide-spread using community. In fact, complaints about this approach usually center around the imprecise nature of the tagging. Since users typically apply tags to content, the tags are often ambiguous, overly personalised and inexact. But, Guy and Tonkin make a persuasive argument that user applied tags are in fact converging -- that the overall universe of applied tags is becoming self-limiting. If so, then the universe of tags that are being created in services like Flikr and Del.icio.us are becoming useful bases for classifications.

And in other, related developments, these services are beginning to categorize their tags. Especially at Del.icio.us where there are now classification tags and action tags, among others. To me these seem like we gray-beards call facets.

But that's not really my point, though I think it is important. I think we need some tools that can work with content management to allow tagging, maybe even super tagging, wherein the tags are members of controlled facets. This isn't really hard. Virtually every content management system "knows" about content by a URI. And there are some very cool features in the Del.icio.us service, including keeping my bookmarks and tags private, and retrieving them via an API later. So, we can set a bookmark in Del.icio.us containing the URI of the content we want to tag with the tags we want for that content.

I tried this very simply by registering at Del.icio.us, turning on the "private bookmarks" setting, and putting the Del.icio.us buttons on my browser toolbar. Then I pointed my browser at a content item in a CMS (Alfresco) and clicked the Del.icio.us "Tag" button. Added tags and saved it. I can see the tags and the URIs in my items on the Del.icio.us website.

But, to actually use this information, we need to pull the tags and content URIs back out of Del.icio.us. XML to the rescue. Or, rather, XML and the Del.icio.us API. We can fetch the tags we're using by using this URL: https://api.del.icio.us/v1/tags/get. We can also see all of our content using this URL: https://api.del.icio.us/v1/posts/get. And we can retrieve by tag: https://api.del.icio.us/v1/posts/get?tag=C

Pretty cool stuff.

Library Takes 'Talking Books' Digital, Washington Post, March 5, 2008

2008-03-07T14:24:00.003-05:00

I love this article in the Washington Post. For a couple of reasons. First, I have several friends who are legally blind (which I think is different from "illegally blind"). These kinds of tools help them enormously. It makes me feel like, as a society, we're doing the right things at least sometimes.

But, and perhaps more importantly, I'm thrilled to see this because WE DEVELOPED IT. That's right folks, the excellent programming staff at Retrieval Systems (who pay me) wrote the underlying software for the digital talking book for the Library of Congress, National Library Service for the Blind and Physically Handicapped (NLS). So this makes us proud.

We've been doing work in the background for the blind community for many years. Mostly this has involved library data exchange. For both NLS and the Recording for the Blind and Dyslexic (RFB&D) we wrote bunches of code to help them exchange bibliographic records, supporting creation of a union catalog so that blind patrons could more readily find books. And for NLS we've also developed some production tools to help in the creation of digital talking books.

But the DTB player is something of which we're especially proud. Not only was it technically challenging, it was socially responsive.

A New Kind of Content Management

2008-02-13T10:28:00.009-05:00

I've been enamored with classification of content through my entire career (which is certainly long enough). Early on, I was working on library automation systems and in particular with Library of Congress Subject Headings (LCSH) data and I always thought classification of documents and other content types was a panacea we should try to achieve. Librarians used these and other tools as finding aids to locating specific information, usually books.

Along came the internet and the amount of information available grew exponentially. Search engines provide keyword access to text-base content. Publishing was redefined to include anyone putting information out on the internet. Published content became a thing to be managed with buzzwords like single-sourcing, XML, multimedia, syndication, blogs, Wikis, and so on.

Well, the world is very different now and with everything from books and documentation to audio and video as well as thought streams all published on the web, finding a particular bit of information can be a nightmare. Even traditional textual content is hard to find in the vast archive that is the internet.

So how do we find things now? What are our modern finding aids starting to look like?

Some very cool new technologies have come along recently: folksonomies and mashups, to name two.

Folksonomies in particular offer a new approach to an old problem -- how to capture the "aboutness" of content. Classifying content is an age-old process that has spawned many fields including taxonomies and indexing. These, in turn, have created an arcane set of rules and procedures wherein the maintenance of the indexes or thesauri become big efforts on their own -- at times larger than the content management effort they are supporting.

Mashups are a cool way to combine content from multiple sources into a single presentation. Kind of like federated search and portals in a social-networking environment.

I'll be exploring these more over the next several posts.

Offshore Development Woes

2007-12-07T14:44:00.000-05:00

In his excellent article: "Using an Agile Software Process with Offshore Development", Martin Fowler summarizes his experiences in using offshore development for large projects. While his particular emphasis is on agile versus waterfall development approaches and the specific twists for offshore agile development, he makes a startling conclusion regarding offshore development: "Certainly anyone doing it because they think they'll get cost savings similar to the rate differences is seriously deluding themselves. "

This really hit home for me. We've been seriously examining our rates, partly because of our perception of downward pressure caused by the offshore "threat". But as I looked at it more, it seemed to me that there were several misperceptions going on here. First, it turns out that offshore arrangements we're competing with are under a contract for a minimum number of hours. Well, I'm certainly willing to offer a generous discount for a commitment like that! But more importantly, there is the hourly rate perception.

Just because the hourly rate is lower does not mean the project will be cheaper. I've always felt that way, but Martin Fowler supports it with evidence. Fact is, offshore development carries a large communications and distribution overhead that offsets the lower hourly rate.

He also says that "anyone who thinks that onshore developers will triumph because they are more skilled is very wrong. We've found that we can hire just as talented developers in India as we can in North America and Europe." And this has been my experience as well. The talent available there is excellent, though (IMHO) not better than we have here.

So, at the moment, it would seem to me to be a question, not of savings or talent, but of industry knowledge. The real savings should come about by using developers skilled and knowledgeable, or at least conversant in the industry area the development is supporting.

I say, the playing field is level, competition is good, let's sharpen our tools and have at it...

Estimating Software Development

2007-11-15T16:49:00.000-05:00

I'm an excellent software architect, developer, project manager, and even (as needed) sales person. But I can't estimate a project development effort to save my life. I waaay underestimate. Every time. I can lose money faster than a gambler in Las Vegas! So I figure I need to study the science (if such there is!) of estimating. In the past I've just thrown numbers into MS project and hoped for the best.

I was initially turned on to this idea of a systematic approach to estimating by one of the smarter folks I know, Norbert Winklereth, formerly of Omnimark and Stilo. He espouses an approach I had never heard of -- Wide Band Delphi Blind (WBDB). Being a believer in Agile methods, this appeals to me. I started reading up on it and I like what I see. It depends on several people looking at the whole problem (Wide Band), to develop a list of tasks and an estimate for each (The Oracle of Delphi) operating anonymously to reduce or eliminate political influence or pressure, (Blind). There's a fair writeup of it HERE.

There are a lot of parts to this, but the essence of it is to get several people working together to bring their collective expertise and experience to the issue of estimating an effort. And, it can be used to estimate a number of things, not just cost, but anything to which we can apply a unit of measure. One thing I began to see early on is that estimating the effort of a project and estimating the schedule are two completely different, though related things.

Then I read an article by Joel Spolsky (Joel on Software) on something he calls "Evidence Based Scheduling." Joel argues for (and, indeed, has a software product based on) creating schedules based on historical performance data (from time sheets or other records) combined with Monte-Carlo simultations. Again, recognizing the difference between effort and schedule, Joel makes a point that any task planned for more than 16 hours is not going to work and needs to be further factored. I think the same thing applies to estimating effort -- break it down into units of not more than 16 hours.

Another factor that Joel recognizes is the relative historical accuracy of an individual as an estimator - what he calls velocity, the ratio of an individual's estimate to the actual. A perfect estimator would always have a velocity of 1. A person's history of these ratios is his velocity history and is a factor in the Monte-Carlo simulation.

So, I'm thinking that a blend of these techniques may represent a reasonable estimating approach. We need to assemble a blind group, and a project leader who has and continues to build a velocity history for each blind participant. We need some Monte-Carlo simulation software, and maybe a tool to organize all this.

More to come

Agile and Budgeting

2007-08-17T11:10:00.000-04:00

Scott Ambler has an excellent discussion of the relationship between, resources, and schedule in the "Iron Triangle" of software development and the need for flexibility in at least one of these areas to ensure project success. Scott concludes by outlining several scenarios in which one or more of the sides of the triangle can vary.

In my experience, the budget for a project is usually fixed, or at least proscribed to a certain upper boundary. So there are really only two sides of the triangle that can vary -- the scope and the schedule. Of these two, I find that the schedule is usually less flexible that the scope -- often there is a market-driven date establishing an outer boundary on the development process.

Which leaves us with the scope of the project as the only negotiable or variable side of the triangle. This leaves the development manager in a somewhat difficult position -- how to satisfy the user community (who's focus is on features) and also the client bursar (who's focus is on budget and schedule). The key to this is to develop the most important features first and leave the less important features for the end of development. This, in turn, requires a frank and realistic discussion of feature priorities. So, at the outset of a project, perhaps the most important thing the development manager can do is to get a solid list of the features and their priority.

OOXML - The Votes are In!

2007-07-18T11:05:00.000-04:00

On Friday, July 13th, INCITS V1 took a role-call vote on our recommended position regarding DIS-29500, Office-Open XML. In what I imagine as a strictly partisan vote, we were not able to arrive at any consensus (a 2/3 majority). This, in spite of extending the time limit and even a certain amount of bullying pressure (or so it seemed to me) from some of the voting members. And the partisan division pretty much seemed to be Microsoft and their business partners, and the rest of the membership.

The motions we entertained were a) a recommendation of "Yes, with comments", which means we recommend acceptance of DIS-20500, and have comments that should, but are not required, to be addressed; "No, with comments", which means we recommend acceptance of DIS-29500 only if the comments are all addressed, and "Abstain, with comments" -- and I never did clearly understand that one! We did not entertain a straight-up yes or no vote.

I'm a little surprised by the outcome. Not so much that we could not agree on either a yes or a no, but that we could not reach a compromise position on abstaining. It seemed to me that the members voting yes were completely unwilling to meet in this middle position, voting down the abstain position along the same party lines. This unwillingness to compromise reinforces (to me, anyway) the feeling of being bullied -- the pro-OOXML membership essentially saying "my way or the highway".

It is of interest that in the past few months, 19 new members have joined this committee. And of those 19, 14 voted "yes". Certainly from my viewpoint, this looks like ballot-box stuffing.

OOXML -- What's Happening Here?

2007-07-09T11:59:00.000-04:00

Lucky me, I'm a voting member of of INCITS-V1 -- the sleepy standards group that has been working on such arcane things as TopicMaps. We're now working through the 6k+ pages of DIS29500 -- Microsoft's proposed standard for Office Open XML (OOXML). And my first question is, why?

Now, I'm not invested in or in any way partisan to the so-called-competing Open Document Format (ODF) that is already a standard and is in the process of getting a new version standardized. ODF is a part of the open-office software suite. I only just recently learned that it existed at all.

And, by the way, I don't see this (as some apparently do) as a Microsoft v. IBM fight, though I do see it as a fight of some kind. For me, a "standard" should serve a couple of purposes: to promote interoperability and to enhance portability being chief among these.

So, why do we need two competing open document standards? The answer from the proponents of OOXML is that there is no way to augment ODF (the extant standard) to include the features of OOXML. This is partly because OOXML supports the Microsoft Office binary formats (from Office 97 forward, I think) and partly because "the process model is different".

I'm trying hard to understand this. If OOXML can only be implemented with knowledge of the internals of various versions Microsoft's office serialization formats, how does that promote either portability or interoperability, except, maybe, within Microsoft's product lines. And I just don't get the "different process models" response -- I would have though that a standard, especially an XML-based standard, would be above that.

To me at least, the issue is more subtle. If we (V1) approve OOXML as a standard and it fails to pass the litmus test of promoting portability and interoperability, I think we weaken our standards process as a whole, making standards less trustworthy. Kind of like what's happened to Wikipedia; a phenomenal web resource that has been co-opted by corporate marketing efforts (try searching for "Windows").

XML: Implications and Opportunities

2007-05-15T14:26:00.000-04:00

Everybody's heard of XML and all the hype that comes with it. But what is XML? And what does it mean for publishers?

Often, content is authored, edited and published using a markup form that only controls style and presentation and not meaning. Examples of this are word processor files and web pages. XML offers the chance to capture the meaning of the content in the markup. Instead of this:

Allen, Thomas B. Vanishing Wildlife of North America. Washington, D.C.: National Geographic Society, 1974.

We have this:

<citation>
<author>Allen, Thomas B.</author>
<title>Vanishing Wildlife of North America</title>
<publisher>
<name>National Geographic Society</name>
<place>Washington, D.C.</place>
<date>1974</date>
</publisher>
</citation>

From that XML we could generate the same citation or any of a number of other formats. But, with this XML, we can also find our content with searches such as "author=Allen" and so on.

The key to understanding XML and it's implications for publishers is "Content Use and Reuse." Let's look at a scenario from a project on which I'm currently working.

Current awareness (news) content is published daily. It was originally written in a proprietary markup format, for which there were lots of in-house custom tools. All the editorial staff know the markup language. New hires have a long learning curve. We replaced the proprietary markup with an XML form. We hid the XML behind a word-processor interface and publish in a variety of forms (old and new) from the XML. This is feasible because of the open standards basis of XML and the freely available tools for managing XML. But really, it's just economics. Saved money. A publisher embarking on this kind of restructuring can use any of a number of standardized content models as a starting point: ODF, DocBook, DITA, etc.

The real opportunity comes in the new ways the content can be extended and also reused. More can be done to describe the content, down to a very granular level. The XML form of the content can contain not only the content, but metadata describing the content. This might not appear in any one published version, but can be used for searching and linking applications. It can be published in many ways on different media through simple reformatting.

Semantic markup is the next big step. We capture not only the metadata, but also describe the "aboutness" of the content. From simple keywords to linked thesauri, to external, linked ontologies, we can capture many levels of domain meaning for the content.

Back to the GWU Summer Publishing Institute...

2007-05-15T13:28:00.000-04:00

So, getting back to the GWU Summer Publishing Institute, and the topic on which I need to speak, "Agile vs. Traditional: Methods for Building a Software Infrastructure."

It seems to me that, after exploring what Agile Publishing might mean, it's clear that an Agile Publisher needs a software infrastructure that supports the key tenets. I take that to mean a content management and publishing infrastructure that supports the tenets of Agile Publishing:

Customer satisfaction by rapid, continuous delivery of useful product
Working product delivered frequently
Working product is the principal measure of progress
Even late changes in requirements are welcomed
Close, daily, cooperation between business people and developers
Face-to-face conversation is the best form of communication
Projects are built around motivated individuals, who should be trusted
Continuous attention to technical excellence and good design
Simplicity
Self-organizing teams
Regular adaptation to changing circumstances

So what kind of framework is that? I'm envisioning a content management and publishing system that easily pushes out releases on a frequent basis, allows content and structural changes to be readily made, enhances communication between participants, enables remote meetings, provides templates for good design, has easy-to use process and product review tools, enhances workflow with loose rules, and can be easily adapted to changes.

This goes a ways beyond a content management system, or any publishing system on the market today because it demands flexibility and adds project collaboration tools.

In "The Emerging Art of Agile Publishing", XML.COM, 3/8/2006, Michael Fitzgerald makes some specific suggestions about how to be an Agile Publisher:

"Build real trust through constant, informal communication
Interleave work processes between writers, editors
Share work openly, don't store it in secret silos
Store source files in an online repository and give all members of the team access to it
Agree on and share tools to reduce the waste of format conversions
Carve up your work into small, easily consumed and exchanged pieces
Keep track of what you are doing daily with some other suitable tool
Publish your work early and often, charge a little something and get free reviews
Let Herb do things his own way, but not on your team"

I would add to that, "Keep it simple".

So what software is needed? First and formost, a communications tool. Especially in these days of telecommuting, getting face-time is hard. I like instant messaging and the Skype product in particular. Skype has a secure chat and telephony in one kit. Second, a content management system that will permit easy and frequent releases of product. There are many of these, including several open source ones. Third, open project management. The basecamp.com approach seems nice as does the offering from project.net. I'm personally in favor of an open project document repository for sharing project related materials. This could also be used for openly managing project management materials, such as an MS project plan.

Agile Publishing -- Continuous attention to excellence, good design, and simplicity

2007-05-02T14:41:00.000-04:00

Principles Number 8 and 9 of the Agile Software Development approach . What do these mean for an Agile Publisher?

As with some of the previous principles, this is partly just good business and I feel these are all closely related. Simplicity => Good Design => Excellence.

Especially when developing a complex product with many content components, it is easy to design an overly complex, difficult to build and maintain product. Simplicity is the name of the game here. Also, good design -- keeping sight of both the immediate need and the growth room and designing with these in mind.

Agile Publishing -- Projects are built around motivated individuals, who should be trusted

2007-05-01T15:17:00.000-04:00

Number 7 of the Agile Software Development approach . What does this mean for an Agile Publisher?

It's all about trust. This also could be said of every project everywhere. There's nothing different here for an Agile publisher than for any other development effort. Michael Fitzgerald ("The Emerging Art of Agile Publishing", March 8, 2006, XML.COM) posits these tenets:

People must be trusted
Fewer but more competent people are needed
Organizations must live with the decisions developers make

These are key points. Trust inspires dedication, dedication reduces numbers of people. But (I hear you cry), organizations must live with the decisions developers make? Hard to swallow, that. But remember, a key point of the Agile approach is that the development team includes stakeholders. It is partly their role to ensure that the decisions the developers make are in the best interests of the product and the client. A better product will come of it.

Agile Publishing -- Close, daily, cooperation between business people and developers and Face-to-face conversation is the best form of communication

2007-04-27T18:20:00.000-04:00

Close, daily, cooperation between business people and developers and face-to-face conversation is the best form of communication -- Principles Number 5 and 6 of the Agile Software Development approach . What do these mean for an Agile Publisher?

It's all about communication. This could be said of every project everywhere. There's nothing different here for an Agile publisher than for any other development effort. Michael Fitzgerald ("The Emerging Art of Agile Publishing", March 8, 2006, XML.COM) posits these tenets:

The culture of the organization must be supportive of negotiation
Organizations need to have an environment that facilitates rapid communication between team members

Bravo.

Agile Publishing - Even late changes in requirements are welcomed

2007-04-27T15:48:00.000-04:00

Even late changes in requirements are welcomed -- Principle Number 4 of the Agile Software Development approach . What does this mean for an Agile Publisher?

As the publishing world changes to include more and diverse assets in a publication, we introduce the possibility (or even probability) that changes will occur during the product development life-cycle. These could be as simple as changes in artwork or as complex as an overhaul in related assets.

One of the significant features of the iterative nature of Agile development is that the stakeholders (client reps) can see if the product is taking shape as was expected. If not, changes are not only possible, but encouraged, at just about any point in the development cycle. This same benefit obtains for changes to the underlying business model -- should the market conditions for the product change, we can respond to these changes.

Retaining this ability to accept changes to the requirements, no matter when they occur in the development cycle is a pre-eminant feature of Agile and applies equally to Agile publishing as it does to Agile software development.

Agile Publishing -- Working product is the principal measure of progress

2007-04-16T17:20:00.000-04:00

Working software is the principal measure of progress -- Principle Number 3 of the Agile Software Development approach (http://en.wikipedia.org/wiki/Agile_software_development). Let's change this to "working product is the principal measure of progress" and explore what this might mean for an Agile Publisher.

Norbert Winklareth, one of the smartest people I know, asserts that the measure of value for which customers pay for is not any traditional metric of development but features:

Value => Functionality That Works
Functionality => Set of Features
Therefore: "Features are the true measure of development"

It would seem that the same principle can apply to a publishing project -- a publication has, among other things, a set of features. In software, features are specific characteristics, such as a search function or an RSS feed. In publishing, features could be harder to describe. In essence, these are the characteristics of a product for which customers pay. As a practical matter, features could be publication components (or assets) for example, written text, artwork, multi-media software functions, summaries or abstracts, and so on. Each of these can be considered as a feature of the published product.

Deciding on the priority of the features of our product is important here. By developing the most important features first, we can plan the creation of our product in stages (or iterations) in which we get our most desirable or most important features built first. The result is that we have a working, functional product at the end of each iteration that could be considered a final product if need be. It certainly permits us to respond to budgetary or other outside business pressures and still end up with a publishing product.

It also provides us with a true metric for progress. As features are developed, we can measure that easily against our plan for the product. We then know how well our project is fairing.

Agile Publishing -- Working Product Delivered Frequently

2007-04-14T10:23:00.000-04:00

Working software is delivered frequently (weeks rather than months) -- Principle Number 2 of the Agile Software Development approach (http://en.wikipedia.org/wiki/Agile_software_development). What does this mean for an Agile Publisher?

One of the key issues that arises in traditional software development is that the client's vision of the features and functionality of a product are different from the developers' vision. Another is the inability of the stakeholders to control development against their budget. These two issues are at least partially addressed through the use of Agile's frequent delivery of working software principle. Specifically, by getting frequent releases, clients can see for themselves how conformant the developers' vision is to their own. Also, and especially when combined with another concept -- planning for priority features in the early releases, should the development cost begin to approach the project budget, the project can potentially be terminated early with much of the priority feature set in working order. Powerful stuff.

Developing a publishing product can follow a similar flow to software development, one in which iterations of content are created and reviewed. This is especially true for products with content derived from many sources. Following a similar flow of development, we can exert the same control -- create iterations of working versions of the product and with more important features (included content sources perhaps) developed first.

In the realm of electronic publishing, it is almost always true that a product is derived from many sources and I've seen many publishing efforts held up in order to include a particular content asset. I wonder if that asset was important enough to the product to incur the delayed release and resulting revenue.

Agile Publishing - Continuous Delivery of Working Product

2007-04-04T09:40:00.000-04:00

Customer satisfaction by rapid, continuous delivery of useful software -- Principle Number 1 of the Agile Software Development approach (http://en.wikipedia.org/wiki/Agile_software_development). What does it mean for an Agile Publisher?

To my mind, this is one of the most important pieces of Agile. This is what ensures that what the customer envisions is what is being built and that if not, corrections/realignment can be done early and at the least cost. It has some secondary advantages as well: regularly tracking progress against a budget; early start on customer training; and easier collaboration with client/stakeholders, to name a few.

What does this mean to electronic publishers? Publishing describes a large, complex industry that delivers information to a user community. There are many forms of electronic publishing: newsletters, online books, references, electronic databases, and so on. There is a commonality to software development in that published products (or assets) go through a development/authoring process, a QA (or editing) process and delivery.

An Agile Publisher would apply the principle of "rapid, continuous delivery" to each electronic product. In practice this could mean author submissions of partial manuscripts, developer delivery of database subsets, and so forth. Any of these early and frequent data feeds provides a foundation for "rapid, continuous delivery", which the permits the goals of early and ongoing review and realignment, budget tracking, etc.

But, to take advantage of this, the electronic publisher must also have a deployment infrastructure that will create a working version of this electronic asset. In other words, the software and processes must be in place to publish the early edition of the assets.

Agile Software Development v. Traditional "Waterfall"

2007-04-03T12:41:00.000-04:00

I've been involved in two projects recently in which I got a first hand opportunity to compare the results of software development under Agile and under the traditional "waterfall" method. The two projects were not really similar in any way, but they still bear some scrutiny for the results of the two different approaches.

The project done using a tradition (or waterfall) methodology (which I'll call "Project W") began with a requirements gathering phase in July/2004. This was followed by an architecture and design phase, programming, testing and acceptance testing. We will (hopefully) deploy this system to customers in June, 2007, nearly three years after the project started.

The project done using Agile (which I'll call "Project A") was started in August, 2006 and was delivered and live in production in December 1006, around 5 months after the start.

Now, there are some very big differences in these two projects, including the complexity of the underlying data models and in the overall software architecture. However, it is interesting to note that a big part of the 3-year time to deliver on Project W came from two factors: redesign of system components that were found to be faulty during programming, and under-developed requirements. Especially in the past six months, we have spent a significant amount of developer resources adding/changing/deleting user features long after the requirement were solidified. In contrast, on Project A, at the point where the software was ready for testing, there were few feature changes.

Why? A significant aspect of Agile is that the client/user/stakeholder "sees" working versions of the software early on. So if there are any mis-understood requirements, these are found and fixed very early in the development life-cycle. Also, in Agile there is a much more limited design
process, relying instead on developing working code. As a result, any design errors were uncovered earlier in the cycle.

An interesting thing happened on Project A -- we exceeded the budgeted costs. Although this was conveyed to the customer as it happened, it still caused a great deal of consternation. However, and in spite of that issue, I view the project as a huge success. Had we followed a "waterfall" paradigm, it would likely have cost the same or more and would have taken longer to complete. So although we missed our target budget, we delivered a market-ready product in five months, enabling the customer to begin generating revenues quickly.

I'm totally convinced that Agile is superior for the kinds of projects on which I tend to work.

Agile Publishing

2007-04-02T14:36:00.000-04:00

I've been invited to speak at George Washington University on the subject of Agile Publishing. This is a new application of Agile programming principles applied to content rather than to code. So I'm starting to explore the relationship and applicability.

The basic tenets of Agile Software Development (taken from http://www.agilemanifesto.org/) are:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

And Agile at work in software development usually means following these principles (http://en.wikipedia.org/wiki/Agile_software_development):

Customer satisfaction by rapid, continuous delivery of useful software
Working software is delivered frequently (weeks rather than months)
Working software is the principal measure of progress
Even late changes in requirements are welcomed
Close, daily, cooperation between business people and developers
Face-to-face conversation is the best form of communication
Projects are built around motivated individuals, who should be trusted
Continuous attention to technical excellence and good design
Simplicity
Self-organizing teams
Regular adaptation to changing circumstances

So, how does this apply to publishing? Well, there are both philosophical and practical answers. Michael Fitzgerald published an article on XML.COM on this at http://www.xml.com/pub/a/2006/03/08/agile-publishing.html. In it, he examines the basic tenets of Agile in a publishing context and makes some specific suggestions on how to become agile. I'd like to extend his work. In the next several blog entries, I'll take a look at each of the principles in a publishing context. But first, let's look at what makes for a successful Agile project:

The culture of the organization must be supportive of negotiation
People must be trusted
Fewer but more competent people are needed
Organizations must live with the decisions developers make
Organizations need to have an environment that facilitates rapid communication between team members

I believe this holds true for an Agile publishing effort as much as for an Agile software development effort. These key elements center around trust and communication. Find good people, empower them to success, trust them to get there and communicate regularly. It may mean compromising certain aspects of traditional publishing, such as corporate style oversight, high-overhead decision-making, and trusting decisions to workers.

Content Management Frameworks -- Tractare and Configuration

2007-03-19T17:32:00.000-04:00

There are a large number of Content Management "Frameworks" out there -- http://en.wikipedia.org/wiki/Content_management_framework lists just a few. What is a "Content Management "Framework?" In essence, it is a programmable API for creating customized content management systems. In other words, a programmers way to create (using the framework) a CMS that supports all the stages of content lifecycle: Organization - Workflow - Creation - Repository - Versioning - Publishing - Archives (or some subset thereof). In essence, a CM framework is a foundation on which a custom CMS can be built.

The key features I would expect to see in a CM framework include (http://www.cmsreview.com/Features/Lists.html):

a way to acquire both text and non-text content (acquisition, aggregation, authoring)
a way to store and retrieve content
a way to control workflow (roles/permissions, checkin/checkout, messaging/routing)
a way to control versioning
a way to control personalization and localization
an interface to administration (reporting, management, etc.)
a way to control content delivery (extraction, slicing, publishing, syndication, update)
a way to implement business rules
and others...

The trick is making this work as a framework. How much needs to be customized by a programmer and how much can be done by a very tech-savy non-programmer.

In our CM Framework (Tractare), we permit most of these features to be customized by scripting. For example, many organizations have IT departments that mandate the use of a particular DBMS (Oracle or SQL Server, for example). So in Tractare, there is a fairly simple configuration setting that allows selection of the database interface driver. But, since Tractare is a CM framework, we also expose the driver interface so that a programmer can create a driver for a database for which we have not supplied one. We make heavy use of configuration files so that non-programmers can customize the CMS.

Another area that we use scripting is in the interface elements (the "view" of the CMS). Internally, Tractare generates most interaction responses as an XML stream. This way, the entire user interface to Tractare can be tailored using XSLT scripts.

Depending on the application however, Tractare could still require an investment in programming. It is written in JAVA and intended to run as a web application (using servlets, etc.). So extending the code is a matter for a JAVA web programmer. But many features can be customized without this level of involvement.

Open Publish Conference

2007-03-11T11:58:00.000-04:00

I got to speak at the Open Publish conference in Baltimore (http://www.open-conferences.com/baltimore/) on Friday, 3/9/2007 amongst many of the content management and publishing luminaries. It was humbling. While the conference was small, it was very interactive and the audience was firing questions both during and after the talk. Kept me on my toes!

My particular talk was on topic maps, and on using topic maps as an information architecture to improve hypertext linking. The premise is that users frequently click on links embedded in web pages that take them somewhere they didn't want to go and that with better knowledge captured on the "aboutness" of the link, we (as content providers) can give the user more information and options about the links they are following (anyone interested can get the presentation slides from the conference directly or contact me and I'll email them).

My presentation was either fabulous or boring (depending on whether you're asking me or an attendee). But what I came away with were a bunch of questions about how to glean a topic set from extant content. I've spoken on the subject of topic maps on several occasions and many of the questions follow this same theme -- "How do we collect and organize the topics of the content in a meaningful way?"

I always have some lame answer -- "that's a job for the subject matter experts." While this is true -- the current state of the art is somewhat limited in our ability to parse content and determine subject matter, it really begs the question. Often (in my experience) there is something with which to begin this process. Usually, our content has already been touched in this way, from something as simple as the application of keywords to creation and maintenance of an outside topical index and the positioning of a particular content object within that. And often there are tables of content that can be used to further glean the "aboutness" of the content. What we (developers) need to consider is how to create a toolkit that both captures as much of this meta data as possible and imputes a relationship structure to it that in the least can serve as the foundation for a subject matter expert's work, and in the larger sense can provide a fully automated creation of a functional, integrated taxonomy.

At this conference, I was asked if I knew of any translation program that would convert the output of an index creation/management program (CINDEX) into a topic map -- a good example of what I'm talking about here.

I'm going to explore this further. We (Retrieval Systems) have a bunch of content processing tools, including some tools to work with the CINDEX data formats. Perhaps we can develop a reasonable toolkit for this kind of conversion.