Maple Syrup Map: January 5, 2009
Smellings on a Monday night (into Tuesday morning)
This map identifies the reported locations of a maple syrup smell in NYC. It uses distributed, unverified reports of odors that could be used in conjunction with a windrose map to estimate the emission source. Building downwash and wind directions would cause problems in estimating exactly where an odor came from.
This system could be expanded to allow reports of odors or visible emissions from anywhere in the world. With enough data points the larger sources would become apparent, and could be automatically tracked.
Tuesday, January 6, 2009
Tuesday, December 30, 2008
automating paperwork
By Maj Dale Long, USAF
Welcome to the latest installment of the "Lazy Person" series, where I'll tackle everyone's favorite nightmare: automating paperwork. This Holy Grail of the paperless office gang has achieved mixed results over the last few years. However, recent advances in technology and a better understanding of how not to automate (acquired painfully by many pioneers along the way) lead me to believe that we're on the verge of finding some relief.
First, let's play a little numbers game.
Last year, the headquarters I work for recorded over 52,000 staff actions in our automated tracking database. This number is twice what we processed three years ago, and no one here really likes the angle of ascent in the measurement.
The amount and complexity of our staff work is increasing. We are being asked to handle more issues with fewer people. Even though some of us are using e-mail vice stacks of paper to handle taskers, our work processes still appear rooted in paper-based paradigms for routing, format and coordination.
In addition, there are over 3,500 electronic DoD forms in use with no connection to a database. Most of them are simple "fill-to-print" forms generated on our computers and routed around the office, sometimes attached to e-mails, but more often than not in a folder or envelope. Instead of being an active interface to some type of automated process, they're the virtual equivalents of lead weights strapped to e-mail and used to fill up space in our mail servers. Eventually someone will take pity on the poor things, print them so they can fulfill their main function and zap the electronic versions into oblivion.
At some point in recent history, someone decided the staffing process that has served the military in various incarnations for most of this century is now a millstone around our necks. Whether they reached this conclusion on their own or with the help of the marketing machines of various workflow tool vendors, computer pundits or at a management seminar, I don't know. Whatever the reason, apparently a large number of people are quite convinced we need new workflow technology, right now, to solve our problem.
Before we go install any software, though, let's define the problem.
Welcome to My Nightmare
As usual, my first problem with workflow came in an e-mail from Zippy - and not just any e-mail. Apparently he and Zippette had built a Web-enabled database to act as a workflow system for the annual (and completely unofficial) office Super Bowl pool.
Like most of their efforts, this one was a masterpiece of hacking, complete with an interactive matrix where you could fill in which blocks you wanted to buy. Money never changed hands on government property, of course. Zippy may be a few fries short of a Happy Meal, but he's not foolish enough to actually run an overt gambling operation, even one as benign as an office pool, over a government computer network. For all outward appearances, this particular application was a "test pilot" of interactive matrix development.
As usual, Zippy developed the process models, and Zippette handled all the technical details. In his favor, he got most of it right. The problem is that Zippette can pretty much code anything Zippy can think of into a working application. With the right input she could probably crack the code for the human genome.
With Zippy's input, however, all we normally get is cracked code.
A football pool is usually pretty simple. You take a 10x10 matrix and have everyone pick blocks until they're full. Then you randomly assign numbers to the X and Y-axes. At the end of each quarter and the game, if the last digit of each score matches the two numbers that intersect one of your cells in the matrix, you win a piece of the pot.
The model was far too simple for Zippy. To those of us filling in the matrix, it looked like we expected. However, with his usual excruciating enthusiasm, Zippy built in a few dozen other winning combinations. These apparently included payoffs based on your cell being a number that could factor into the jersey number of anyone scoring a touchdown, the relative angle of the moon from Atlanta when a field goal was kicked, and a host of other odds so bizarre even Jimmy the Greek couldn't figure them out if you gave him a Cray and a team of MIT mathematicians.
On game day, Zippette somehow connected to all the NFL network television and space tracking systems needed to verify the winning conditions. At $1 a cell, the money in the pool came to $100. As the game progressed, quirks of fate of a probability so low that they could only be considered miraculous, split the money 573 ways between 21 players.
Fortunately, no one really expected his system to work. We figured $5 each was a small price to pay to watch Zippy's brainchild self-destruct.
But it was clear that adding in a host of extraneous features (a.k.a. "requirements creep") can doom even simple, well understood projects. It served as a reminder of the old maxim: "A fool with a tool is still a fool."
What Is Workflow?
"Workflow" is one of those terms someone coined based on the idea that work "flows" through an organization. The visual image suggested by the term is supposed to be a boat (the work) flowing down a stream or river with the current carrying it smoothly towards some ultimate destination.
In the military when people talk about workflow they are often referring exclusively to staff work. While staff work is a workflow process, it is not the only one. Any activity that can be explicitly defined and modeled, from ordering a chair to submitting a performance report, can be automated with the right system.
The people clamoring loudest for relief at the moment, however, are mostly involved in staff work - and with good reason. Unlike that boat on the river, our staff work frequently looks a lot more like a salmon trying to swim up a waterfall.
Defining Workflow
Workflow is, first and foremost, about defining processes. Workflow process models normally have two levels of representation. The first level defines the process of the flow of work through the system. The second level represents the state of execution of a particular process, usually described as an "enactment."
For the purposes of workflow modeling, we'll assume a process consists of a name and a sequence of activities needed to execute the process.
At the next level in the process, an activity consists of a name, a set of roles and actions associated with that activity and usually a completion condition.
One thing to bear in mind at this point is some activities contain enough complexity to qualify as complete processes in and of themselves. These key activities are encapsulated within the larger process and are often the critical components whose performance will make or break the entire system. It's quite possible to model the larger process correctly but then have a workflow system fail because of a lack of sufficient detail at the activity level.
My personal preference when modeling activities is to bound them within single roles. A role is what an actor plays in the execution of an activity. An actor can be a person, a group of people, a computer program or an organization that provides the functionality for the system by executing activities.
If we define our actors' roles clearly for each activity, we'll have a better chance of defining and meeting the specific requirements of those activities. Each activity should consist of the actions executed contiguously by a single role. When the role changes, we've entered a new activity. We should be able to reduce just about any process to manageable chunks for implementation with this method. We'll talk more about roles later.
An action is a description of something to be done. "After coordinating, forward this to the next reviewer," "send an e-mail to acknowledge receipt," and "apply digital signature" are all examples of specific actions that may be executed by a person or software agent.
The tedious part of building a workflow system is ensuring that every action executed by every actor in every role for every activity in the process is accounted for somewhere in the workflow system. Each action should also satisfy an argument ("if this, then this; if that, then that") that generates further action based on a perceived (either by human or machine) value.
That's the process level. At the execution level, each iteration of a workflow process is considered an "enactment." This consists of a unique enactment ID number, a reference to the process being enacted, a reference to current activity within that process, binding actors and roles to activities and matching arguments to values.
In plain English, every time we send work through the system we should know the following at any given point in the process:
Something that uniquely identifies each work package (ID number).
What action are we performing? (Ordering, coordinating, modifying, etc.)
Where are we in the process? (How many activities have we completed?)
Who or what is currently involved in the action?
How is the work proceeding? (Are the actors responding appropriately?)
When is the work complete? (At any level: action, activity or enactment.)
Here's the scary part: as complicated as it appears, the description above greatly understates the difficulty of actually modeling a process with the intent of building workflow around it. In addition to explicitly defining every action in the process, particularly the ones to be automated completely, we must also define the networks of activities that specify parallel, iterative and conditional interactions between actions, actors and activities.
One thing that helps is that workflow enactments are case-based. Every piece of work we do, for example, is executed for a specific business case and almost always for a specific customer. The goal of workflow management is to handle the enactment of those cases as efficiently and effectively as possible.
So, we design workflow processes to handle similar types of cases, with workflow management being what we do to try and shepherd our work through our organizations. With that basic understanding, we will now move on to workflow management systems.
Automating Chaos
The Workflow Management Coalition defines a workflow management system as: "A system that completely defines, manages and executes workflows through the execution of software whose order of execution is driven by a computer representation of the workflow logic."
On a slightly higher level, workflow management systems support the definition, registration, execution and control of processes. This is a fairly broad definition - just about any automated production system fits here. But, as we will examine later, workflow is more than simply staffing or transaction processing.
Unfortunately, workflow management is often thought of only in the context of computer-mediated workflow management systems. While computer-based workflow systems are becoming more common, this unnecessarily limits the application of workflow management as a discipline. It is entirely possible to do workflow management without using a workflow management system and improve the work process without making any change at all in technology.
However, computer-mediated workflow management systems are becoming the rule rather than the exception. I think it's safe to say if we're going to do something about the glut of staff work clogging our organizational arteries, it's going to take some technology changes to support it.
In the commercial world, workflow systems are used to support a variety of business processes. A classic example is the system your local video rental store probably uses to manage its business. It encompasses account management of customer information, inventory control of the movies and a host of other functions associated with running a multi-billion dollar business with a staff consisting in large part of high school students working part time.
We should be so lucky as to achieve that level of simplicity and ease of use in any system, let alone workflow.
Now, with the understanding of workflow and workflow systems, let's move on to how workflow management specifically applies to our current challenge: automating staff work.
Charity Begins At Home
In a military headquarters, regardless of the mission of the organization, the core function of that headquarters is to make decisions and give direction. Therefore, we have staffing processes dedicated to giving our decision makers what they need to make those decisions and give direction. The decision-making process is as crucial to a headquarters as the aircraft maintenance process is to a flying unit.
I've seen many attempts in military organizations to automate all or part of the staffing process used to support decision making. They range from simple databases or spreadsheets used to track the work in an office to full-scale, base-wide systems that try to account for everything on the base. Most were developed locally using commonly available tools to avoid the cost of buying a "real" workflow management system from a commercial vendor for millions of dollars.
One of the lessons learned from these development efforts is that any workflow system implementation will be limited by our understanding of the work process. Just what is staff work and how should we automate it?
As people are the principal processors of our staff work, let's start by defining the various roles in the staff activity. There are two basic roles: administrative and business.
Administrative roles include initiators, monitors and channelers. Initiators enter tasks into the system, monitors keep track of the work as it progresses and channelers redirect or delegate work.
On the business side, we have workers, reviewers and approvers. Workers collect, create, transform and integrate the various information objects that form the input and output of the system. They produce the product from which the system derives value.
Reviewers are intermediate approval levels. In many cases, reviewers are the same people who channeled the work downstream to the worker. Their role simply changes as the direction of the work shifts. Review activities include quality control (editing, fact checking, etc.) information aggregation, filtering and, in some cases, censorship.
Approvers can approve the closure of an action. There may be some mechanism for appeal by other levels, but at some point each action should come to an end.
There may be other roles depending on the business process, but these are the standard ones I've observed so far.
Who's On First?
Now comes the fun part: defining the requirements people (or automated subsystems) executing these roles need to function effectively.
While I've seen requirements developed based on functional descriptions of different types of work, I recommend we model our requirements based on the activities actors must perform to fulfill their roles. For the staff process, this means either having the system give these actors the information required for their role or automating all or part of their role into the system.
Administrative roles don't require subject-matter experts. You don't have to know anything about the work itself to know who's working on the project, when it's due back for final action and where it is in the coordination process.
Administrative requirements might include:
Having the "subject" as a mandatory field.
Sending an e-mail reminder to the person currently working on the project three days before it's due to the next level.
Showing everyone who has been assigned to work on this project.
Showing who can be assigned to this project.
Of those requirements, the first three can be completely automated. Remember when we listed "matching arguments to values" as a function of an individual workflow enactment? Well, this is where you apply it most: during automation.
A computer can check to see if the subject block has been filled in, but if the entry was gibberish, it would never know. But as a condition of initiating an enactment it will know, absolutely, whether or not this field has data entered. We must then depend on the initiator to make the entry meaningful to everyone else involved.
The next two are also based on simple conditional arguments and explicit values. The workflow system should be able to identify both the due date and who is currently responsible for taking action and sending a "nag note."
The last requirement, however, is an action normally completed by a human. Yes, if the system knows enough about who has done what work in the past it can assign all the computer security projects to Tim in the network center. But unless there's only one person in the organization who does all the staff work, we're most likely going to reserve the act of assigning or delegating to a human.
Our business role actors, however, do need to know something about what we're working on. They transform questions into answers, or frequently in the military, observations into command decisions. Requirements for business roles might include:
Showing who else has been assigned to work on this project.
Using a polling/voting mechanism to facilitate documented consensus between action officers.
Having a version control mechanism for group editing.
Linking to the organizational records archive.
The first, which supports building virtual teams, is fairly straightforward. The system should be able to identify the current actor at the lowest level of each delegation chain and build a list. This is useful - the only way to do this outside a workflow system is to personally call or e-mail every office working the project and ask whom they've assigned to it.
The second requirement, however, rises to the level of a process all by itself. Achieving consensus is the key activity in coordinating staff work at any level. This is where we spend the most time and effort, and it is the activity most likely to make or break a particular enactment (project). We can be on time and go through all the right motions, but if we don't build the decision properly, we might as well not bother doing it at all in most cases.
The Coordination Game
Now we'll dissect a typical coordination process. Someone writes a policy document that has to be coordinated across every subunit at every level: branch, division and directorate. The document works its way down to the actor performing as the integrator.
At this point, the document is disseminated across the different branch offices via paper, e-mail attachment or as a file shared by all from a central location. Whatever the method, two dozen people are now busily involved in reviewing and commenting on the content of the document.
Ideally, our test document is well written and meets everyone's expectations. However, one or more of our collateral reviewers usually has input on some of the content.
The fun begins when our integrator receives all these comments. Maybe somewhere, sometime, some group makes their comments concurrently on one shared copy of an electronic document. This would constitute a virtual discussion as part of a group review.
I have yet to see it personally, though. I treat reports that this actually occurs in nature with the same skepticism I reserve for Loch Ness monster sightings or paid political advertisements. At the end of the review period our integrator is usually presented with two dozen individual inputs and must somehow meld them into the original document to produce Version 2.
As this version is essentially a new document, it gets re-coordinated across the organization again. It's entirely possible for some of the input from different reviewers to conflict and for changes submitted by one reviewer to be unacceptable to another. The process of global review continues until everyone on the same level comes to some agreement (or are beaten into submission) and the document can progress up to the next review level.
We also touched upon the third sample requirement: version control. As with any process, some activities facilitate others, like version control facilitates the prime activity of consensus building. Identifying and mapping these activity relationships is an important and often overlooked part of the process modeling effort.
The last sample requirement, linking to our archives, recognizes how people frequently need to recycle older information to answer the current question. Reinventing these wheels can be both time consuming and frustrating. Even if the link is to nothing more than an index engine pointed towards our archive, it can save workers a lot of independent, unstructured (and possibly clueless) searching.
Please bear in mind that our sample list is a very small subset of the requirements I'm used to seeing for workflow systems. While most lists include dozens of requirements, I have yet to see one over the last six years that actually appeared comprehensive.
Congratulations on making it this far (unless a friend told you to skip ahead to this point) and getting through the admittedly lengthy discussion. At this point we're ready to finish this topic discussion by tackling four important issues associated with workflow system development.
1. Don't Let The Inmates Run The Asylum
This advice is based on the Alan Cooper book of the same title. Cooper submits, among other observations, that software is difficult to use because software designers typically focus on the nature and needs of the computer, rather than the nature and needs of the user. This produces feature-rich software to either conquer or try to avoid.
Cooper states the main reason we've achieved this level of system dysfunctionality is despite appearances, business executives are not really in charge of the software development process. The engineers run the asylum. With software, nothing is normally visible until it's been coded, which means that input from non-programmers in the user acceptance phase is usually too late to make any effective changes.
In addition to the inherent difficulties with using software built by people who do demoscene coding for fun, there is another group of inmates to deal with: the people who administer the work process. Sometimes knowing too much about the current process can get in the way of improving it. More often than not, they want to keep all the artifacts (even if they're dead weight) in the new, automated process.
Then they wonder why the rest of us never seem to like the "new and improved" system.
Electronic forms come to mind, particularly electronic forms that we do nothing with except fill and print. If all the information I need to coordinate a staff action is in the workflow envelope, why would I need to attach a redundant chunk of information in an electronic form? Instead of routing a fill-to-print form with my computer systems requirements, why not replace it with a Web form tied to living databases by a true electronic work process dedicated to receiving, sorting and satisfying our computer requirements?
Albert Einstein said:
"We cannot solve the problems we have at the same level of thinking we were at when we created them."
This will be particularly true of automating (dare I say re-engineering?) a process with a workflow management system. If we're not willing to raise our level of thinking above inmate level, no amount of technology will help.
With that in mind, my second admonition is:
2. Build the User Interface First
Let's face it: to 99 percent of the people who come in contact with a workflow system, the user interface is the system. They have no interest in the database, the SQL query structure, the process model or even what software they're using. They just want a simple uncluttered interface that tells them what they want to know and lets them take action without having to read a 51-page manual.
Anything beyond that becomes an annoyance.
So, design the input and report screens first and then build system functionality to support them. Rather than belabor this point on my own, I'll indulge my fondness for quotes here with two that apply to this situation. The first is from Michael Simpson, Chairman, A.M. Castle & Co:
"Anything that the customer is not willing to pay you one penny more for, you have to question whether you should be doing it. There's no point in doing something extremely well if you shouldn't be doing it in the first place."
The second is from Spanish writer Enrique Jardiel Poncela:
When something can be read without effort, great effort has gone into its writing.
These are good words to live by, particularly in interface and system design. The best engineered system in the world will still fail if the interface is ugly.
3. Acquire Tools Instead of Building Them
Given the difficulty of actually modeling our staff process to the necessary level of detail, I'm continually surprised that people keep trying to build their own workflow systems out of software that wasn't originally designed to automate work. I can understand their motivation, though.
First, they think commercial systems cost too much. No argument there, as most proprietary commercial workflow management systems will run into millions of dollars for a full implementation.
Second, they think they know enough about how their work flows to build a bet ter system on their own. While you've got to love optimism, please see the previous reference to inmates running the asylum. If the software designers and the user community have never done workflow activity modeling before, it's going to be an interesting journey.
Third, our core military competencies don't usually include building office automation systems. Why would we try to reinvent that particular wheel? We don't build our own database management systems because we're not in the database management tool business. We just use existing tools to build our databases based on our understanding of the data.
Yes, commercial workflow toolkits are expensive. However, the companies that sell them have already paid the cost of development and have accumulated some experience at fielding them. Instead of reinventing wheels, let's just attach them to our car. Being an expert on tool making and on tool use are two entirely different things.
One other thing to consider: do we want to save money or make money? If we choose the least cost alternative, we avoid spending a certain amount of money. But if the investment produces only a marginal (or no) gain, what have we achieved? Instead of developing each workflow management system separately (staffing, travel and purchasing, for example) using different tools, approaches and databases, maybe there is an advantage buying one standard set of development tools and leveraging them across all our workflow applications.
If we don't like the cost today, we may shortly have another alternative: an open-source workflow toolkit.
Vivtek has been commissioned through SourceXchange to develop an open source workflow toolkit. SourceXchange is a forum that links open source developers to investors with well-defined, financially backed desires for open source products. The company backing the development of the workflow toolkit is Galactic Marketing Incentives, Inc.
If they are successful, we can use the toolkit for free instead of paying costly licensing fees for proprietary systems. Of course, this only deals with one cost area - licenses - and not support, system development or process analysis. However, the licensing cost is a significant percentage of commercial systems, and the choice of systems can restrict you to a fairly narrow range of system providers.
An open-source workflow toolkit would be the equivalent of Linux for workflow. Any company or organization, including our own military systems development community, could acquire and use the toolkit for free. Instead of building proprietary, stovepipe applications for every business process, we could develop all of our automated processes using the same free toolkit across our bases, services or even the entire Department of Defense.
Maybe I'm dreaming at the moment. The big commercial workflow and transaction system vendors probably aren't quaking in their boots at the mere mention of open source competition. However, many of those same vendors have released or are releasing Linux versions of their software. Even IBM recently announced it would be using a version of Linux on its mainframes. Organizations with lots of money on the line are starting to buy into the open source movement.
If you want to hedge your bets, I recommend, "The Cathedral and the Bazaar," by Eric Raymond, one of the founders of the open source movement. Raymond details the differences between the commercial (cathedral) and open source (bazaar) approaches to software development and why he thinks open source development has a very viable future. You will find this and other Raymond essays at: www.tuxedo.org/~esr/writings/
If you're interested in Vivtek's open source workflow toolkit project, you'll find Michael Roberts' musings on his experiences trying to build the workflow tool kit at: www.vivtek.com/wftk.html. Information on SourceXchange is available at www.sourcexchange.com.
4. The Product Is More Important Than The Process
Here's a quote attributed to Peter Drucker, one of the leading management gurus of our day:
"To start out with the task rather than the end product may result, however, in beautiful engineering of work that should not be done at all."
As we set out to automate anything - staff work, planning, purchasing, whatever - if we only focus on the tasks involved in the process and not the product of that process, we will never make a significant difference in the outcome.
Let's refer back to 4,600 words ago, where I likened workflow to water flowing down a river. Left to its own devices, a river will find the path of least resistance and flow smoothly along. We can divert the river or dam it up, usually at considerable expense and effort, to achieve some end result. However, we will have to exert some effort over time to maintain our constructs. If we don't, the river will eventually return to its original state.
Instead of struggling to maintain all of the artificial constructs we've attached to our work processes when we automate them, perhaps we should concentrate on the output and outcomes associated with our work. We can be lazy, in the good sense and strip away all the artifacts that don't add value to the process and let our systems and our people do what they do best: get the job done with as little hassle as possible.
Welcome to the latest installment of the "Lazy Person" series, where I'll tackle everyone's favorite nightmare: automating paperwork. This Holy Grail of the paperless office gang has achieved mixed results over the last few years. However, recent advances in technology and a better understanding of how not to automate (acquired painfully by many pioneers along the way) lead me to believe that we're on the verge of finding some relief.
First, let's play a little numbers game.
Last year, the headquarters I work for recorded over 52,000 staff actions in our automated tracking database. This number is twice what we processed three years ago, and no one here really likes the angle of ascent in the measurement.
The amount and complexity of our staff work is increasing. We are being asked to handle more issues with fewer people. Even though some of us are using e-mail vice stacks of paper to handle taskers, our work processes still appear rooted in paper-based paradigms for routing, format and coordination.
In addition, there are over 3,500 electronic DoD forms in use with no connection to a database. Most of them are simple "fill-to-print" forms generated on our computers and routed around the office, sometimes attached to e-mails, but more often than not in a folder or envelope. Instead of being an active interface to some type of automated process, they're the virtual equivalents of lead weights strapped to e-mail and used to fill up space in our mail servers. Eventually someone will take pity on the poor things, print them so they can fulfill their main function and zap the electronic versions into oblivion.
At some point in recent history, someone decided the staffing process that has served the military in various incarnations for most of this century is now a millstone around our necks. Whether they reached this conclusion on their own or with the help of the marketing machines of various workflow tool vendors, computer pundits or at a management seminar, I don't know. Whatever the reason, apparently a large number of people are quite convinced we need new workflow technology, right now, to solve our problem.
Before we go install any software, though, let's define the problem.
Welcome to My Nightmare
As usual, my first problem with workflow came in an e-mail from Zippy - and not just any e-mail. Apparently he and Zippette had built a Web-enabled database to act as a workflow system for the annual (and completely unofficial) office Super Bowl pool.
Like most of their efforts, this one was a masterpiece of hacking, complete with an interactive matrix where you could fill in which blocks you wanted to buy. Money never changed hands on government property, of course. Zippy may be a few fries short of a Happy Meal, but he's not foolish enough to actually run an overt gambling operation, even one as benign as an office pool, over a government computer network. For all outward appearances, this particular application was a "test pilot" of interactive matrix development.
As usual, Zippy developed the process models, and Zippette handled all the technical details. In his favor, he got most of it right. The problem is that Zippette can pretty much code anything Zippy can think of into a working application. With the right input she could probably crack the code for the human genome.
With Zippy's input, however, all we normally get is cracked code.
A football pool is usually pretty simple. You take a 10x10 matrix and have everyone pick blocks until they're full. Then you randomly assign numbers to the X and Y-axes. At the end of each quarter and the game, if the last digit of each score matches the two numbers that intersect one of your cells in the matrix, you win a piece of the pot.
The model was far too simple for Zippy. To those of us filling in the matrix, it looked like we expected. However, with his usual excruciating enthusiasm, Zippy built in a few dozen other winning combinations. These apparently included payoffs based on your cell being a number that could factor into the jersey number of anyone scoring a touchdown, the relative angle of the moon from Atlanta when a field goal was kicked, and a host of other odds so bizarre even Jimmy the Greek couldn't figure them out if you gave him a Cray and a team of MIT mathematicians.
On game day, Zippette somehow connected to all the NFL network television and space tracking systems needed to verify the winning conditions. At $1 a cell, the money in the pool came to $100. As the game progressed, quirks of fate of a probability so low that they could only be considered miraculous, split the money 573 ways between 21 players.
Fortunately, no one really expected his system to work. We figured $5 each was a small price to pay to watch Zippy's brainchild self-destruct.
But it was clear that adding in a host of extraneous features (a.k.a. "requirements creep") can doom even simple, well understood projects. It served as a reminder of the old maxim: "A fool with a tool is still a fool."
What Is Workflow?
"Workflow" is one of those terms someone coined based on the idea that work "flows" through an organization. The visual image suggested by the term is supposed to be a boat (the work) flowing down a stream or river with the current carrying it smoothly towards some ultimate destination.
In the military when people talk about workflow they are often referring exclusively to staff work. While staff work is a workflow process, it is not the only one. Any activity that can be explicitly defined and modeled, from ordering a chair to submitting a performance report, can be automated with the right system.
The people clamoring loudest for relief at the moment, however, are mostly involved in staff work - and with good reason. Unlike that boat on the river, our staff work frequently looks a lot more like a salmon trying to swim up a waterfall.
Defining Workflow
Workflow is, first and foremost, about defining processes. Workflow process models normally have two levels of representation. The first level defines the process of the flow of work through the system. The second level represents the state of execution of a particular process, usually described as an "enactment."
For the purposes of workflow modeling, we'll assume a process consists of a name and a sequence of activities needed to execute the process.
At the next level in the process, an activity consists of a name, a set of roles and actions associated with that activity and usually a completion condition.
One thing to bear in mind at this point is some activities contain enough complexity to qualify as complete processes in and of themselves. These key activities are encapsulated within the larger process and are often the critical components whose performance will make or break the entire system. It's quite possible to model the larger process correctly but then have a workflow system fail because of a lack of sufficient detail at the activity level.
My personal preference when modeling activities is to bound them within single roles. A role is what an actor plays in the execution of an activity. An actor can be a person, a group of people, a computer program or an organization that provides the functionality for the system by executing activities.
If we define our actors' roles clearly for each activity, we'll have a better chance of defining and meeting the specific requirements of those activities. Each activity should consist of the actions executed contiguously by a single role. When the role changes, we've entered a new activity. We should be able to reduce just about any process to manageable chunks for implementation with this method. We'll talk more about roles later.
An action is a description of something to be done. "After coordinating, forward this to the next reviewer," "send an e-mail to acknowledge receipt," and "apply digital signature" are all examples of specific actions that may be executed by a person or software agent.
The tedious part of building a workflow system is ensuring that every action executed by every actor in every role for every activity in the process is accounted for somewhere in the workflow system. Each action should also satisfy an argument ("if this, then this; if that, then that") that generates further action based on a perceived (either by human or machine) value.
That's the process level. At the execution level, each iteration of a workflow process is considered an "enactment." This consists of a unique enactment ID number, a reference to the process being enacted, a reference to current activity within that process, binding actors and roles to activities and matching arguments to values.
In plain English, every time we send work through the system we should know the following at any given point in the process:
Something that uniquely identifies each work package (ID number).
What action are we performing? (Ordering, coordinating, modifying, etc.)
Where are we in the process? (How many activities have we completed?)
Who or what is currently involved in the action?
How is the work proceeding? (Are the actors responding appropriately?)
When is the work complete? (At any level: action, activity or enactment.)
Here's the scary part: as complicated as it appears, the description above greatly understates the difficulty of actually modeling a process with the intent of building workflow around it. In addition to explicitly defining every action in the process, particularly the ones to be automated completely, we must also define the networks of activities that specify parallel, iterative and conditional interactions between actions, actors and activities.
One thing that helps is that workflow enactments are case-based. Every piece of work we do, for example, is executed for a specific business case and almost always for a specific customer. The goal of workflow management is to handle the enactment of those cases as efficiently and effectively as possible.
So, we design workflow processes to handle similar types of cases, with workflow management being what we do to try and shepherd our work through our organizations. With that basic understanding, we will now move on to workflow management systems.
Automating Chaos
The Workflow Management Coalition defines a workflow management system as: "A system that completely defines, manages and executes workflows through the execution of software whose order of execution is driven by a computer representation of the workflow logic."
On a slightly higher level, workflow management systems support the definition, registration, execution and control of processes. This is a fairly broad definition - just about any automated production system fits here. But, as we will examine later, workflow is more than simply staffing or transaction processing.
Unfortunately, workflow management is often thought of only in the context of computer-mediated workflow management systems. While computer-based workflow systems are becoming more common, this unnecessarily limits the application of workflow management as a discipline. It is entirely possible to do workflow management without using a workflow management system and improve the work process without making any change at all in technology.
However, computer-mediated workflow management systems are becoming the rule rather than the exception. I think it's safe to say if we're going to do something about the glut of staff work clogging our organizational arteries, it's going to take some technology changes to support it.
In the commercial world, workflow systems are used to support a variety of business processes. A classic example is the system your local video rental store probably uses to manage its business. It encompasses account management of customer information, inventory control of the movies and a host of other functions associated with running a multi-billion dollar business with a staff consisting in large part of high school students working part time.
We should be so lucky as to achieve that level of simplicity and ease of use in any system, let alone workflow.
Now, with the understanding of workflow and workflow systems, let's move on to how workflow management specifically applies to our current challenge: automating staff work.
Charity Begins At Home
In a military headquarters, regardless of the mission of the organization, the core function of that headquarters is to make decisions and give direction. Therefore, we have staffing processes dedicated to giving our decision makers what they need to make those decisions and give direction. The decision-making process is as crucial to a headquarters as the aircraft maintenance process is to a flying unit.
I've seen many attempts in military organizations to automate all or part of the staffing process used to support decision making. They range from simple databases or spreadsheets used to track the work in an office to full-scale, base-wide systems that try to account for everything on the base. Most were developed locally using commonly available tools to avoid the cost of buying a "real" workflow management system from a commercial vendor for millions of dollars.
One of the lessons learned from these development efforts is that any workflow system implementation will be limited by our understanding of the work process. Just what is staff work and how should we automate it?
As people are the principal processors of our staff work, let's start by defining the various roles in the staff activity. There are two basic roles: administrative and business.
Administrative roles include initiators, monitors and channelers. Initiators enter tasks into the system, monitors keep track of the work as it progresses and channelers redirect or delegate work.
On the business side, we have workers, reviewers and approvers. Workers collect, create, transform and integrate the various information objects that form the input and output of the system. They produce the product from which the system derives value.
Reviewers are intermediate approval levels. In many cases, reviewers are the same people who channeled the work downstream to the worker. Their role simply changes as the direction of the work shifts. Review activities include quality control (editing, fact checking, etc.) information aggregation, filtering and, in some cases, censorship.
Approvers can approve the closure of an action. There may be some mechanism for appeal by other levels, but at some point each action should come to an end.
There may be other roles depending on the business process, but these are the standard ones I've observed so far.
Who's On First?
Now comes the fun part: defining the requirements people (or automated subsystems) executing these roles need to function effectively.
While I've seen requirements developed based on functional descriptions of different types of work, I recommend we model our requirements based on the activities actors must perform to fulfill their roles. For the staff process, this means either having the system give these actors the information required for their role or automating all or part of their role into the system.
Administrative roles don't require subject-matter experts. You don't have to know anything about the work itself to know who's working on the project, when it's due back for final action and where it is in the coordination process.
Administrative requirements might include:
Having the "subject" as a mandatory field.
Sending an e-mail reminder to the person currently working on the project three days before it's due to the next level.
Showing everyone who has been assigned to work on this project.
Showing who can be assigned to this project.
Of those requirements, the first three can be completely automated. Remember when we listed "matching arguments to values" as a function of an individual workflow enactment? Well, this is where you apply it most: during automation.
A computer can check to see if the subject block has been filled in, but if the entry was gibberish, it would never know. But as a condition of initiating an enactment it will know, absolutely, whether or not this field has data entered. We must then depend on the initiator to make the entry meaningful to everyone else involved.
The next two are also based on simple conditional arguments and explicit values. The workflow system should be able to identify both the due date and who is currently responsible for taking action and sending a "nag note."
The last requirement, however, is an action normally completed by a human. Yes, if the system knows enough about who has done what work in the past it can assign all the computer security projects to Tim in the network center. But unless there's only one person in the organization who does all the staff work, we're most likely going to reserve the act of assigning or delegating to a human.
Our business role actors, however, do need to know something about what we're working on. They transform questions into answers, or frequently in the military, observations into command decisions. Requirements for business roles might include:
Showing who else has been assigned to work on this project.
Using a polling/voting mechanism to facilitate documented consensus between action officers.
Having a version control mechanism for group editing.
Linking to the organizational records archive.
The first, which supports building virtual teams, is fairly straightforward. The system should be able to identify the current actor at the lowest level of each delegation chain and build a list. This is useful - the only way to do this outside a workflow system is to personally call or e-mail every office working the project and ask whom they've assigned to it.
The second requirement, however, rises to the level of a process all by itself. Achieving consensus is the key activity in coordinating staff work at any level. This is where we spend the most time and effort, and it is the activity most likely to make or break a particular enactment (project). We can be on time and go through all the right motions, but if we don't build the decision properly, we might as well not bother doing it at all in most cases.
The Coordination Game
Now we'll dissect a typical coordination process. Someone writes a policy document that has to be coordinated across every subunit at every level: branch, division and directorate. The document works its way down to the actor performing as the integrator.
At this point, the document is disseminated across the different branch offices via paper, e-mail attachment or as a file shared by all from a central location. Whatever the method, two dozen people are now busily involved in reviewing and commenting on the content of the document.
Ideally, our test document is well written and meets everyone's expectations. However, one or more of our collateral reviewers usually has input on some of the content.
The fun begins when our integrator receives all these comments. Maybe somewhere, sometime, some group makes their comments concurrently on one shared copy of an electronic document. This would constitute a virtual discussion as part of a group review.
I have yet to see it personally, though. I treat reports that this actually occurs in nature with the same skepticism I reserve for Loch Ness monster sightings or paid political advertisements. At the end of the review period our integrator is usually presented with two dozen individual inputs and must somehow meld them into the original document to produce Version 2.
As this version is essentially a new document, it gets re-coordinated across the organization again. It's entirely possible for some of the input from different reviewers to conflict and for changes submitted by one reviewer to be unacceptable to another. The process of global review continues until everyone on the same level comes to some agreement (or are beaten into submission) and the document can progress up to the next review level.
We also touched upon the third sample requirement: version control. As with any process, some activities facilitate others, like version control facilitates the prime activity of consensus building. Identifying and mapping these activity relationships is an important and often overlooked part of the process modeling effort.
The last sample requirement, linking to our archives, recognizes how people frequently need to recycle older information to answer the current question. Reinventing these wheels can be both time consuming and frustrating. Even if the link is to nothing more than an index engine pointed towards our archive, it can save workers a lot of independent, unstructured (and possibly clueless) searching.
Please bear in mind that our sample list is a very small subset of the requirements I'm used to seeing for workflow systems. While most lists include dozens of requirements, I have yet to see one over the last six years that actually appeared comprehensive.
Congratulations on making it this far (unless a friend told you to skip ahead to this point) and getting through the admittedly lengthy discussion. At this point we're ready to finish this topic discussion by tackling four important issues associated with workflow system development.
1. Don't Let The Inmates Run The Asylum
This advice is based on the Alan Cooper book of the same title. Cooper submits, among other observations, that software is difficult to use because software designers typically focus on the nature and needs of the computer, rather than the nature and needs of the user. This produces feature-rich software to either conquer or try to avoid.
Cooper states the main reason we've achieved this level of system dysfunctionality is despite appearances, business executives are not really in charge of the software development process. The engineers run the asylum. With software, nothing is normally visible until it's been coded, which means that input from non-programmers in the user acceptance phase is usually too late to make any effective changes.
In addition to the inherent difficulties with using software built by people who do demoscene coding for fun, there is another group of inmates to deal with: the people who administer the work process. Sometimes knowing too much about the current process can get in the way of improving it. More often than not, they want to keep all the artifacts (even if they're dead weight) in the new, automated process.
Then they wonder why the rest of us never seem to like the "new and improved" system.
Electronic forms come to mind, particularly electronic forms that we do nothing with except fill and print. If all the information I need to coordinate a staff action is in the workflow envelope, why would I need to attach a redundant chunk of information in an electronic form? Instead of routing a fill-to-print form with my computer systems requirements, why not replace it with a Web form tied to living databases by a true electronic work process dedicated to receiving, sorting and satisfying our computer requirements?
Albert Einstein said:
"We cannot solve the problems we have at the same level of thinking we were at when we created them."
This will be particularly true of automating (dare I say re-engineering?) a process with a workflow management system. If we're not willing to raise our level of thinking above inmate level, no amount of technology will help.
With that in mind, my second admonition is:
2. Build the User Interface First
Let's face it: to 99 percent of the people who come in contact with a workflow system, the user interface is the system. They have no interest in the database, the SQL query structure, the process model or even what software they're using. They just want a simple uncluttered interface that tells them what they want to know and lets them take action without having to read a 51-page manual.
Anything beyond that becomes an annoyance.
So, design the input and report screens first and then build system functionality to support them. Rather than belabor this point on my own, I'll indulge my fondness for quotes here with two that apply to this situation. The first is from Michael Simpson, Chairman, A.M. Castle & Co:
"Anything that the customer is not willing to pay you one penny more for, you have to question whether you should be doing it. There's no point in doing something extremely well if you shouldn't be doing it in the first place."
The second is from Spanish writer Enrique Jardiel Poncela:
When something can be read without effort, great effort has gone into its writing.
These are good words to live by, particularly in interface and system design. The best engineered system in the world will still fail if the interface is ugly.
3. Acquire Tools Instead of Building Them
Given the difficulty of actually modeling our staff process to the necessary level of detail, I'm continually surprised that people keep trying to build their own workflow systems out of software that wasn't originally designed to automate work. I can understand their motivation, though.
First, they think commercial systems cost too much. No argument there, as most proprietary commercial workflow management systems will run into millions of dollars for a full implementation.
Second, they think they know enough about how their work flows to build a bet ter system on their own. While you've got to love optimism, please see the previous reference to inmates running the asylum. If the software designers and the user community have never done workflow activity modeling before, it's going to be an interesting journey.
Third, our core military competencies don't usually include building office automation systems. Why would we try to reinvent that particular wheel? We don't build our own database management systems because we're not in the database management tool business. We just use existing tools to build our databases based on our understanding of the data.
Yes, commercial workflow toolkits are expensive. However, the companies that sell them have already paid the cost of development and have accumulated some experience at fielding them. Instead of reinventing wheels, let's just attach them to our car. Being an expert on tool making and on tool use are two entirely different things.
One other thing to consider: do we want to save money or make money? If we choose the least cost alternative, we avoid spending a certain amount of money. But if the investment produces only a marginal (or no) gain, what have we achieved? Instead of developing each workflow management system separately (staffing, travel and purchasing, for example) using different tools, approaches and databases, maybe there is an advantage buying one standard set of development tools and leveraging them across all our workflow applications.
If we don't like the cost today, we may shortly have another alternative: an open-source workflow toolkit.
Vivtek has been commissioned through SourceXchange to develop an open source workflow toolkit. SourceXchange is a forum that links open source developers to investors with well-defined, financially backed desires for open source products. The company backing the development of the workflow toolkit is Galactic Marketing Incentives, Inc.
If they are successful, we can use the toolkit for free instead of paying costly licensing fees for proprietary systems. Of course, this only deals with one cost area - licenses - and not support, system development or process analysis. However, the licensing cost is a significant percentage of commercial systems, and the choice of systems can restrict you to a fairly narrow range of system providers.
An open-source workflow toolkit would be the equivalent of Linux for workflow. Any company or organization, including our own military systems development community, could acquire and use the toolkit for free. Instead of building proprietary, stovepipe applications for every business process, we could develop all of our automated processes using the same free toolkit across our bases, services or even the entire Department of Defense.
Maybe I'm dreaming at the moment. The big commercial workflow and transaction system vendors probably aren't quaking in their boots at the mere mention of open source competition. However, many of those same vendors have released or are releasing Linux versions of their software. Even IBM recently announced it would be using a version of Linux on its mainframes. Organizations with lots of money on the line are starting to buy into the open source movement.
If you want to hedge your bets, I recommend, "The Cathedral and the Bazaar," by Eric Raymond, one of the founders of the open source movement. Raymond details the differences between the commercial (cathedral) and open source (bazaar) approaches to software development and why he thinks open source development has a very viable future. You will find this and other Raymond essays at: www.tuxedo.org/~esr/writings/
If you're interested in Vivtek's open source workflow toolkit project, you'll find Michael Roberts' musings on his experiences trying to build the workflow tool kit at: www.vivtek.com/wftk.html. Information on SourceXchange is available at www.sourcexchange.com.
4. The Product Is More Important Than The Process
Here's a quote attributed to Peter Drucker, one of the leading management gurus of our day:
"To start out with the task rather than the end product may result, however, in beautiful engineering of work that should not be done at all."
As we set out to automate anything - staff work, planning, purchasing, whatever - if we only focus on the tasks involved in the process and not the product of that process, we will never make a significant difference in the outcome.
Let's refer back to 4,600 words ago, where I likened workflow to water flowing down a river. Left to its own devices, a river will find the path of least resistance and flow smoothly along. We can divert the river or dam it up, usually at considerable expense and effort, to achieve some end result. However, we will have to exert some effort over time to maintain our constructs. If we don't, the river will eventually return to its original state.
Instead of struggling to maintain all of the artificial constructs we've attached to our work processes when we automate them, perhaps we should concentrate on the output and outcomes associated with our work. We can be lazy, in the good sense and strip away all the artifacts that don't add value to the process and let our systems and our people do what they do best: get the job done with as little hassle as possible.
Subscribe to:
Posts (Atom)
