Zen and the Art of Systems Analysis

Meditations on Computer Systems Development

by Patrick McDermott

7. METHODOLOGICAL MINDFULNESS

“Begin at the beginning, and go on till you come to the end: then stop.”—K♥The King of Hearts in Lewis Carroll, Alice’s Adventures in Wonderland

To have the Right Mindfulness you need the right methodology—Being ever mindful of the correct practice toward Enlightenment. Chapter 7 talks about methodologies and techniques. In the simplest analysis, a methodology is just an approach—a combined recipe, checklist and tutorial to help get through a project successfully. As with most things in life, the Middle Way is best: having no methodology, and slavishly following a methodology, are equally bad.

The Power of Babel

First we need to get a little terminology straight, since this is a case where there is a many-to-many relationship between terms and concepts, to use the vernacular. The bewildering variety of different terms used by various analysts and gurus impedes communication. In hopes of reducing the confusion, I’ve prepared the little chart below. You’ll see how the different terms are used for the same thing and the same terms are used for different things. That’s exactly what we Data Administrators keep telling everyone else they should never do! Oh well, as I keep telling you, there are many ways to the mountaintop; so I suppose we should stop being so sanctimonious about inconsistent data names throughout our systems.

At first I called the first column “camp” thinking the terminology set was associated with a group of people, but I later realized it also depended on context, as I caught myself using terms from the various camps throughout a Project’s life cycle. For example, when you’re referring to a database table it won’t do to call it an entity, it’s become a table.

[   [   [

I should say a few words here to mollify the Objective Orientalists (i.e., strict followers of Object Oriented methods) in the audience. You will no doubt be saying to yourself “Patrick’s trying to say a class is just an entity in disguise”. (Actually, I am, but I use this subterfuge to befuddle the O/O zealots—shh, don’t tell them my secret). We need to make a distinction between classes that take advantage of the O/O programming mechanisms and those that are about the business. Many objects are not part of the problem domain, which is to say, the business user wouldn’t recognize them. What you need to model is the business, so call the category business objects, or entity classes if you like. For data modeling, this is ALL you care about: the rest are just clutter. We are primarily concerned here with business objects, the classes we’ll be looking at are entity objects, so I’ll call them entities for short.

In truth Object Oriented methodologies might have been better named “Class Oriented”, since we are typically focusing on classes, which are categories, and not objects, which are individual instances of the class. In C++, you use the keyword class, not the nonexistent keyword object, to use OOP features. Maybe they didn’t want the acronym COP, which after all is better than Oops.

If you think this discussion confusing, you should attend one of my classes in C++ programming that covers O/O classes. I use a school registration system as an example, in which one of the O/O classes represents a school class (as in a course), so find myself saying things like: “Now class, in this class, the Class class will be represented by this classy Class class.” So I’m afraid I’m not convinced when I hear OO advocates say “Object Oriented is more like the way people think”.

Religious Wars

A methodology can be too rigidly followed, in that the practice becomes the goal, which is usually unfortunate. Some consider a methodology to be Holy Writ, and some practitioners would sooner change their religion than their methodology. These people believe those who use the wrong methodology are not only mistaken, they’re evil. I’m certainly not one of those people, and you shouldn’t be, either—there are many paths to the mountaintop—and there is no one true methodology. Further, take a loose interpretation of what is required of a methodology—a methodology must be freely adaptable to the problem at hand.

The early Christian missionaries were pleased when they easily won converts among the Japanese, who then earnestly requested crucifixes to take home to keep in the place of highest honor. But they were shocked to discover the place of highest honor was right next to the statue of Buddha. The Japanese saw no conflict between the two religions and so had no problem being both Christian and Buddhist. If you ask a Japanese “Are you Shinto, Buddhist, Taoist or Confucian”, the answer will probably be “Yes”.

When it comes to methodologies, be like the Japanese. If a formal methodology is followed too strictly, it can become what Tom DeMarco & Timothy Lister’s Peopleware calls a “Big M Methodology”. They cite the example of a methodology that required an 18-part operator’s manual, so the project team produced one, even though the device in question was going on a satellite and no one could “operate” it without going out in space. Ed Yourdon, a famous methodologist himself, in Death March speaks of techniques to circumvent the “Methodology Police”, even recommending threatening to resign on the spot if forced to use a cumbersome methodology.

[   [   [

Karate is learned in three stages, called Shu-Ba-Ri. At first, the student must strictly follow forms (Shu) that minutely detail prescribed sequences of moves called kata. Each motion must always be conducted precisely according to the ancient ritual, and one’s progress is evaluated based on how faithful one is to the form. Only after mastering the kata are you allowed to ignore (Ba) the exact sequences, experiment, and choose your own techniques. Eventually you are allowed to transcend the rules entirely (Ri), and develop your own strategy in complete freedom. This is a good way of looking at methodologies. When you are inexperienced, it’s best to follow a methodology scrupulously. As Geoffrey James says in The Tao of Programming: “What is appropriate for the master is not appropriate for the novice. You must understand Tao before transcending structure.” After a little experience, feel free to mix and match techniques, and skip steps as you see fit. When you’ve reached the higher level of Ri, you can develop your own techniques without regard to established methodologies, establishing your own methodology, even write a book on it!

No successful methodology absolutely requires every step be completed. But skipping any step brings risks that might be avoided. Beware the practice at many companies: “We never have time to do it right, but we always make time to do it over”. Never break a rule out of ignorance, but rather after careful consideration of the situation. For some Research & Development projects a project plan is almost silly since you have no idea where you’re going, when, or how. At the other extreme, in routine development, an established team can skip many of the steps. The key to successful shortening is constant vigilance to detect and correct problems as they develop.

Even writing a computer program follows this pattern. When you are first learning, you need to diagram, pseudocode, walkthrough, and perhaps even flowchart the simplest programs. When you have become an advanced programmer, you can skip these steps if you deem it safe. Eventually, you might have occasion to write high-performance code that completely violates all the usual rules of good programming.

Sashimi isn’t SAD

In this section we’ll compare the traditional method, unfortunately labeled SAD (Systems Analysis & Design), with a variation suggested by Nonaka & Takeuchi in The HBR article “The New New Product Development Game” which has come to be called the Sashimi approach. First let’s consider the traditional methodology built around the SDLC—System Development Life Cycle, in combination with a Yourdonesque (Much of this methodology was developed and popularized by Ed Yourdon and his associates) Current Physical → Current Logical → New Logical → New Physical analysis cycle as being “traditional Systems Analysis and Design” methods. This life cycle is often pictured as a waterfall, with the product of each phase cascading down to the phase below. Some see the diagram as a staircase or escalator; Those who see an escalator usually feel the escalator is running against them, not with them.

The cycle begins with a feasibility study, essentially just the step in which the project is approved; it ends with funding for the project, or at least for the next step. Analysis is where we figure out what the business problem is and what problems we need to solve, and we produce the requirements for the new system. In design, we decide how we will solve the problem, producing specifications, or specs. Coding produces the actual system, which we test rigorously, unless we’re like Microsoft and leave all the testing to our customers. Installation will include training, data conversion, and software installation. New work rules will have to be implemented, and psychological resistance from the users will be at its height. So this phase might turn out to be the most critical of all—it’s your last chance to really blow it, so don’t give it short shrift.

Most OOAD advocates reject the waterfall, calling it part of the problem, not a solution. The Three Amigos, Ivar Jacobson, Grady Booch & James Rumbaugh, for example, describe the Unified Software Development Process in a book of the same name as Use-case driven, Architecture-Centric, Iterative, and Incremental (—quite a mouthful). The life cycle is variously pictured as a spiral (Hopefully, not a death spiral!), or a mandala, or a fountain, or a cycle within a cycle. No matter how you slice and dice the phases, though, the activities are the same. All experience all steps whether they admit it or not, but OOAD tends to emphasize a series of small steps (iterative & incremental) as opposed to the big sequential phases of the SDLC. The nature of O/O is that one object might be completely implemented before another is even started, so you might be doing parts of all the SDLC phases at any given moment. This parallelism is seen as a great advantage, and is believed to mitigate the Rayleigh effect seen in traditional projects with uneven staffing over the life cycle, rising as analysis and design crest, peaking with the coding phrase, and then dropping off for installation as the project wraps up and winds down.

This nonsequential approach is most typified by Nonaka & Takeuchi’s Sashimi life cycle. Sashimi is a Japanese dish, similar to sushi but without the vinegrated rice. Sashimi cooks spend many years mastering cooking techniques, which seem deceptively simple since the cooking consists of NOT cooking the fish. In any event, the fish is usually sliced, and arranged on the plate with each slice stacked upon, and overlapping, the others. Likewise, system development phases can be stacked upon, and overlap, each other.

The successful new product development teams in the leading companies in Nonaka & Takeuchi’s study had the following six characteristics:

1. Built-in instability. Teams were given broad goals or general strategic direction, not a clear-cut product or work plan. The teams had challenging goals, and great freedom, but no specifics. Paradoxically, ambiguity can be a great clarifier.

2. Self-organizing project teams. This requires autonomy, which means the team is truly empowered, with top management rarely intervening. By truly empowered, I don’t mean the oft-used definition seen in many companies where empowerment is a buzzword: “You are empowered to make any decision, as long as it’s the same decision I, the manager, would have made.” The ideal manager has an open wallet and a closed mouth.

3. Overlapping development phases. As the Sashimi cycle shows, several, maybe even all, phases are proceeding simultaneously. This avoids the throw-it-over-the-wall syndrome that happens when an analysis team spends months developing requirements that are thrown over the wall to the design team that spends months developing specs that are thrown over the wall to the coding team.

4. Multilearning. Both multilevel and multifunctional learning must be encouraged. Individuals, teams, and the company as a whole should be encouraged to learn. And in keeping with Musashi’s fourth dictum—Know the Ways of All Professions—you need to be open to learning from other fields. The most critical is technicians need to learn about business and economics, and nontechnical team members must learn technology.

5. Subtle control. If the team is to have autonomy, management must avoid the temptation to take direct control. Such tactics as balancing the team’s members, tolerating mistakes, understanding the rhythm of the cycle, and encouraging direct customer contact can be used to subtly control the team’s direction.

6. Organizational transfer of learning. In addition to the multilearning within the group, it is important the lessons they learn spread throughout the organization. One way to do this is by osmosis: disperse the members of the successful team among other teams to serve as examples.

Which Methodology is Best?

As usual: “It Depends”. The control system for a 500-passenger aircraft might need somewhat more rigor than a video rental system.

Stephen R. Schach in Classical and Object-Oriented Software Engineering finds the Waterfall model a more disciplined approach, achieved by the requirement to produce documentation at every step. The problem can be, to quote Schach: “In general, specification documents are long, detailed, and, quite frankly, boring to read”. As a consequence, the delivered product may not meet the client’s needs. He notes that iterative techniques characterized by Object Oriented methodologies have the advantage of incrementalism and parallelism, but warns they can degenerate into an undisciplined CABTAB (code a bit, test a bit) approach that might never meet the need, either.

In general, either approach can work, and either might be preferable depending on your staff and your project. If your staff is thoroughly familiar with and enthusiastic about a certain methodology, you might do well to use that approach unless project considerations preclude it. As to projects, large, well structured, familiar problems will probably succumb to traditional approaches, while small projects, and projects dealing with unstructured or groundbreaking advances will probably be better with Sashimi.

Brain Typhoon

Brainstorming is one of the most useful tools to stimulate creativity available to the Analyst. I use it almost daily in analysis and in all creative work. Brainstorming is a process you probably have done unconsciously, but it improves with some formal rules. I suggest you post them in a chart form in meeting rooms.

1. No Criticism

This is the most important rule of all. If someone offers the stupidest idea you ever heard in your life, the correct response is to say, “Fantastic!” Under no circumstances do you want to stop the creative flow by intimidating anyone. Some of the craziest ideas you’ve ever heard turn out to work once given a chance. But more important, a dumb idea can flow into another better idea if you give it a chance.

2. No Self-Censorship

This is almost as important. Don’t hold back. If you think of the stupidest thing you’ve ever heard of in your life, blurt it out. Sometimes you are able to come up with an idea which is completely absurd, except that it works—because you look at a problem from the odd perspective—you “think different”. This often lets you attack a difficult problem from another angle, one that leads to a surprisingly elegant solution. Even if it isn’t adopted, it might start someone else thinking, or at a minimum will get a laugh and help to loosen up the group.

3. Piggyback

Hey, for once in life stealing is okay, in fact encouraged. Try to modify or expand on the ideas offered by the other participants. And if someone steals your idea, remember those who see farther are standing on the shoulders of giants, and that means you’re a giant. Ideas feed on other ideas, and piggyback is not only okay, it’s admirable. You want to take every idea to its illogical conclusion.

0. Have Fun

Although this one appears first on my chart, I have PowerPoint animated to have it appear last when I’m explaining the rules to a group. I explain that as a C++ programmer I naturally count from zero, not one. This rule is first because I’ve found that groups that are having fun usually do the best job.

I often use a silly brainstorming exercise as an icebreaker, a way to loosen a group of people up, especially if they don’t know each other. You might consider something similar for a meeting of a new group, especially if you will be using brainstorming and need to warm up. Explain brainstorming, form the group into teams of four to eight people, and then pose a silly problem to the group, such as this one I often pose at my UC Berkeley Extension class:

Berkeley is in a terrible fix. With the rainy season approaching, we decided to put coat racks and hangers in all the classrooms and offices. It seems a new employee went on to CoatHangers.com on the Internet, and like many UI’s, this one was confusing, so she accidentally ordered not the 5,000 hangers wanted, but 5,000 cases of hangers, with 10,000 hangers in each case. There is no way to cancel this order. Your mission is to come up with as many possible uses for these hangers to get Cal out of this jam.

If asked “what kind of hangers, wood or wire?” Tell them “Every imaginable kind”. This further opens the road to thought. You can vary the problem by using just about any item, although I tried it once with clothespins and it didn’t work very well. Some of the students didn’t know what a clothespin was!

Without telling them what I’m doing, I tally the number of times I hear laughter coming from each group. When we get to the reward phase (see below) the first behavior I reward is the “Fun Group”, the one that laughed the most. In many learning situations, noise equals learning. For my course on systems analysis consulting, I consider the number of times the neighboring classes complain about the noise an indicator of how well things are going.

Quality will come from the process. The point of the exercise, and of brainstorming, is to generate as many ideas as you can. With more options, you are more likely to come up with a good solution. It’s easier to cross out the bad ideas than to come up with good ones.

The Brainstorm Process

1. Brainstorm

Discuss the rules, and perhaps do an icebreaker, such as the hanger exercise, or any from the Games Trainers Play series of books. Someone needs to be the scribe, and record all the ideas on a white board or flip chart. If your session is facilitated, the facilitator might be the scribe, and not participate. So just have the people shout the ideas out, with the caveat the scribe(s) need time to record them. With all meetings, especially free wheeling sessions with lots of ideas and decisions, this rule applies:

2. Reward

The next step is to reward the best performers. The reward can be a cheap gag toy, some candy, or just a pat on the back. In my class, the students in the rewarded groups get an extra point toward their final grade. All students receive one point for participating in the class exercise, plus each student gets an extra point for each category they “won” in. Often one group wins in more than one category—the fun group especially seems to win in one or both of the other categories.

These are the categories rewarded: 1. Most Fun—the group that laughed the most. 2. Most Ideas—the group with the greatest total number of ideas. 3. Most Bizarre, Strange or Ridiculous—the group with craziest idea. Note you should NOT reward the best idea, nor the one that is eventually chosen to work on. The best idea arises from the process, and everyone was part of that. This is to encourage the correct behavior. If you want people to think out of the box, reward them for escaping.

3. Reality Check

Now reality must raise its ugly head. If your boss has said, “Under no circumstances will you go to Hawai'i”, then Hawai'i, alas, must be cut from the list. If you’re doing brainstorming right, you will have come up with some things that are just plain impossible. But remember, what seems quite absurd can in fact be quite feasible. After all, if the infrastructure weren’t built, the idea of the ubiquitous automobile would seem absurd. Not just to make the autos themselves, requiring materials from all over the world, but consider the requirements: You must build a ribbon of concrete from every place to every other place, with gas stations strategically located dispensing oil that comes from deep under ground in the far corners of the planet. It will never work! So only remove ideas that are patently absurd, illegal or clearly impossible given company policy or politics.

4. Clean the List Up

Categorize and combine items. By classifying them on some characteristic (brainstorm a list…), you’ll find the same idea appears twice, or will see a hole or a whole. Try fusion and fission: some ideas combine, some split into two. Combine any two ideas that are the same, or split any cases where two ideas are posing as one.

5. Select

Hopefully you will have more ideas than you can possibly work on. So how to decide which merit further investigation? There are essentially three ways:

Autocracy—Some expert or authority shall decree the answer. Perhaps your assignment has been to come up with alternatives, not recommendations, and so you can pass the problem to the boss.

Democracy—or a modification of it. You could simply vote, or better, use a variation called multi-voting. In multi-voting, it’s a weighted democracy, because each person gets multiple votes. How many? Usually three or four votes, but sometimes a good number is the number of alternatives you want to narrow down to. If you plan to look at six alternatives, you might give each participant six votes. Votes can be “blocked”, i.e. you can cast several votes for the same choice, even cast all your votes on one choice if you really feel strongly about it. Multi-voting avoids several problems. Sometimes a lot of people are mildly in favor of an idea but a smaller number are very enthusiastic about another choice. Their enthusiasm can be registered by extra votes. Sometimes everybody’s second choice is the best answer, which would be missed if everyone got only one vote.

Consensus—this involves getting broad agreement. It is covered in Chapter 3 on Consulting. Consensus is preferable when the success of the decision depends on cooperation and coordination.

Brainstorming works for either a group or an individual. Use it alone, use it with friends. Use it whenever you need to think out a problem or generate alternatives.

Naturally Normal

With Zen Analysis, you don’t need normalization. E.F. Codd of IBM originated the concept of Normalization. Normalization has fallen out of favor, or at least fashion, in current practice. The Object Oriented design movement seems to have left normalization behind; I’ve seen dozens of O/O books, and I recall none even mentioning it. The Three Amigos (Booch, Rumbaugh and Jacobson—the inventors of UML) don’t cover it in any of their comprehensive tomes. In academe, Database textbooks usually do cover normalization, and three leading Systems Analysis & Design textbooks I checked (Hoffer et al., Kendall & Kendal, and Whitten) include coverage, but four Software Engineering textbooks (Sommerville, Hamlet, van Vliet, and Braude) do not, despite the fact good software engineering requires a normal database. One database design author, Robert J. Muller, Database Design for Smarties: Using UML for Data Modeling, even rants about it, calling it an unnecessary ritual, saying some DBAs “insist on going though the stages of normalization and arguing Talmudic points along the way. Pilpul aside, (I had to look this one up in an unabridged dictionary. Pilpul is Aramaic (Hebrew) and means a dispute over subtle details, especially in the Talmud.) normalization is a ritual, and often not a particularly useful one”.

The normalization ritual can help clarify thinking and avoid many errors. Doing a normalization review requires you consider every individual attribute, why it’s there and what it does. Such a painstaking review, although tedious, can pay off by uncovering mistakes—even many unrelated to normalization as such.

Most normalizers strive for third normal form (3NF). There are also three further normal forms, 4NF and 5NF, and Boyce-Codd, which falls between 3NF and 4NF: 3½NF?). In this discussion we’ll limit ourselves to 3NF, which is sufficient for any real-world database you’re likely to encounter, and because they are usually beyond the ken of even the most advanced Zen master. To be in third normal form (3NF), a relation must be in second normal form, and have no transitive dependencies. So what does that mean?

The various normal forms build on one another: to be in 3NF you must first be in 2NF, and to be in 2NF you must first be in 1NF. So what’s first normal form? To be in 1NF, you need to have a key, which is a unique identifier, present in every tuple (A tuple is a fancy word for row, which I avoid because I can’t remember whether to pronounce it tuhp-ul as in couple [correct] or too-pull, as in Tupelo, Mississippi). All attributes should be dependent on this key, so your relation cannot be just a random assortment of attributes. Technically, all relational databases are in 1NF, although they can fail to be in 1NF logically. For example, I’ve seen a “database” that was one table containing an exact image of the records that had existed in the predecessor flat file with all its redundancies and inconsistencies intact. There also must be no repeating groups (i.e. no attribute may have multiple values). For example, if an employee has multiple assignments, this relation is technically, but not logically 1NF:

EMPLOYEE (SSN, Assignment1, Assignment2, Assignment3)

Although Assignment1, Assignment2 and Assignment3 don’t repeat in exact name, they repeat the same attribute logically and therefore will cause problems, and violate 1NF in spirit (so they are not “spiritually” 1NF?). By definition, the real-world Assignment is not dependent on SSN because one SSN can yield several different Assignments.

The next level, 2NF, is only a concern when there is a composite key. A relation violates this form when an attribute is dependent on a subset of the key, for example in a two-part key the attribute might be dependent on only one of the components of the key. Again, it’s possible to be technically in 2NF although not logically. Some DBA’s assign an arbitrary key to every relation, which is either randomly or sequentially generated. Since this is a single attribute key, all relations are technically in compliance with 2NF. In this case, if there is an obvious candidate key that violates the 2NF rules the relation has qualified for 2NF on a technicality and so probably shouldn’t be called 2NF. In other words, you can’t get into 2NF by simply using an arbitrary device to eliminate the compound key.

So finally we get to the defining requirement needed for 3NF: “no transitive dependencies”. Transitive dependencies are dependencies among the attributes. So no attribute may be dependent on any attribute of the relation except the key and any candidate keys. It’s called transitive because the attribute’s dependency on the key is transferred through its dependency on the nonkey attribute. So, the oath of the Third Normal designer:

Normalization is important for entities and relational databases, and is at least twice as important for objects, since not only attributes, but also operations, need to be normalized. So how do thoroughly modern designers survive without it? Muller asserts that “it is very easy to put your database into fifth normal form with a simple, direct design.” Although I don’t agree fully with Mr. Muller’s ranting (perhaps he was frightened by a DBA as a young child), Zen analysts are usually “Naturally Normal”: if you carefully identify the entities (objects) in your system, and put the attributes with the entity they logically relate to, you’ll be nicely normal even without going through the “ritual” of normalization. So you can in fact achieve the same result using other techniques instead such as:

Patterns	as per the Gang of Four: Erich Gamma, Richard Helm, Ralph Johnson & John Vlissides, Design Patterns: Elements of Reusable Object-Oriented Software.
Heuristics	as per Arthur J. Riel Object-Oriented Design Heuristics.
Refactoring	as per Martin Fowler, Refactoring: Improving the Design of Existing Code.
Zen Analysis	as per Patrick McDermott, Zen and the Art of Systems Analysis: Meditations on Computer Systems Development.

Naturally, the last method is the best.

Chapter 8. Meditations on a Model

Copyright ©2003, 2008 Patrick McDermott