sci.engr.* FAQ on Engineers and Quality

sci.engr.* FAQ on Engineers and Quality
Contents

What is Quality?
Return to Contents

Nobody knows. :-)

Quality is that aspect of your product or service which the customer most wants. For this reason, your customers know more about quality than you do. You cannot define it apart from your customers. Guaspari puts it this way:

They're not buying a product. They're buying your assurances that their expectations for that product will be met. And you haven't really got anything else to sell them but those assurances. You haven't really got anything else to sell but quality.

While many authors have contributed to the field of quality in this era, the one perhaps most often pointed to as being responsible for the mania we know today as Total Quality Management (even though he swore while he was alive that he had nothing to do with it) is W. Edwards Deming. Deming, who built his business and reputation helping Japanese industry recover from World War II, developed what he referred to as a "system of profound knowledge" that would enable managers to lead their organizations to quality. The system of profound knowledge consists largely of

Some grounding in statistics fundamentals, resulting in an understanding of the nature and sources of variation.
Some knowledge of systems engineering, resulting in an appreciation for interactions between subsystems.
Some idea where knowledge comes from and how it is passed on, resulting in an ability to teach and lead by example.
Some knowledge of psychology, resulting in the ability and desire to help people change current practices and move into a new way of thinking.

Deming's 14 Points for Management (per Out of the Crisis) are as follows:

Create constancy of purpose toward improvement of product and service, with the aim to become competitive, to stay in business, and to provide jobs.
Adopt the new philosophy. Western management must... take on leadership for change.
Cease dependence on inspection to achieve quality. Eliminate the need...by building quality into the product in the first place.
End the practice of awarding business on the basis of a price tag. Instead, minimize total cost. Move toward a single supplier for any one item, on a long-term relationship of loyalty and trust.
Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
Institute training on the job.
Institute leadership -- the aim of supervision should be to help people and machines, and gadgets) to do a better job.
Drive out fear.
Break down barriers between departments. People in research, design, sales and production must work as a team to foresee problems in production and use.
Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity.
Eliminate quotas, management by objective, and numerical goals.
Remove barriers that rob workers of their right to pride of workmanship. Abolish merit ratings (and performance appraisals).
Institute a vigorous program of education and self-improvement.
Put everyone in the company to work to accomplish the transformation.

Here are a couple Web sites with more information on Deming's teachings:

Adams (in The Dilbert Principle) adds that individuals within the company should avoid as much as possible involvement in anything not directly related to (a) employee effectiveness or (b) product quality. This often includes serving on quality teams. :-)

WHAT SHOULD WE ALL KNOW ABOUT QUALITY?

There are three scientific aspects of quality which I recommend we learn -- not in detail, but conceptually only will do fine. They go to what data gets collected, what analysis gets done, what conclusions can be drawn from the analysis, and to what degree of certainty. I have them listed in order of what I see as their importance to the company, and coincidentally that's also the order of how well I know them myself.

Statistical Process Control
Design of Experiments
Theory of Constraints

How does one achieve Continuous Improvement?
Return to Contents

Deming says "improve constantly and forever your product and system of service." This is most effectively done by concentrating on "bottlenecks" in the process you are trying to improve. Bottlenecks in a process are analogous to overloaded members in a structure, or overloaded circuits in an electrical system: the process is only as effective as its slowest bottleneck -- as a chain is only as strong as its weakest link.

How do you know whether you have a bottleneck? Measurements (known in the literature as metrics) are necessary -- and not simply cost. There is no organization that has the goal of minimizing its costs. If this were the goal of any organization, then that organization could achieve maximum performance immediately, by going out of business. The use of cost as a metric focuses management attention, and bases local decisions, on something that is not the goal of the organization.

One right set of measurements is defined by E. M. Goldratt, the originator of the Theory of Constraints:

THROUGHPUT	sale price minus raw material cost
INVESTMENT	all money trapped in the organization
OPERATING EXPENSE	money spent to generate throughput

Throughput is (again) not necessarily money. It could be "health units" (for a hospital) or "education units" (for a school) or "satisfied customer units" (for a business). But this is the metric that you base continuous improvement on, while trying to hold investment and operating expense nominally constant (though there may be capital expenditures necessary for "optimal" process improvement).

The Theory of Constraints (TOC) suggests that for any process, the bottleneck can be modeled as a constraint on throughput. Relaxing the constraint is not the same thing as firing inputs at the bottleneck at the fastest rate -- it is rather to maximize the throughput per unit time at the bottleneck. The following process from the TOC summarizes the necessary steps toward continuous improvement:

Identify the constraint. This can be a policy as well as a physical bottleneck.
If the constraint is a physical bottleneck, then exploit it fully -- maximize its throughput. This may involve determining an "optimum" mix of products or scheduling of the prouction line. If the constraint is a policy, find a way around the policy.
If the constraint is a physical bottleneck, then subordinate all operations to your decision to exploit the constraint. Get everyone involved in the process to cooperate with the effort.
If the constraint is a physical bottleneck, and if you've already performed the first three steps, then increase the capacity of the constraint.
If during step 4 you've increased the constraint's capacity to the point that it is no longer the constraint, then go back to step 1 and begin the process all over again.

In fixing bottlenecks, it's important to remember that overdesign of the fix may waste time and money. You want to correct the bottleneck enough that it doesn't have to be corrected again in the near future. Having done so, you find the next bottleneck reveals itself -- if it hasn't already. (There is no need to identify them all at once, since you will be correcting only one.) But the tendency toward optimizing every decision is itself a bottleneck, and is a chief cause of "TQM burnout" in organizations.

To look at the whole system as indicated above assumes you're the "puppet master" -- that you know all the steps in the process and have authority to change them. This may not actually be the case. Which is why Deming says "adopt the new philosophy." Leadership must come from the top -- from the one who IS the puppet master. Continuous improvement can thus be implemented by individuals who are puppet masters; or by small teams when they are empowered to do so and are effective; or by large teams when those teams have systems engineering capability, and are monitoring the overall process. Small and large teams can act as puppet masters over certain processes.

The implementation of that complete solution will require the cooperation of many who are not on the team, and of many higher level managers who aren't even in the team's organization. A sales job will then be part of the complete solution. This is why you will hear the term "buy-in" frequently spoken of in quality initiatives. Of course, the buy-in on the continuous improvement effort is much easier (or, "possible") when buy-in has been achieved for the quality initiative in the first place.

There are several books by Goldratt on the subject, and some excellent posts by Rizzo (which he has copyrighted, so I assume he is answering as to their availability), and Goldratt's TOC operations have a web site.

What is Design of Experiments?
Return to Contents

Design of Experiments, or DOE, is the procedure that invokes statistical knowledge to help you to

Maximize the information gleaned from a given number of experiments.
Minimize the number of experiments required to validate (or invalidate) an hypothesis.

What you need to take advantage of DOE principles is basically a good problem definition. A good one will include

an agreed-upon objective
one or more hypotheses to examine
a selection of measurements/response variables/"Ys"
a selection of factors/predictor variables/"Xs"
a desired degree of certainty for everything that's uncertain when you start

Each hypothesis you will examine fall into one of two categories:

significant differences in Ys due to selection of some X
significant relationships between some Y and some Xs

Determining significant relationships generally involves performing regression analysis (you might call this "least-squares") on the data (with the number of coefficients in the relationship depending on the number of levels an X may take on); determining significant differences involves performing an analysis of variance of some sort on the data.

The concerns in DOE are as follows:

Population size. This goes to the power of the test. Too small a population can cause you to imagine a significant difference between sets of data where none exists, or to imagine no difference exists where there really is one.

Components of variation. What should be allowed to vary in your experiment (i.e. occurs naturally)? And what shouldn't (i.e. you have to fix it or account for it)? If you have control over some component of variation, you may be called upon to use that control; if you don't, you may be called upon to block out some component of variation.

Randomization. To remove time-varying effects from the results.

Blocking. To remove known or suspected biases.

"Block what you can; randomize what you cannot." -- Sir George E. P. Box

The actual design in DOE involves using the statistical and intuitive tools at your disposal to define the four concerns above. For instance, a desired error rate may be used to determine a number of blocks; a desired magnitude of discernable difference between data sets may be used to determine population size.

If you have two sets of data, here's how to compare them to one another:

Check sets for normality (if they aren't all normally distributed, then you can't do the rest of this stuff until you transform the data to make them so; once they are, find the mean and standard deviation of each set).
Perform an F-test on the standard deviations to see if they are significantly different from one another.

If they are, check for heteroscedasticity: are the standard deviations a function of the means? are there any outliers? You may need to recheck the way the experiment was performed if heteroscedasticity arises -- if it's not done right, or if data is recorded incorrectly, well, there's your outlier. :-)
If they aren't, you get to "pool" the data.

Perform an appropriate t-test on the means (meaning that the right test depends on whether the standard deviations are significantly different) to determine significant differences.

When you have more than two data sets, the procedure is roughly the same, except that the comparison procedure for standard deviations (Foster-Burr test) and means (Student Newman Keuls if not pooled, 1-way ANOVA if pooled) must take into account the multiplicity of data sets.

An "optimal" sample size for the data sets depends on

how big an effect (i.e. difference in means) you want to detect
how much you want to risk missing it (type I error rate)
how much you want to risk overestimating it (type II error rate)
how much variation you expect in your final results (pooled standard deviation).

So how many experiments to you run? If you know an "optimal" sample size, and can run the experiment over and over for each combination of settings for each X, then you have what is called a factorial experiment. If instead this turns out to involve more data taking than you can afford, then you must find a way to "fractionalize" the factorial experiment. When you do this, you lose some accuracy in your results. A way you can do this is to recognize that a factorial experiment involves orthogonal arrays. For instance: if each X has two settings, you can pretend those settings are +1 and -1. Then, if you take the "dot product" of the settings for any two Xs, the "dot product" is zero. Simple example:

X1 -- +1 +1 -1 -1
X2 -- +1 -1 -1 +1

Two parameters, two settings each, four runs of the experiment, dot product of settings is zero. A fractional factorial experiment would take only part of this information, changing the settings of more than one X at a time. For instance,

X1 -- +1 -1
X2 -- -1 +1

Taguchi is one expert who recommends highly fractional factorial experiments. His orthogonal arrays may involve a small percentage of the total repetitions involved in a full factorial experiment, and thus would be run quickly and for low cost. The tradeoff involved in Taguchi design is as follows:

You lose the effects of interactions between Xs, which must be assumed not to exist.
Taguchi assumes processes that are already under statistical control, and used in a context of continuous improvement. His design therefore assumes that, even though you can't be quite sure of the results, they're only provisional anyway.

What is Statistical Process Control?
Return to Contents

Deming refers in his works to "common causes" and "special causes."

COMMON CAUSE	data within a normal distribution -- inherent process randomness
SPECIAL CAUSE	data within a shifting distribution -- "correctable" phenomena

Special causes are generally things that you can correct for, and are always things that you can catch in the data. Examples would be part wear, environmental disturbances, loose fasteners, untrained workers, impurities in materials or media, changes in shift or suppliers, etc. etc.

You catch special causes in the data by means of control charting. With a control chart, you're able to follow what the data is doing in time. If a visual inspection of the control chart causes you to be suspicious, there's a procedure to follow that tells you whether or not you have a special cause. If you have one, the behaviour of the special cause may help you to diagnose what it is, or you can (with the help of everyone involved in the process) design an experiment to help you to isolate it. If you don't have one, then your process is said to be under statistical control.

How to Find Special Causes

Plot the data.
Designate on the plot the "target value" (generally the mean).
Compute the common-cause standard deviation (the variation is the square of the standard deviation).

Find the difference, or moving range, between each consecutive pair of data points you've plotted.
Take the average of these moving ranges.
Throw out any moving range that's more than 3.27 times the average you found in (3b).
Repeat steps (3b) and (3c) with the moving ranges that are left, until you only have moving ranges which are not greater than 3.27 times their average.
Once (3d) is done, your common-cause standard deviation (which we'll call S_cc) is the resulting average moving range divided by 1.128.

Plot "control lines" horizontally for mean plus or minus 1, 2, and 3 times S_cc. (Don't forget to label 'em.)
The mean has shifted significantly, and your process has gone out of statistical control, if any of these rules of thumb are violated:
- one point outside 3S_cc on either side of the mean
- two-out-of-three consecutive points outside 2S_cc on the same side of the mean
- four-out-of-five consecutive points outside S_cc on the same side of the mean
- eight consecutive points on the same side of the mean

On your chart, make sure to mark the following, because you may need it to diagnose special causes:

mean and standard deviation of all data
S_cc
the number of data points
the source of the data
the contact info
the date, time, and duration of charting

Here are some SPC software packages:

OMatrix shareware
MicroConsultants shareware

What is Just-in-Time?
Return to Contents

Just-in-Time (JIT) is an inventory procedure that primarily involves bringing spare parts to the assembly line (or to the loading dock for delivery) only when they are needed, and minimizing in-house spares. Reid describes in his book the advantages seen by Harley-Davidson in adopting the JIT philosophy:

Reduces setup time
- moves some steps in setup "off-line," thus reducing delays
- eliminates unnecessary movement
- eliminates some nuts and bolts at the work station
- eliminates some machine-based adjustments
- standardizes tooling, dies, fixtures, part design and specs
- uses block gauges and templates for adjustments
In short, because the spare parts aren't going to be there, the parts coming through pretty much have to be right and the machines pretty much ready to accommodate them.
Focuses the flow of the process

Encourages containerization and parts control

Encourages operator preventive maintenance

The JIT system, where it is seen as useful, can be set up in three phases: first, examine the "little things" that can be done to save time; second, examine minor modifications of tools, dies, fixtures, machines and procedures; third (after you've saved as much $$ as you can :-)) look at larger capital expenses for new machinery. Again, don't buy the machinery until you've made improvements to your process without the expense. Otherwise the expense won't help as much. Maybe not at all.

What do I need to know about Total Quality Management?
Return to Contents

What we know today as Total Quality Management may well vary from organization to organization, and may also depend on which consultant is teaching TQM to the organization. For this reason, we will ignore most of the management BS that seems to accompany quality initiatives and concentrate on the fundamentals.

Why TQM is not held in universal esteem:

TQM is often active in organizations that are downsizing and overworking the remaining staff.

TQM is identified with excessive documentation. Excessive, boring, irrelevant documentation, couched in "psycho-babble."

TQM is identified with consultants that teach concepts to an audience they do not know, in an industry they do not understand.

TQM is identified with managers who make a formal process out of the least common denominator of sound day-to-day work practices, and impose it on their entire workforce in a "one-size-fits-all" fashion.

TQM is identified as a way of telling highly skilled professionals that their work is not good enough, and that they must work harder, sometimes doing things totally unrelated to their previous tasks, to be competitive.

With all these negative pictures being conjured up in the worker's mind, it's understandable that there would be opposition to a quality initiative -- even if the initiative is based on sound understanding. So many people have these pictures in mind -- so many have been burned by quality initiatives -- that it can't be their fault they have them.

How TQM comes to be thought of this way:

Quality initiatives built on sound principles still contain ingredients that arouse suspicion. Buzzwords like "profound knowledge," "team-building," and so forth. They sound like they contain "soft" elements.

The more scientific aspects of TQM are not second nature to us, or to those who impose TQM on us, or who teach it to us. Teachers will sometimes revert to those topics that are easy to teach.

We assumed that the quality initiative was designed to replace dependence on competence.

We assumed that even the most basic, elementary operations were looked at as candidates for improvement, sometimes equally to complex processes.

Previous bullshit quality initiatives have left us jaded.

Where we are without organizational attention paid to continuous improvement:

Depending on inspection to "inspect in quality." Inspection in and of itself cannot put quality into a system -- it can only reject what fails to meet the standards.
Working with an unchanged product, because to change it we must work differently.
Missing the connection between statistical variance and corporate bottom line.
Fixing only what's broken, and only after we hear it break.
Depending on behavior that, even in masters of their craft, has an element of randomness.
Specifying tolerances outside of product and process knowledge and customer understanding.
Determining process improvements via full factorial trial & error.
Seeing new initiatives as extraneous to our real work.
Always hoping our work environment will be like a Skunk Works, except that we will be individually respected as well. (Since Kelly Johnson respected almost no-one. :-))

What could we really do instead?

Introduce the scientific aspects of quality (SPC, DOE, TOC) to everyone in the organization
- without adding new worker requirements (though some present ones may be changed)
- without "talking down" to anyone, and without reduction to the "least common denominator"
- and with the "big picture" always in the background.
Enable all workers to contribute to the improvement of the process they are obligated to work under
- process improvement must be shown as part of our work, not something that keeps us from getting our work done
- "losses" must be defined for all to understand -- both those intrinsic to the product's function and those due to variances
- training must be made available that enables workers at all levels to continue to "do things right" as the processes are made to change
Modify the role of inspectors
- give them the ability and authority to fix problems
- make them responsible for SPC charting
- reward them on the basis of working with processes to improve them, and on the impact of those improvements on the bottom line
- allow them to address the needs of new markets
Cease and desist immediately the practice of all bullshit
- concentration on trivial problems (e.g. charters)
- emphasis on slogans
- insistence on management tools for non-management personnel

Deming, in The New Economics, points out a few fundamentals for managers, based on the idea that understanding of the way a system works includes understanding of the way people work. No improvement of any system can be undertaken outside of improvement of the management of people.

Individual performance is largely a function of the system the individual works in. The system is the responsibility of management. Every manager (at least of engineers) should review Deming's "red bead experiment" before embarking on performance appraisals.

Fear will cause engineers, inspectors, and lower-ranking supervisors to "fudge" numbers in order to prevent unjust punishment. Likewise, performance measures based on sales, revenue or costs will lead to another type of number-fudging, and to dishonest customer relations. Psychology and variation are thus locked together.

Once the numbers have been fudged, predictions and decisions based on them are fundamentally flawed. If management therefore wants the truth, it must drive out fear.

Don't tighten tolerances first. If the process is already out of statistical control, tightening tolerances will make the process more expensive without making it better. Likewise, if you crack down on people, you make them unhappy without making them better.

Here is a brief description of the Red Bead Experiment. Choose several "operators" at random. Give each a sifting box. Each is to take a sample of beads from a larger box, one which contains mostly white, but also red beads. The samples are to be sifted until there are just enough beads to fill each slot in the sifting box. The red beads are counted. Those with the most red beads are the "best performers." Deming makes the point (absurd, but truth is stranger than fiction) that real-life processes can tend to be only slightly less random than the drawing of beads from a box.

Here is another experiment that illustrates a similar idea. Hold a funnel over a target on a table. Then cause a ball to fall through the funnel toward the target. Miss the bulls-eye? Too bad. Have to have management intervention. Move the funnel from its previous position to compensate for the error. Try again. Miss the target again? Management intervention again. Before long, you have all sorts of scatter in the points where the ball hits. Solution? For the time being, ignore the target and try to achieve uniformity. Hold the funnel in the same place over and over and plot the scatter. Once the scatter is stable, then you can make adjustments.

Overmanagement is perhaps the biggest impediment to quality. Adler points toward a modern misrepresentation of the time-and-motion-study ideas of Frederick W. Taylor:

Routine, repetitive tasks require standardized procedures.
Standardization reduces motivation and creativity.
Lack of motivation leads to dysfunctional employee behaviour.
Counterproductive behaviour leads to more authoritarian management.
Micromanagement leads to more routine, repetitive tasks.

This is what we see, and why Adams makes so much money writing "Dilbert" books. :-) The reason the picture is incorrect is because the second step above is misapplied. Taylor didn't recommend it. Business evolved into it -- by having formal work standards imposed on employees, instead of designed by employees. A switch from the one to the other can reverse a situation that's going down the plughole.

Effective Teams

A lack of understanding of effective teamwork is an impediment to quality.

There are five ingredients in most effective teams:

purpose
leadership
membership
interaction
deliverables

It appears that each is necessary, but none are sufficient without all the others.

Purpose. You would suppose that a team is formed precisely because of a purpose. (You would suppose that, but I at least have seen other things happen. :-)) The effective team's purpose has the following qualities:

Based on organizational need
Founded according to sound, reasonable principles
Tied into the self-interest of the team members, thus making it easier to sell to them
Faced with a competitive imperative

Leadership. The team leader is not to be confused with line management, even when the line manager is the team leader. (In that case, it's necessary for the leader to wear a different hat.) The effective team's leader has the following attributes:

Willingness to work to achieve (and maintain) "buy-in," to build confidence
Willingness to make decisions based on team findings, rather than solely on personal judgment
Willingness to let experts be experts, and to manage work rather than do it
Ability to grant or obtain authority for members to carry out decisions
Ability to grant incentives for team members to make one another "look good"
Understanding of the skills and behavior of the team members

Membership. If the team members have to be drafted, then the purpose of the team should probably be reexamined. So for this aspect of the effective team, we assume "buy-in" has been achieved -- that the members start with commitment to the purpose. (Some teams are partly founded on religious fervor. It was unclear to me from this discussion whether a religious conversion to the team's purpose is necessary.) To achieve the rest of what the membership needs, it is quite possible in some cases that training will be necessary:

Mutual respect
Reliance on one another to be willing and able to do their jobs
Common, confident view of group objectives
Freedom to raise questions and criticism

Interaction. The term "trust" was raised again and again in this discussion, and it became clear that in the context, "trust" didn't mean "with our family secrets." It meant recognition of professional responsibility and reliance on having said responsibility fulfilled. Having established trust in this sense, interaction is based on this trust in the day-to-day functioning of the team:

Protection of one another's self-esteem
Soft-pedaling our natural tendencies to cover our asses
Setting aside of all preconceived notions of how things (other than fundamental engineering principles) are "to be done," of natural resistance to change
Responsibility for our own actions and group-assigned tasks

Deliverables. The obvious deliverable is the fulfillment of the group's purpose, if it can be fulfilled; and some detail as to why not otherwise. (Some in this discussion felt that a truly effective team could not fail to fulfill a truly worthy purpose.) Not so obvious is the deliverable of "lessons learned." There are two kinds of those: the group kind, that should be documented in plain "English" (or, written clearly in whatever your language happens to be); and the individual kind, to be filed away in our memories to be drawn on (or recorded on resumes) later.

Rewards. Individual performance evaluations and rewards are simply inconsistent with team performance, particularly if the individual is on the team full-time, and may even undermine the team. (Deming was one author who showed why individual performance evaluations tend to be tied into random chance anyway.) Most participants in this discussion felt that rewards should be tied into team performance, and shared nearly equally among members (some suggested peer evaluations could be used to determine MVPs if appropriate), and might well be used as a "carrot" to help achieve "buy-in." Confidence is, after all, tied into the employee's self-interest, and there's more than one way self-interest is expressed. :-)

Personality Types

Recognizing what personality types make up your team can go a long way toward helping you predict what contributions each member can make.

Sociability: Extroverts	direct energetic action-oriented
Sensitivity: Intuitive	cautious sensitive detail-oriented
Perceptivity: Perceiving	influential persuasive optimistic
Intellect: Feeling	steady agreeable dependable

This list is VERY incomplete. A Myers-Briggs test would place you in one of 16 different categories, depending on which of two ways you fall in the Intellect, Perceptivity, Sensitivity, and Sociability classes. There is a Web site below that goes into a bit more detail, and a host of information out there on Myers-Briggs testing. The bottom line is that companies will take test results from Myers-Briggs for their employees, and try to take those results into account when forming working teams. (Of course, you need a critical mass of players to make that process work.)

Group Responsibilities

Individuals master project knowledge for themselves; then help others on the team master that knowledge too.
Ensure no undue stress placed on any member.
Maintain a shared vision.
Maintain cooperative involvement in team meetings.
Allow for free information flow. The team leader should work with the others to ensure that the team has an information model.
Prioritize the quality of ideas over the talent, resources, and influence of any individual member.
Balance risk and reward, and try to achieve consensus on what the balance is.

Leader's Responsibilities

Choose members carefully, offering them the group responsibilities given above.
Consider temperament and personality type as a qualification.
Make sure team goals are clearly defined; get "buy-in" at the start.
Allow free expression of ideas when ideas are actually needed; make sure everybody is comfortable in the group even when ideas aren't needed.

What is Quality Assurance?
Return to Contents

"A quality assurance system is needed to help ensure that the proper materials of construction are used, that fabrication and inspection procedures are proper, and that installation procedures recognize field installation concerns. The quality assurance program is an essential part of the mechanical integrity program and will help to maintain the primary and secondary lines of defense that have been designed into the process to prevent unwanted chemical releases or those which control or mitigate a release. 'As built' drawings, together with certifications of coded vessels and other equipment, and materials of construction need to be verified and retained in the quality assurance documentation." -- from OSHA 1910.119, "Process Safety of Highly Hazardous Chemicals"

This may be hard to believe, but much of quality assurance (or, QA) involves reading and writing. That's because improving quality can involve analysis of behavior -- something that's neither numerically quantifiable nor consistent. Although control charts are critical to SPC, they're not the whole story in a quality assurance program. (NOTE: some programs don't take quantifiable data into account at all. Beware!)

A QA system will ask the following questions:

How do we do things now?
How do we SAY we do things now? What requirements do we have in place?
Where do the first two answers deviate?
How do we fix the deviations?

...and perform the following actions:

Fix the deviations.
Document the way things are done according to standards.
Get "buy-in" from everyone involved.
Adopt a policy of continuous improvement which includes auditing these processes.

ISO 900x is an international standard that helps companies align the way they do things with the way they say they do things. A QA system that conforms to ISO 900x has procedures for reporting and correcting problems. This doesn't guarantee product quality, but it provides a framework for conscientious companies to pursue continuous improvement.

Event Reporting

As part of a QA system, you'll analyze a report of an event. Maybe a control chart reveals a process has fallen out of statistical control. Maybe there's been a failure in a part or machine. Maybe a new module has been written for a legacy software program. These are events, and witnesses or participants will be called on to contribute to the report. If you have to prepare the report, you may need to interview those involved. Even if you don't have to prepare the report, you certainly must read it to become familiar with the event.

When you've become familiar with the report, then you can analyze the event and all behavior leading up to it. You have to be able to visualize what each person involved (or, "actor") did. This means you may have to

mark up the report
clarify the identities of multiple actors
match up each action with a single actor
infer actions that aren't stated explicitly, such as

when the narrative is written in passive voice
when there's been a change in process state
when some action is assumed in the narration (e.g. estimated data -- but whose estimate?)

place the actions in context
organize the actions in such a way as to avoid creating conclusions (e.g. with "did" format instead of "did not")

With all the actions in context, you can make value judgments on each. The judgments can resolve

whether more data is needed by the actor
whether measurements or estimates taken at that moment could be considered reliable
whether the narration was at that moment reliable

Anything that's left after those judgments are made is probably a deviation in the process -- one that you alone can't fix. But that's what you're there to identify!

Software QA

Software QA is analogous to the process above in that you want to know what each method does, when it does it, and on what variables it does it. A computer program must supply an answer for all possible instances within its scope, and those answers must be correct and reasonable, before its results can be released to a customer. Then you "lock down" the program, give it a version number, and provide documentation as needed.

The QA role is to keep a program from being updated and used without everyone concerned knowing what the update IS and therefore not using it the same way. It's a function that sometimes must be performed pretty much concurrently with the actual collection of results for the customer, unless you're selling the software as a consumer product. Customer requirements dictate how the program is to be tested.

As is the case with any consumer product, there is a difference in "generation quality." That difference can be significant. Software will (hopefully) improve with successive generations, as will most consumer products. But a "bootleg" product (for instance) will deteriorate with successive generations.

Sample Quality Assurance Review

I have completed a review of [company name's] QA document, and as a result feel comfortable with collecting information that we need to prepare a similar document of our own. I understand the need for such a document: we want our customers to feel that we can build repeatedly to their specs. (Or, if they don't give us specs, to specs that we generate from their requirements.)

Here I will summarize what I think is directly adaptable to our needs from the [company name] document, and what I think will make our document even better. It may not be necessary to have a better document, but I think we have to take a different approach than theirs for the following reasons:

They don't really define "quality," and I think this is because the term changes meaning depending on the product and the customer. I believe we can and should define the term, and with enough meat that our customers know we have direction.

There are certain scientific aspects of quality for which everyone on the team should have a basic knowledge. This is because in so knowing we'd all be on the same page, and everyone on the team bears some responsibility for the quality of the product. This is necessarily different from what [company name] is doing: [company name] is a big company and has a dedicated QA department, along with other functional departments that we just can't have for a couple growing years or so. [company name] does train "all personnel performing activities affecting quality," and tests them, but my experience tells me that much of what these folks are learning is either (a) disconnected from the job they actually do, or (b) too simple to change their approach to their work, or both. We can't afford the same approach.

Because (I assume) [company name] has a more diversified product line, or at least more customers, than we do, they give few specifics about how their QA system is applied to a job. (Where they do give specific instructions, those instructions are generally excellent -- but they're buried in the document more often than not.)

[company name]'s entire QA program is centered around inspection. (In fairness, this may be because inspection is the only way to judge compliance on certain items. But for whatever reason, the document reflects this focus.) We can't do this, because you cannot inspect quality into a product. All inspection will do is reject noncomforming parts -- it doesn't improve the processes or those who carry them out. [company name] also gives the inspection responsibilities to separate individuals (which IMPO is good) who have no direct ownership of the process (which IMPO is dubious) and who have no direct powers to change it (which IMPO is bad).

Individual roles are unclear. You will have individuals whose only role in the process is to receive a document, date-stamp it, and route it to someone else. You will have other individuals who have heavy responsibility but where they come from is unidentified. And the document is about responsibility: who takes the fall if a product fails. We need to be about responsibility in a different way: who really *needs* to support each aspect of the process. Since quality is the responsibility of each of us, we must all take the fall in event of a failure.

VERY INTERESTING MATERIAL

Here's what [company name] considers important considerations in inspection of a given component:

Consequences of failure.
Uniqueness & complexity of design or fabrication.
Special control or observation of processes or equipment.
Ability to demonstrate compliance via inspection or test.
Quality history and degree of standardization.
Difficulty in repair or replacement.

Throughout this document, [company name] refers to reviews. Preliminary and Critical Design Reviews, from which formal changes in design can be formally requested. This is a common way of doing business, which if adopted shows we can speak the same language.

Throughout this document, [company name] refers to identification and traceability. We've not been particularly good at keeping records up to this point, primarily because we have so much crap that there's no time. But it's a good idea to adopt specs on these things, like [company name] does:

Which records really need to be maintained.
Part numbers and characteristics.
Restrictions on where and how documents and parts can be marked.
Traceability from parts to associated documentation.
Review and approval cycles.
Storage, handling, and shipping.
How to handle noncompliance.

They actually have a "Quality Plan," though they bury it under "Inspection Planning." Which is another way of saying they are trying to inspect in quality, and again I say unto you: thou shalt not. But the plan is interesting, because all this stuff is identified:

Relevant drawings, part lists, and specs.
Technical drawings and documents requiring review and approval, and by whom.
Inspections needed, and by whom, and by what methods, and at what points in each process.
Specific identification and traceability requirements.
Detailed requirements for regular process control and observation, equipment condition, and personnel qualifications and training.
Required sampling procedures (if sampling is needed).
Documentation as evidence of quality compliance.

Their argument is that all this stuff must be in place before a part is shipped, and we are going to have to head in this direction as well.

DEFINITION OF QUALITY

Here is what I think will pass as a start for a definition of quality for our purposes:

The extent to which a product meets requirements agreed upon by ourselves and our customers.

Nothing this simple is stated in the [company name] document, yet if you ask around in industry you will find that a common problem in quality initiatives is having such terms undefined.

Once we define this term (whether by the above definition or some other), we have some idea where to go next. The above definition suggests that we reach an agreement with a customer over requirements. Then we work on component specs to fit into the requirements. These things are pretty much what we are doing now. But the component specs then suggest what we must do to verify compliance with specs. As you know, there are five ways to verify compliance:

analysis
demonstration
inspection
similarity
test

We have to date relied on demonstration and test. This is not surprising because we are trying to sell a system, not its components, and because our systems have tended to be one-of-a-kind. We are now moving away from one-of-a-kind systems, and that opens up to us the other three means of verification. Even so, we are not headed toward a situation in which inspection should be our most common means of verification. We are, however, headed toward a situation where we must start inspecting, which means we have to agree in advance on the how, why, and who.

We also have to agree on what constitutes compliance, and what constitutes satisfactory test or inspection results. That's something we would probably do as a management team early in the job, but we wouldn't document the meeting.

What is ISO 9000?
Return to Contents

There are a fair number of sites out there that talk about ISO 9000. Here's the summary: ISO 9000 and subsequent numbers are certifications for a documentation system. The certification says that you have a quality system in place in your company, one which addresses the following:

people
work
activities
tasks
records
documents
forms
resources
rules
regulations
reports
materials
supplies
tools
equipment

...in short, what makes up processes. The certification process verifies that you HAVE a quality system, and that you have it DOCUMENTED. That you actually do what the documents SAY you do. BUT... it does nothing whatsoever to ensure continuous improvement of quality in products or services. That's still up to you.

Nevertheless, many potential customers, including the US Government, place a high value on the certification. It's not to be overlooked.

Here are the guys who do a better job of writing this than I ever could: the ISO itself.

References
Return to Contents

Goldratt, E. M. The Goal. ISBN 0-884-27061-0
Goldratt, E. M. It's Not Luck. ISBN 0-884-27115-3
Goldratt, E. M. The Haystack Syndrome. ISBN 0-884-27089-0
Goldratt, E. M. Theory of Constraints. ISBN 0-884-27085-8
The Goldratt Institute
The PD Institute
Brainstorming
"The Memory Jogger." Methuen, MA: Goal/QPC, 1988.
Brainstorming 101 shareware
www.brainstorming.co.uk
Brown, Hitchcock and Willard. Why TQM Fails. Burr Ridge, IL: Irwin Professional Publishing, 1994. ISBN 0-786-30140-6
Starline Software Ltd. "CATALYST EventAnalyzer."
Supply Chain Solutions. "SCS Case Studies."
whatis.com. Definitions -- quality assurance.
OSHA 1910.119, "Process Safety of Highly Hazardous Chemicals."
Johnson, R., D. Johnson, and K. Smith. "Cooperative Learning: An Active Learning Strategy for the College Classroom." Baylor Educator, Winter 1990.
Robinson, S. "Build highly effective teams with personality testing." http://articles.techrepublic.com.com/5100-22-1049633.html, 10.30.2002