Steps to Validation
Conquest - Spring 2008
By Sandi Stromberg
Editor's Note: Active in pain research for many years and instrumental in developing the Brief Pain Inventory now used in most clinical trials, Charles Cleeland, Ph.D., chair of M. D. Anderson’s Department of Symptom Research, knew the importance of measuring and attending to patients’ symptom distress long before it became a concern for federal agencies.
“Now, the U.S. Food and Drug Administration is asking for more rigor in the assessment of symptoms and other patient-reported outcomes,” he says. “It wants more systematic and validated measures of symptoms that are both relevant and intelligible to patients.”
M. D. Anderson has been a leader in this field through the institution’s symptom inventories, collaborative studies in which health care professionals across the institution collect solid, scientific evidence that can be used to design interventions to relieve the symptom burden caused by cancer and its treatments.
This is the third article in a series.
Have you ever wondered who develops standardized tests, like the Scholastic Aptitude Test for getting into college or the Graduate Record Exam for graduate school?
How do they know what questions measure knowledge or intelligence?
And how can they calculate what constitutes a passing score that says a person has the ability to follow a course of study?
Those who practice this specialized field are psychometricians, and their science is psychometrics, the study of the design and analysis of tests and questionnaires. Besides standardized tests, psychometricians also play a prominent role in the construction of patient-reported assessment tools for cancer-related clinical trials, such as symptom burden outcome.
Tito Mendoza, Ph.D., assistant professor in M. D. Anderson’s Department of Symptom Research, is one such psychometrician. Along with Charles Cleeland, Ph.D., chair of the department, he’s worked with researchers and health care professionals across the institution to assess the reliability and validity of tools that measure the symptom burden of cancer patients.
Recently published are the results of an investigation into the overall symptoms of head and neck cancers. The M. D. Anderson Symptom Inventory-Head and Neck (MDASI-HN) is a study he worked on with principal investigator David I. Rosenthal, M.D., professor in the Department of Radiation Oncology and director of Head and Neck Translational Research.
Physician with a mission
When he arrived at M. D. Anderson six years ago, Rosenthal brought with him a deep concern about the symptom burden his patients experienced as a result of treatment.
With a deep concern about the side effects his patients experience as a result of treatment, David I. Rosenthal, M.D., and his team developed a symptom inventory for head and neck cancers to better assess symptom burden.
While successful advances have been made in the treatment of head and neck cancers over the last 10-15 years, the acute toxicity can be significant and also cause long-term functional impairment whose symptoms are not well understood. This is especially true for patients with cancers, such as those of the oropharynx (the soft palate at the back of the mouth, the base of the tongue and tonsils), many of whom underwent operations in the past. Today, standard treatment for them consists of combining chemotherapy as a sensitizer to intensify the effect of radiation on the tumor and to kill more cancer cells.
While cancer control and survival rates are excellent, and the organs are numerically preserved, certain functions, such as swallowing, may be impaired.
“I was looking for some instrument to assess the side effects they deal with,” Rosenthal says. “But most instruments were concerned with ‘quality of life.’ While this is important, the tools to measure it often miss many of the most relevant symptoms. Of the 30 to 40 questions, I would find that only one or two were relevant.” With colleagues at the University of Pennsylvania, he developed a rudimentary head and neck symptom burden tool, which he shared with Cleeland.
“He was very encouraging and said this was a good start,” Rosenthal says. “But it still needed work.”
That’s when Mendoza entered the picture.
Simple Measures, Valid Results
Tito Mendoza, Ph.D., assistant professor in M. D. Anderson’s Department of Symptom Research, used the following methods to validate specific symptoms experienced by patients with head and neck cancers.
Construct Validity: This method helps determine underlying factors — latent constructs not directly measurable but that can be observed with indicators. For example, a family’s socioeconomic status (SES) can’t be directly measured, but we can measure a host of variables such as the parents’ occupation, education level and income that may represent SES. In the MDASI-HN, two factors were identified: (1) mouth sores, tasting, constipation, teeth and gum problems, and skin pain; and (2) voice and speech difficulties, choking, coughing, chewing, swallowing and mucus. Each set of items indicated high levels of reliability.
Known Group Validity: Using the Eastern Cooperative Oncology Group (ECOG) performance status, one measure of how ill the patient is, as the grouping variable, patients with poor ECOG performance status reported severe symptoms on the MDASI-HN. On the other hand, patients with good ECOG performance status (healthier patients) had fewer and less severe symptoms.
Concurrent Validity: This measures how well the MDASI-HN correlates with another test that measures something similar, such as the SF12 from the Rand Corporation. The overlap in results between the two tools provided another element of validity. If patients have good health status, they should have less symptom burden, and vice versa.
Where to start?
“Before a statistical analysis plan can be written and data collected, I need to know what the researcher wants to show — what is the primary question,” Mendoza says. “I help figure out what information needs to be collected, how many patients to recruit and how many time points to include.”
Rosenthal wanted a general screening instrument for all patients with head and neck cancers that would be valid before, during and after treatment, independent of whether they had surgery, radiation, chemotherapy or any combination.
To develop the specific content for the patient questionnaire, Rosenthal worked with Cleeland and other head and neck specialists, including medical, dental and surgical oncologists, and speech-swallowing-language pathologists, as well as patient and family focus groups. They identified symptoms and functionality issues, which they passed on to Mendoza.
In the protocol’s statistical analysis plan, Mendoza also designed how they would test the reliability and validity of the data they collected. He calculated they would need to enroll 205 patients to achieve this result. Then, he stepped back while Rosenthal collected data.
From his years of experience, Rosenthal knew one of the most common symptoms for patients with head and neck cancers is dry mouth, which can have both functional and comfort implications.
“Another is mucositis, sometimes called ‘the sun-burn effect’ of radiation given with or without chemotherapy, on the mouth, throat and esophagus,” he says. “Other important symptoms are loss of taste and the inability to eat or swallow.”
Ensuring reliability and validity
The first thing Mendoza tested was reliability.
“Reliability means that if you ask patients about a symptom at one point, they give you the same answer at the second point, if nothing in the patient’s condition changes. For example, if you take your temperature with a thermometer, unless you develop a fever, you should get a similar result a few days later. If you don’t, your thermometer isn’t reliable. A variation, or noise, is coming from somewhere else. It’s like trying to hear a conversation above some ambient noise that we need to get rid of to understand the conversation.”
Next, Mendoza measured validity. To analyze data collected by Rosenthal and his colleagues for the MDASI-HN, he chose three validity measures: construct validity, known groups and concurrent validity. (See sidebar)
Of reliability and validity, Mendoza says they used three forms of validity because it’s more important. “You can have a reliable tool, but if it isn’t relevant or valid, it’s not very helpful.”
What they learned
“One thing we noticed in our trials is that some patients get a lot of mucus in their throat,” Rosenthal says. “It’s a significant symptom that hadn’t been reported before. Yet, mucus in the throat can be so copious that some patients’ entire lives are focused on management of secretions. They’re constantly gargling, suctioning. They can’t sleep. They gag and regurgitate.”
Counting this discovery, the study identified nine head and neck cancer-specific symptoms, separate from the 13 core MDASI symptoms that may be experienced by any cancer patient (see Conquest, summer 2009). They’re mouth and throat sores, problems with tasting food, constipation, problems with teeth or gums, skin pain, burning or rash, difficulty with voice or speech, mucus in the mouth or throat, choking or coughing, and difficulty chewing or swallowing.
Not only were they able to validate the MDASI-HN, but also in a subsequent study where they compared it with the quality-of-life Functional Assessment of Cancer Therapy-Head and Neck, it provided a better measure of the severity of radiation-induced mucositis.
Since validation, the tool has been incorporated into some clinical trials. However, hopes are that when technology allows, the MDASI-HN will become an integral part of a patient’s electronic medical record as M. D. Anderson researchers work toward understanding symptom burden and finding interventions to treat and ultimately prevent debilitating side effects for cancer survivors.
In the summer issue, the series continues with a look at M. D. Anderson’s recent alliance with AstraZeneca and the new opportunities for symptom research it provides.
Conquest - Spring 2008
Download pdf version of Spring 2008