QUALITY CONTROL PLAN & QUALITY SYSTEM MANUAL
This document outlines the requirements for a Quality Control Plan for Hot Mix Asphalt production and placement.
SCOPE
The Primary Contractor of a Massachusetts Department of Transportation (MassDOT) contract shall provide and maintain a detailed Quality Control Plan, hereinafter referred to as the “QC Plan.” The QC Plan along with a MassDOT approved Laboratory and Quality System Manual(s) (QSM) shall sufficiently document the quality control (QC) processes of all Contractor parties (i.e. Prime Contractor, Subcontractors, Producers, etc.).
The QC Plan is not intended to be a generic document, but must be project specific. As such, it must address specific issues related to the scope of work and also sufficiently address all QC matters depicted in the applicable specifications. In order to be a practical reference document, the QC Plan should be abbreviated wherever possible and should use charts, tables and bullets to convey necessary information to the user in a clear and concise manner.
It is expected that the QC Plan shall make references to the related specifications and will not repeat them. It is understood that all Contractor parties will be mandated to meet the requirements of the specifications unless specified in the QC Plan and approved by the District Quality Engineer (DQE). The QC Plan shall only present specific information that clarifies the contract requirements as it pertains to Contractor activities. Nevertheless, the Contractor has the option to copy information from the specifications should they believe it will provide their personnel with necessary information. Also, it is also understood that where a specification may indicate that a requirement is “As specified per the QC Plan” that this information will be provided within the QC Plan.
Prior to the start of the work governed by the QC Plan, the Contractor shall be prepared to discuss the plan. Information to be discussed shall include the proposed QC Plan submittal date, QC organization, and sources of materials. The Contractor shall submit the QC Plan to the DQE for approval, not less than thirty (30) days prior to the start of related work items. The Contractor shall not start work on the subject items without an approved QC Plan. The approval process for the Contractor’s QC Plan may include inspection of QC testing equipment, sampling and testing demonstration by the QC technician(s) and testing of proficiency samples to assure an acceptable level of performance.
Revisions and/or modifications to the QC Plan may be required prior to approval or during construction. Such refinements/modifications are subject to approval by the DQE. If information within the Approved QC Plan changes, then the QC Plan must be revised to show these adjustments and submitted to the DQE within 48 hours of the change. Changes may consist of, but are not limited to, a revision in the specifications or other reference documents regarding material or testing requirements, a change in the source of materials, or a change in personnel or testing facilities. If the requirements of a section are not fulfilled, the DQE may also require that the Contractor include more information/detail within the specified section. QC Plans shall reference the current contract documents. If the designs used are not the approved 100 percent designs, then the QC Plan shall be modified accordingly as the design progresses.
Approval of the QC Plan will be based on the inclusion of the required information and does not imply any warranty by the Engineer that the QC Plan will result in production that complies with the specifications. It remains the responsibility of the Contractor to
accomplish such compliance. A QC Plan shall not be approved prior MassDOT’s approval of the QC laboratory and Quality System Manual.
QC Plan Contents
The QC Plan shall be structured to follow the sections outlined below. As a minimum, the QC Plan shall address the items detailed under each section and subsection. The pages shall be sequentially numbered. The Contractor is not limited to addressing the issues below and should ADD additional sections or information under the outlined sections as needed.
QSM Contents
The QSM shall be structured to follow AASHTO R 38: Standard Practice for Quality Assurance of Standard Manufactured Materials and the sections outlined below. As a minimum, the QSM shall address the items detailed under each section and subsection. The pages shall be sequentially numbered. The Contractor is not limited to addressing the issues below and should ADD additional sections or information under the outlined sections as needed.
TABLE OF CONTENTS
QC PLAN SECTION DESCRIPTION 5
Cover Page 5
Table of Contents 5
Section 0: Terms and Definitions 5
Section 1: Scope and Applicable Specifications 5
1.1 Scope 5
1.2 Contract Items 5
1.3 Applicable Specifications 6
1.4 Project Drawings 6
1.5 Locus Map 6
1.6 Additional Notes or Comments 6
Section 2: Quality Control Organization 6
2.1 QC Organizational Chart 6
2.2 QC Plan Administrator 7
2.3 HMA Plant Technicians 7
2.4 HMA Paving Inspectors 7
2.5 Plant Production Personnel 7
2.6 Field Production Personnel 7
Section 3: Quality Control Laboratories 7
3.1 Primary Laboratory 8
3.2 Secondary Laboratory 8
3.3 Independent Laboratory (if applicable) 8
3.4 PG Binder Laboratory 8
3.5 Field Testing 8
Section 4: Materials Control 8
4.1 Material Types and Source of Supply 8
4.2 Additional Notes or Comments 9
Section 5: Quality Control Sampling and Testing 9
5.1 Definition of Lot and Sublots 9
5.2 Define Random Sampling Plan 9
5.3 Sample Identification System 9
5.4 Additional Notes or Comments 9
Section 6: Production Facilities 9
6.1 Primary Production Facility 9
6.2 Secondary Production Facility 10
6.3 Additional Notes or Comments 10
Section 7: Field Operations 10
7.1 Preliminary Schedule of Field Operations 10
7.2 Control Strips (if applicable) 10
7.3 Equipment and Procedures 10
7.4 QC Activities 12
7.5 Placement Control Charts (if applicable) 12
7.6 Evaluation of QC Data and Non-Conforming Material 13
7.7 QC Records 13
Appendices 13
QSM SECTION DESCRIPTION 14
QC PLAN SECTION DESCRIPTION
Cover Page
The QC Plan shall contain a cover page containing, as a minimum, the following:
* Project name/description
* Project location (town(s))
* Massachusetts Department of Transportation (MassDOT) Contract number
* Name of Prime Contractor
* Date of the original QC Plan
* Dates of any revisions (after the original QC Plan has been accepted)
* Location for Contractor’s name and signature
* Location for MassDOT’s approval signature and date
Table of Contents
The QC Plan shall contain a table of contents listing each section and corresponding page numbers and any appendices.
Section 0: Terms and Definitions
The QC Plan must use terms and definitions that are consistent with MassDOT’s Specifications, along with the Contract’s Specifications and Special Provisions. The terms and definitions defined and used in the specifications do not need to be listed in the QC Plan. The Contractor should only define any additional terms, not outlined under the Specifications, which are to be utilized in the QC Plan. If the QC plan contains no terms or definitions, outside of those outlined in the Specifications, then this QC Plan section should contain a simple statement to that effect.
Section 1: Scope and Applicable Specifications
1.1 Scope
This section shall contain a brief description of the work controlled by the QC Plan. This should include information described in the contract Special Provisions as well as station referencing and proposed project specific work.
1.2 Contract Items
All items of work which will be addressed under the QC Plan shall be listed and at a minimum contain a table listing the following:
* MassDOT Pay Item Numbers (use standard MassDOT Standard Pay Items where available, if not, use Project Designated Item Numbers)
* Item Description
* Approximate Quantity of each item based on current design
* Units of measure for each item
* The Contractor or other outside organizations, such as Subcontractors, Producers, etc., that will be performing each of the tasks under each work item.
1.3 Applicable Specifications
* Project Special Provisions
* Supplemental Specifications
* Standard Specifications and Standard Special Provisions
1.4 Project Drawings
* Include typical drawings.
* Reference project plans, if applicable
1.5 Locus Map
* Include Locus Map from Contract, if available
1.6 Additional Notes or Comments
* Insert project specific conditions which may differ than conventional work.
* If Special Provisions or Project Drawings are not included in the contract then a reference to the applicable Design Packages, where they can be found, shall be made.
Section 2: Quality Control Organization
This section shall contain the following subsections:
2.1 QC Organizational Chart
The Organizational Chart shall identify the key personnel involved in production, placement and testing. The layout of the chart shall clearly show lines of communication and authority. The chart shall identify who is responsible for the implementation of the QC Plan. This person shall be designated as the QC Plan Administrator. Also, the applicable QC Personnel and key Production Personnel shall be identified. The chart shall clearly define which personnel are employees of the Primary Contractor and which personnel are to be provided by outside organizations, such as Subcontractors, Producers, testing consultant, etc. The Organizational Chart should show lines of communication with MassDOT.
QC Personnel
These subsections should list all QC personnel that will be involved in the work under the plan. Although an effort should be made to maintain the same personnel throughout the duration of the project, alternate or backup personnel should also be listed. For each person listed, the following shall be included:
* Name
* Phone number
* Employer
* Applicable certification course titles and certification numbers
* Authority and responsibilities
* Inclusion of resumes within the QC Plan is not necessary
2.2 QC Plan Administrator
This subsection should list the Plan Administrator and any other related positions such as Assistant Plan Administrator or Subcontractor’s Plan Administrator.
2.3 HMA Plant Technicians
This subsection should list the QC Personnel who will be performing services at the production facility or at an offsite lab for the project. If there is a MassDOT approved QSM which contains the required information listed above then this subsection need only provide a list of the technician’s names and phone number.
2.4 HMA Paving Inspectors
This subsection should list the QC Personnel that will be performing inspection and testing services in the field, at the project location.
Production Personnel
These subsections should list key Production Personnel that are responsible for plant production or onsite placement/installation. Although, an effort should be made to maintain the same production personnel throughout the duration of the project, alternate or backup personnel may also be listed. For each person listed, the following shall be included:
* Name
* Phone number
* Employer
* Authority and responsibilities
* Qualifications (if specifications stipulate explicit requirements)
* Inclusion of resumes within the QC Plan is not necessary unless required by the specifications.
2.5 Plant Production Personnel
This subsection should list any key personnel that will be responsible for the production facility (i.e. Plant Manager, Plant Superintendent).
2.6 Field Production Personnel
This subsection should list any key personnel that will be responsible for the onsite placement/installation (i.e. Superintendent, Forman).
Section 3: Quality Control Laboratories
This section should address all laboratories and/or testing companies (production facility, project site or offsite laboratories) used for Quality Control and shall list the following details for each laboratory:
* Name of company to perform testing (contractor, materials supplier, consultant, etc.)
* Location of laboratory
* Laboratory contact information
* Laboratory qualifications (AASHTO, NETTCP or other)
* Specific tests expected to be performed for the project (with AASHTO designation and name; e.g. T27: Sieve Analysis of Fine and Coarse Aggregates)
This section shall contain the following subsections.
3.1 Primary Laboratory
* Identify the primary laboratory that will be performing the work for the project.
3.2 Secondary Laboratory
* Identify the laboratory expected to be utilized in the event the primary laboratory is not available.
3.3 Independent Laboratory (if applicable)
* Identify the laboratory to be utilized for test methods in which the primary lab is not qualified to perform.
3.4 PG Binder Laboratory
* Identify the laboratory to be utilized for PG binder grading.
3.5 Field Testing
* Identify the testing company to be utilized to perform the field testing, including ride testing.
Section 4: Materials Control
This section shall contain the following subsections: 4.1 Material Types and Source of Supply
This subsection shall contain a table(s) listing the following:
* Bid Item Number
* Material type
* For HMA/WMA:
* MassDOT Mix Identification Number
* LTMF approval date
* DMF approval date, if applicable
* For other items such as emulsions, hot poured rubberized joint sealers, in-line additives, etc.:
* Manufacturer
* Product name
4.2 Additional Notes or Comments
Section 5: Quality Control Sampling and Testing
This section shall contain the following subsections:
5.1 Definition of Lot and Sublots
This subsection shall identify the lot and sublot numbering system to be used by QC. All anticipated Lots and how they will be broken down shall be identified. Any requests for variation from the specifications shall be addressed within this subsection and shall contain supporting rationale and documentation and must be approved by the Engineer.
5.2 Define Random Sampling Plan
This subsection shall clearly define how random sample locations will be chosen (e.g. based on tonnage, distance, etc.), how the random locations shall be documented and when random location information will be transmitted to MassDOT. Furthermore the QC Plan should discuss any deviations from random sampling, such as blanking zones (areas exempt from random testing) that are not already clearly defined in the specifications or drawings. Any request for variance from the specifications shall contain supporting rationale and documentation and must be approved by the Engineer.
5.3 Sample Identification System
The subsection shall clearly define and list all information that will be used to identify QC field samples. Information that could be used to identify samples could include, but is not limited to the following:
* Contract number
* Sample type
* Mix ID Number
* Lot
* Sublot
* Sample number
* Station and offset
* Sample location
5.4 Additional Notes or Comments
Section 6: Production Facilities
The purpose of this section is to identify the specific production facilities that will be utilized to produce material for the project. It shall also identify any major equipment changes or additions to the facility that are not identified in the QSM, such as an asphalt rubber binder plant or other non-typical materials which are only required per contract special provision.
6.1 Primary Production Facility
This subsection shall identify the primary production facility and contact information.
6.2 Secondary Production Facility
This subsection shall identify the secondary production facility and contact information in the event that the primary facility fails.
6.3 Additional Notes or Comments
Section 7: Field Operations
The purpose of this section is to address all of the activities that will be taking place at the project site and shall contain the following subsections:
7.1 Preliminary Schedule of Field Operations
This subsection shall address:
* The anticipated dates of work (It is recognized that the schedule is subject to change)
* General hours of work
* Anticipated outline of progress (i.e. order of locations and paving)
* Estimated daily production quantities
* How daily production will be documented (e.g. test report forms and summary placement sheets)
7.1.1 Weather Limitations
This subsection shall address the Contractor’s procedures for monitoring weather and handling with adverse weather. It shall also discuss any variances to the specifications.
7.2 Control Strips (if applicable)
This subsection shall identify at least the:
* Number of possible control strips
* Quantity of mix that will placed
* Expected length of time between placement of control strip and start of full production
7.3 Equipment and Procedures
This subsection shall provide details on the equipment and responsible personnel and shall include the following subsections. A list of the specific equipment to be used shall be included in the Appendix.
7.3.1 Control of Grade and Cross-Section
This subsection shall describe how the Contractor will provide a longitudinal and transverse reference system, with the purpose of locating and documenting sampling and testing locations.
7.3.2 Surface Preparation
This subsection shall describe the preparation activities of the surface to be paved and shall include, where applicable:
* Subbase or Reclaimed Base
* Milling of Existing HMA Pavement
o Equipment
o Milling Operations
o Protection of Inlets and Utilities
o Vertical Faces
o Opening to Traffic
o Control Strip
* Patching of Existing Pavement Courses
* Leveling Courses
* Preparation of Curbs, Edging, and Utilities
* Sweeping Underlying Surface
* Tack Coat
o Equipment
o Application rate
o Target percent of coverage
o Spray bar height and nozzle angle
7.3.3 Transportation
This subsection shall address the details of transportation of the mix to the project site including, but not limited to:
* Type and number of trucks
* Haul time and distance
* Allowable mix temperatures
o Measured at the plant
o Measured at the project site
* Asphalt release agent used in the body of the truck
7.3.4 Paving
This subsection shall include details of the placement including, but not limited to, types of equipment, construction approach and standard practices. It shall include the following subsections.
7.3.4.1 Paver
* Paving plan (i.e. number of pavers, paving width, number of passes)
* Automation (i.e. manufacturer and type)
* Joints (Transverse and Longitudinal)
7.3.4.2 Rollers
* Number of rollers
* Roller patterns
* Roller location
7.3.4.3 Material Transfer Vehicle (if applicable)
7.3.5 Post-Placement
This subsection shall address any operations that shall take place after placement procedures have been completed, such as scheduling of pavement ride testing and filling of core holes. The specific criteria related to post-construction material requirements, such as maximum pavement mat temperatures to allow traffic shall be addressed.
7.3.6 Bridge Decks
If applicable, this subsection shall describe the paving operation as it pertains to bridge decks and shall highlight any variances to what is described in subsections 7.3.1 through 7.2.6.
7.3.7 Equipment Failure and Non-conforming Material
This subsection shall provide the contingency plan(s) in the event that there is an equipment failure or if it is determined that the material does not meet the requirements of the specifications or the QC Plan.
7.4 QC Activities
This subsection shall address all of the QC activities that are to take place in the field as part of the pre-paving and paving operations and shall contain the following subsections:
7.4.1 QC Inspection Activities
The QC Plan shall discuss the inspection that the Production and/or QC personnel will perform. Inspection shall address the following components:
* Equipment
* Materials
* Environmental conditions
* Workmanship
The inspection shall include:
* Preparation of Underlying Surface
o Patching
o Tack
o Delivery
* Production and Placement of HMA Lots
o Placement
o Compaction
o Joints
7.4.2 QC Testing Activities
The QC Plan shall discuss the testing that the QC personnel will perform in the field. This should include the sequence that tasks are completed. Testing shall include:
* Prepared Underlying Surface
* HMA Lots
7.5 Placement Control Charts (if applicable)
This subsection should address:
* Which Quality Characteristics will be monitored using control charts
* Whether the control chart limits vary from the specification
* The personnel responsible for maintaining the control charts
* Where the control charts will be located
7.6 Evaluation of QC Data and Non-Conforming Material
This subsection shall describe procedures for monitoring QC data and quality levels during placement. There should be a listing of Quality Characteristics and the limits triggering action, potential corrective actions and personnel responsible for monitoring data. Reaction to single/multiple test results and Quality Levels shall be discussed.
7.7 QC Records
This subsection shall identify the location where the QC records will be stored and the person responsible for the records. It shall also list any forms (non-standard forms should be included in the appendix) or other methods of documentation.
Appendices
* Equipment List
* Non-Standard Inspection and Test forms
* Others as the Contractor deems necessary
QSM SECTION DESCRIPTION
The QSM shall meet the requirements of AASHTO R 38 Section 5 and its format shall be in conformance with Section 5.2. The following additional information shall also be required within the QSM:
* Under Manufacturing facility and products the QSM shall include:
o For Hot Mix Asphalt facilities
* Plant type (drum or batch)
* Production capacity
* Number of silos and capacity
* Number of binder tanks
* Binder blending capabilities including:
* WMA
* Latex
* Rubber
* Under QC Laboratories the QSM shall include:
o Independent laboratories, if applicable
* A list of test methods performed by the lab shall be included
* A list of testing equipment is not required
o Binder testing laboratories, if applicable
* A list of test methods performed by the lab shall be included
* A list of testing equipment is not required
COMMERCIALIZATION OFFICE - PILOT OPERATIONAL REQUIREMENTS DOCUMENT
Predictive Modeling for Counter-Improvised Explosive Devices
February 2009
Point of Contact:
Mark Protacio
Commercialization Office
SETA Support
Mark.Protacio@associates.dhs.gov
Contents
1. General Description of Operational Capability 3
1.1. Capability Gap 3
1.2. Overall Mission Area Description 3
1.3. Description of the Proposed Product or System 4
1.4. Supporting Analysis 4
1.5. Mission the Proposed System Will Accomplish 4
1.6. Operational and Support Concept 5
1.6.1. Concept of Operations 5
1.6.2. Support Concept 6
2. Threat 6
3. Existing System Shortfalls 7
4. Capabilities Required 7
4.1. Operational Performance Parameters 7
4.2. Key Performance Parameters (KPPs) 10
4.3 System Performance. 10
4.3.1 Mission Scenarios 10
4.3.2 System Performance Parameters 10
The Performance Parameters are addressed above in Section 4.1 10
4.3.3 Interoperability 10
4.3.4 Human Interface Requirements 11
4.3.5 Logistics and Readiness 11
5. System Support 11
5.1 Maintenance 11
5.2 Supply 11
5.3 Support Equipment 11
5.4 Training 11
5.5 Transportation and Facilities 12
6. Force Structure 12
7. Schedule 12
8. System Affordability 13
2
1. General Description of Operational Capability
1.1. Capability Gap
This operational requirements document (ORD) addresses the capability to predict the threat of an IED attack, identified by the Counter-IED Capstone IPT. It also covers a number of technology needs identified to further data fusion from law enforcement, intelligence partners and other sources to support the common operating picture.
1.2. Overall Mission Area Description
The Department of Homeland Security (DHS) plays a major role in fulfilling Presidential Directive/HSPD-19 (Combating Terrorist Use of Explosives in the United States) including national policies, strategies and implementation plans for the prevention and detection of, protection against and response to terrorist use of explosives in the United States.
Terrorists have repeatedly shown their willingness and ability to use explosives as weapons worldwide and there is ample intelligence to support the conclusion that they will continue to use such devices to inflict harm. The threat of explosive attacks in the United States is of great concern considering terrorists' ability to make, obtain, and use explosives, the ready availability of components used in IED construction, the relative technological ease with which an IED can be fashioned and the nature of our free society.
It is the policy of the United States Government to counter the threat of explosive attacks aggressively by coordinating Federal, state, local, territorial, and tribal government efforts and collaborating with the owners and operators of critical infrastructure and key resources to deter, prevent, detect, protect against and respond to explosive attacks, including the following:
(a) Apply techniques of psychological and behavioral sciences, such as social network theory, in the analysis of potential threats of explosive attack;
(b) Use the most effective technologies, capabilities, and explosives search procedures and applications to detect, locate and render safe explosives before they detonate or function as part of an explosive attack, including detection of explosive materials and precursor chemicals used to make improvised explosive or incendiary mixtures;
(c) Apply all appropriate resources to pre-blast or pre-functioning search and render-safe procedures, and to post-blast or post-functioning investigatory and search activities, in order to detect secondary and tertiary explosives and for the purposes of attribution;
3
(d) Employ effective capabilities, technologies and methodologies, including blast mitigation techniques, to mitigate or neutralize the physical effects of an explosive attack on human life, critical infrastructure, and key resources; and
(e) Clarify specific roles and responsibilities of agencies and heads of agencies through all phases of incident management from prevention and protection through response and recovery.
1.3. Description of the Proposed Product or System
The proposed solution shall employ the knowledge and understanding gained in the military environments such as Iraq and Afghanistan to model and take action against IED network activities in the United States. It shall enable investigators to disrupt networks by expanding analysis and investigation beyond the groups and individuals that place devices to analyze and target the finances, materiel and supply line of parts, and “the brains” that build and deploy IEDs.
DHS knows that the insurgents who seek to place IEDs in the United States (as they do in other parts of the world) are often supported by organized networks that finance their operations, supply critical elements for the production of IEDs, create the devices and plan and execute attacks. The proposed solution shall implement powerful analytics to gain critical, data-driven insight into the structure, character, interactions and methods associated with those networks. By analyzing data from a myriad of sources, the new solution shall identify and analyze the linkages between individuals and groups that may indicate a support network.
1.4. Supporting Analysis
DHS has undertaken an array of activities designed to prevent the detonation of IED/VB-IED/suicide bombs inside the United States and against American interests abroad. The department is aggressively working to focus on identifying and attacking the threat before terrorists have the capability to detonate a device. That begins with attacking the foundation of the threat—the social, operational and financial networks. Critical to the efforts is developing an integrated, cross-agency data-driven foundation of intelligence as the basis for deterring and incapacitating those who supply/obtain the funds for IEDs, identifying the organization planning to manufacture and plant the IED and intercepting the gathering and procurement of materials for the IED.
1.5. Mission the Proposed System Will Accomplish
The proposed solutions is envisioned to be a seamless, transparent and an integrated combination of COTS software, training and services that form an intelligence collection and analysis system that will help uncover and target the operational, financial and social networks involved in IED deployment in the United States. The solution shall address the challenges of data access, integration, quality and management of data coming from multiple government agencies and publicly available sources. In the modern and developed world, where most of the explosives/IED support networks operate, government agencies and the private sector generate unprecedented volumes of data. Customer profiles, organizational operational performance and personal behavior of individuals are monitored by multiple service providers. Some data resides in structured form in databases or exists as real-time streams. Some exists in unstructured form, for example as e-mails,
4
electronic documents or media files. Whatever the form, there exists potential to transform these data into relevant intelligence to improve investigative decision-making. A proposed solution shall integrate existing data from all relevant sources, and with its advanced analytics and reporting capabilities, provide actionable information to U.S. investigators in the full range of Federal, state, local and tribal jurisdictions.
This must be a cross-agency solution that is designed to deliver a broad range of intelligence products within a multi-level environment (Federal, state, local and tribal jurisdictions) to provide the full community of decision makers, analysts and investigators with better information to address potential threats.
Because first responders are an integral part of a tactical, pre-initiation response to an immediate threat, the proposed solution must address the appropriate type and level of information that would support those contingencies and how that information would be shared at the first responder level.
A proposed solution can use an integrated suite of tools, including but not limited to data integration and management, data and text analysis, predictive modeling and optimization and social network analysis coupled with link analysis. Analysts and other end users will receive detailed intelligence developed using data driven investigative techniques and link analysis based on social network theory. Analysts will be provided with client tools, such as customizable report creation and delivery capabilities that provide intelligence in the most appropriate format for decision makers and other users. The solution can be customizable, if desired, at all levels of security classification. The data will be searchable via a graphic user interface that will allow the investigator/analyst team to search for unusual behaviors and complex sequences of behaviors across records.
1.6. Operational and Support Concept
1.6.1. Concept of Operations
A solution will bring together key elements of intelligence needed to enable analysts, investigators and decision makers to make data driven decisions to more effectively perform their mission.
Specifically, the proposed solution shall:
* Link disparate, cross-agency data sources and integrate required elements
* Enhance data quality and accuracy
* Develop information across large volumes of integrated data
* Use data/text mining, predictive modeling and other advanced analytics methodology to provide insight to relevant data
* Expediently operationalize intelligence
* Identify suspicious social, financial and operational networks that may be appropriate for further analysis or investigation
5
* Communicate actionable information out to decision makers and investigators via the Internet, LAN/WAN, email, etc.
* Enable the agency to better detect and defeat potentially dangerous networks
A solution must integrate the efforts of analysts and decision makers at a cross-department/cross-agency level in a “fusion center” type environment. Sources of record will be made available so that the solution can automatically pull data on a near real time basis to update the intelligence available. A proposed solution will work within the specified DHS IT environment and will pull data from existing systems. Access to data sources will be granted by the responsible agency.
1.6.2. Support Concept
The responsible department/agency will consider options for the implementation and sustainability of the proposed solution.
The data integration and management, along with the analysis and modeling, social network development, linking and scoring functions shall be maintained at the agency level, with analysis and reports made available over the appropriate networks to users based on role or persona. Analysts, based on their role and mission requirements, will be given additional analytical capabilities to better perform their responsibilities.
A proposed solution will support the full range of services required for sustainability of the system. This shall include data integration and cleansing, data and text mining, predictive modeling, social network development linking and scoring. A solution will include proposed actions that the agency could take to develop the appropriate skills within the organization, particularly those related to data extraction, cleansing, integration and intelligent storage.
Training in the operation of the system, both for the initial implementation and for long term sustainability, shall be provided as part of the solution, as well as tailored courses, delivered on site, that focus on specific agency issues and requirements.
2. Threat
The potential threat to the United States from improvised explosive devices, vehicle-borne IED (car bombs) and suicide bombers is well established. Incidents since the beginning of 2000 in Bali, Madrid, London, Libya, as well as the attacks on the U.S. in Oklahoma City on 19 April 1995 and the World Trade Center bombing on 26 February 1993 attest to the potential threat and difficulty in protecting against them. This is not a new phenomena, as seen in the actions of the Red Brigade and Bader-Meinhof Gang directed against U.S. interests in Europe in the mid 1970, the attack on the U.S. Marine Barracks in Beirut in 1983 and others. Current intelligence and law enforcement estimates predict that these types of attacks against the population of the United States are inevitable.
Specific types of attacks could be the type of IED/VB-IED/suicide bombing widely employed in current combat zones and around the world, but could also include use of explosive devices
6
combined with commercially available, stolen or smuggled biological, chemical or radiological agents to cause further loss of life, widespread panic and economic damage.
Nearly every incident of IED/VB-IED/suicide bombing involves groups or networks of individuals acting in concert. While these networks may vary in type and complexity, they have common characteristics, such as the need to communicate, fund operations, procure materiel and travel to accomplish their objectives. These activities leave transactional records in databases legally maintained by government and commercial entities. By leveraging and extending insight into these data sources, it is possible to assemble a threat profile to protect against future attacks.
3. Existing System Shortfalls
Current systems fall short in the following areas:
a) Lack of the capability to integrate data from disparate sources. Data is contained in multiple systems within disparate organizations. It is often on different platforms and in different environments with different security and access requirements. Some is in transactional systems such as Oracle, SAP, DB2, Microsoft desktop applications and others, some in proprietary databases, some in legacy systems built in FORTRAN, COBOL or ADA. Integrating the sources is a complex technical problem, complicated by the various internal departmental/agency policy and cultural issues.
b) Volume of data to be analyzed. Not only is data in various agencies, formats, platforms and environments, the amount of data that should be considered in a comprehensive program is quite substantial, with gigabytes if not terabytes available for analysis. Since many of the transactional sources are updated in real time and others on a daily basis, the volume of raw data to be extracted, integrated and analyzed as required to provide timely, actionable information to analysts and investigators is a significant challenge.
c) Solutions lack scalability and robustness. Current less-flexible and less-capable systems have issues such as how new agencies or data sources can be incorporated into the data integration regime. Because current solutions are typically single agency efforts, the requirements for scalability and robustness are typically not addressed.
d) Lack of advanced analytics. Agencies typically do not have the capability to apply high end or advanced analytics, such as data indexing and profiling, data and text mining, predictive modeling, forecasting and optimization to their mission requirements. The types of network analysis and network linking required cannot be accomplished using multiple purpose, less sophisticated technology.
e) Absence of a foundation in social network theory. Department/agency personnel do not have a thorough grounding in the theory of how social networks interact and change patterns of behavior. This hampers their ability to gain maximum intelligence from existing data. The use of social network theory domain experts, thought leaders, academicians, investigators and analysts, operating within the appropriate technology environment is not optimal.
4. Capabilities Required
4.1. Operational Performance Parameters
7
The performance metrics included as part of the Threshold (T) and Objective (O) Values are based on 10 government data sources, 300 total users – 50 of them concurrent – located in 3 locations within the United States.
Objective Value
Threshold Value
1
Data Integration
a
Integrate, cleanse and store data from multiple sources.
Demonstrate the capability to perform the objective within 4 hours in 100% of the identified requirements.
b
Pull data on a schedule from disparate data sources
Demonstrate the capability to perform the objective within 1 hour in 100% of the identified requirements.
c
Conduct data cleansing and initial profiling as appropriate
Demonstrate the capability to perform the objective within 2 hours in 100% of the identified requirements.
d
Write the data into a data warehouse
Demonstrate the capability to perform the objective within 1 hour in 100% of the identified requirements.
e
Create a metadata repository with full bi-directional linkages with all Data Integration, Reporting, Data Visualization and Advanced Analytics components
Demonstrate the capability to perform the objective within 1 hour in 100% of the identified requirements.
2
Conduct analysis on the data available. This includes indexing and profiling, integrated data and text mining, predictive modeling, forecasting and optimization as required to meet mission requirements
Demonstrate the capability to perform the objective within 2 hours in 100% of the identified requirements.
3
Develop networks , along with soft and hard links, based on the data provided
Demonstrate the capability to perform the objective within 4 hours in 100% of the identified requirements.
4
Score the networks as benign or suspicious based on criteria established by the PM
Demonstrate the capability to perform the objective within 2 hours in 100% of the identified requirements.
5
Identify and list key network behaviors and potential vulnerabilities
Demonstrate the capability to perform the objective within 2 hours in 100% of the identified requirements.
8
6
Push information out to end users in multiple formats (portal, PDA, reports, alerts, etc) based on responsibilities, roles and access rights
Within 4 hours of completion of analysis and upon release by the appropriate authority
7
Provide standard interactive, parameterized “What if” interfaces for end users based upon model outcomes and parameters.
Interfaces updated with latest models and parameters within 4 hours of completion of analysis and upon release by the appropriate authority
8
Provide controlled data access via system metadata layer for ad-hoc, reporting, data visualization, and advanced analytic modeling to end users based on responsibilities roles and access rights.
Within 4 hours of completion of analysis and upon release by the appropriate authority
9
Provide integrated accuracy monitoring of “predicted” versus “verified” condition monitoring of scoring outcomes with threshold triggers to recalibrate or redevelop scoring algorithms.
Demonstrate the capability to perform the objective within 2 hours of the completion of scoring activities.
9
4.2. Key Performance Parameters (KPPs)
All Operational Performance Parameters are considered mandatory.
4.3 System Performance.
4.3.1 Mission Scenarios
The purpose of a solution is to provide analysts, investigators and decision makers the ability to gain more thorough and detailed insight of cross-agency data, using proven COTS technology, to better identify potential threats to the United States.
The solution shall be deployable into a cross-agency headquarters level environment operating at a minimum SECRET classified level. The three primary purposes of a solution are to:
* Extract, cleanse and integrate data from Federal, state, local and tribal agencies into a data repository, staged for the application of advanced analytics. Data must be accessed from a range of transactional, operational and individual sources in a variety of software platforms residing in different environments with operating systems with various levels of classification, potentially worldwide.
* Apply advanced analytics to develop data-driven intelligence on social, operational and financial networks potentially involved in domestic IED/VB-EID/suicide bomb attacks on the United States. The term “advanced analytics” is interpreted to include but is not limited to the use of text and data mining, predictive modeling, forecasting and optimization, within the context of innovative thought leadership to develop the most comprehensive and integrated understanding of a given threat, as well as how to validate and respond to it.
* Communicate the analytic results to decision makers, analysts and investigators within the appropriate cross-departmental environment for action. Information and intelligence should be available in a variety of outputs. Access and permissions must be based on roles and responsibilities.
4.3.2 System Performance Parameters
The Performance Parameters are addressed above in Section 4.1
4.3.3 Interoperability
* Interoperability, defined as the capability of applications to exchange information and to operate cooperatively using this information, is a critical aspect of this solution, since data will be integrated from disparate sources through the Federal, state, local and tribal infrastructure. The solution must sit atop the systems identified by the department/agency and extract, cleanse and load to a data warehouse or repository (or potentially multiple repositories) with minimal human interaction and full transparency, the ability to audit, read and write in native
10
language to the source systems is also required. A solution shall reduce overhead and make access to the required sources and data elements efficient and timely.
4.3.4 Human Interface Requirements
A solution shall operate in a controlled, IT-type environment at a department/agency owned or contracted facility within the United States.
The solution shall comply with all Federal core configuration requirements. 4.3.5 Logistics and Readiness
* A solution shall include both production and backup architecture to ensure the solution maintains at least a 99% rate with a ? 3 hours resumption of service capability in case of catastrophic failure.
* Logistics (maintenance and supply) requirements are addressed below in Sections 5.1 and 5.2
5. System Support
5.1 Maintenance
Maintenance of the hardware for the hosting system shall be the responsibility of the department/agency. Maintenance of the technology platform, including resolution of technical support issues and installation of upgrades or fixes, will be the responsibility of the solution provider.
5.2 Supply
The operating environment—including a combination of separate development, test, production and backup environments—will be operated and maintained by the department/agency with solution-specific supplies or spares for sustained operation with 99% availability.
A solution will include both system and agency-specific solution documentation, tailored for agency use and delivered both online and as standalone media (CD/DVD or thumb drive), in required.
5.3 Support Equipment
A solution shall require support equipment that is readily available as commercial-off-the-shelf equipment.
5.4 Training
A solution shall provide for the full range of training needed to operate the system and provide the services required at Technology Readiness Level (TRL) 9. A solution will specify the specific training provided to ensure that users are capable of operating and using a proposed system. It will identify the training required for each component of the
11
overall solution along with metrics to verify that each person participating in the training is certified upon completion.
A solution will make use of online and self-paced learning modes, but, where appropriate, provide for classroom training. Classroom training should be tailored to the needs of the
agency-specific solution and be conducted training facilities provided by the agency.
Training materials will be provided, in electronic format—online or via portable media—for all courses, including online or distance learning modules.
5.5 Transportation and Facilities
A solution will be hosted in a government-specified, controlled environment with no anticipated requirement for transportation.
6. Force Structure
A solution must consist of COTS software deployed in development, test, production and backup environments, with the required hardware sets provided by the government. The development and test environments will be used to tailor a production-ready, COTS technology platform to ensure that the environment available to the end user community is at TRL 9. The environments could be separately located.
The system shall be certified for use at least at the SECRET classified level and operate in both the unclassified and classified environments with the appropriate safeguards in place.
The solution should scale to accept classified and unclassified data feeds from an unlimited number of sources, regardless of the host operating system or base application. In addition, the solution should accommodate manual input from unlimited number of users via the Internet or WAN/LAN, as well as an unlimited number of end user via the same connectivity.
7. Schedule
A solution will be at TRL 9 within 6 months from the day access to the identified data sources is confirmed by the agency Program Manager and the solution provider. TRL 9 is generally defined as the ability to do the following:
* Link disparate, cross-agency data sources and integrate required elements into a data warehouse/repository with a supporting metadata layer
* Enhance data consistency and accuracy using data quality/cleansing software
* Use data/text mining, predictive modeling and other advanced analytics methodology to provide insight to relevant data
* Identify suspicious social, financial and operational networks that may be appropriate for further analysis or investigation
12
* Push actionable information out to decision makers and investigators via the Internet, LAN/WAN, email, etc.
8. System Affordability
The hardware will be provided by the department/agency and the software component of the solution shall include all technology required for:
* Data integration, cleansing storage and management
* Analytics, including data and text mining, predictive modeling and forecasting
* Development and scoring of social networks
* Deployment of information to end users with a capability to run stored processes from within Microsoft Office desktop applications, do ad hoc query and analysis and drill down
* Full control of access to the solution by a local system administrator
* Full documentation at the system administrators, analysts and end users levels
* 24/7 Technical support
Training shall be included for system administrators, analysts and end users in Web-deployed and on-site classroom training packages.
The services component of the solution shall include:
* Installation and configuration of all software, including all fixes and upgrades
* Development of analytical models
* Scoring and linking within the context of the social network models
* Knowledge Transfer to government employees or contract personnel as directed
The total delivered price should be ? $500,000 (includes department/agency usage rights). In addition, provide a fixed price proposal for seat or facility licensing fee for users outside of the department/agency.
A conservative estimate of the potential available market is over 250,000 seats in the United States alone. It is conservatively estimated that there are more than 5 Federal departments/agencies identified as potential users for the proposed system.
13
FACILITATORS
MANUAL
By
Moe Armstrong, MA Founder of Vet to Vet
Ererra Community Center West Haven, Connecticut
Naomi Pinson, MA
Psychiatric Rehabilitation Fellowship
Ererra Community Care Center
VA West Haven, Connecticut
Values, Ideas, and Concepts:
Peer Educators Project and Vet to Vet have one central value. The statement below is the value that clarifies and informs all the others. If you ever feel stuck or unsure of what you are doing, read it and apply it to the situation you find yourself in.
Although we may use different techniques to facilitate different kinds of peer support meeting, please remember:
All people who attend peer support meetings have something to teach and share as well as something to learn from.
Mutuality:
Mutuality means that we share rather than lead. Mutuality means that people in the support meeting have similar backgrounds. We receive or have received in the past mental health services and are able to self disclose about our experiences.
Being able to self disclose about having mental illness is difficult for some of us. Within our meetings, we have the opportunity to overcome any shame or stigma that we have experienced because of our psychiatric condition.
We can overcome shame and stigma through sharing strength and hope through mutuality. Mutuality means that the learning and sharing is done through community and not by lone individuals.
Mutual Support:
Often peer support meetings are referred to as "self-help" meetings. This is not the way we see it. 'Mutual support' is a term to replace 'self help' and better reflects what we get out of going to support meetings.
Recovery does not take place in a vacuum - we all need and in fact must have healthy natural supports. We also need the support of our provider(s). Without such supports we are more vulnerable to relapse from both our psychiatric condition and/or our alcoholism/addiction.
Our meetings encourage, validate and support our recovery in an ongoing fashion. Just as we allow others to be there for us, we are also there for them. Therefore, peer support meetings are not "self-help" meetings, although our "self' will be helped through them; rather they are mutual support meetings.
Replacing the term "self-help" with "mutual help" gives definition and direction to the reality that we are learning, living and improving through sharing our strength, hope, and knowledge TOGETHER.
We don't just take. We also give back. With mutuality, there is an ongoing sense of us, rather than "I".
The question is not just "What can I get out of this?" Rather it is "What can I contribute to this?"
Help:
The idea that we need help or need to ask for help can be difficult. Yet, asking for help can be an unrecognized form of empowerment. People might have gone through life too proud, too afraid, or simply too cut off from others to ask for help. We have found that asking for help can give voice to a person.
Going to peer support meeting greatly increases our chances of hearing about or learning about ideas that can help us. Asking for help is part of the daily life of people living in a community. On occasion when we have asked for help it was not forthcoming, or not the kind, of help we needed. Continue to learn how to ask for help.
We can also accept what is offered to us. We may have been too proud or alienated to receive what is offered us. We didn't like the what was offered, the way it was offered, or it wasn't exactly what we asked for. Today we can recognize that getting the exact right help that we identify as needing is not always available.
Our friends, sponsors, family members and providers turn out to be perfectly imperfect human beings just as we are. The help we ask for does not need to be perfect, only good enough. Things can work out even if it's not exactly the way we think it should be.
We can have a million reasons for refusing help. Then, getting help becomes impossible. When we refuse help a lot then we can wear people out. They may not want to work with us or others. Still, we want to learn to ask for help when we need it. We also need to learn to develop the grace and gratitude to accept help. Learning to accept help and explain what help is being offered is something that we can learn in our support meetings.
Help for the Helpers:
As peer facilitators, we both facilitate and participate in our support meetings.
The benefit of this is twofold: 1) it protects from getting a "big head" (something that can very much negatively impact our own recovery), and 2) it means that we, too can ask for help. Many
of us don't ask for help when we need it. We don't want to become 'burned out" and let down. Peer
support facilitators need to learn how to get help along the way. Even though we are helpers, we can need help.
As facilitators, we learn how to get help before we are desperate for it.
Many of us who have mental illness or who are dually diagnosed wait until we are in crisis to ask for help.
Attendance in peer/mutual support meetings is one way to reduce some of the negative aspects of our mental illness. We don't have to wait too long to ask for help. In our meetings, we share on a day to day basis. We head off crisis before it happens. We develop the support that we will need before crisis happens.
This is both "prevention" and "intervention."
Prevention:
Prevention is more than just a form of symptom reduction. Prevention from the negative aspects of our psychiatric conditions usually means that we will need a big change in the way we are living our lives. For instance, we will need to develop relationships with other people who are also in recovery - people who are traveling the same road we are on, or those who have gone before us burning the trail. Prevention might mean that we will have to learn to take better care of ourselves. We might need to learn to exercise every day and to eat right. We might have to learn to turn off the T.V., open a book or take a course. If we don't have money, this may mean asking a friend to show us how to do something, leaning something new that can enhance our lives, not just keep us standing still. For sure, we will have to learn how to have a calmer and healthier life style. This includes having people around us who can support our changes.
Intervention:
Still, our mental illness might not completely go away. We can rid ourselves of some of the triggers which can send us into crises, and we can learn better techniques for dealing with crises, but we can't avoid all of life's problems, big and small. Peer support, including developing friendships that are available to us outside the meeting, can help strengthen us so that when that crisis inevitably presents itself we can meet it with calmness and equanimity rather than blowing ourselves or others out of the water. Therefore peer facilitators need to learn how to keep ourselves healthy so we can continue to learn, grow and work, giving to and taking from others. We continue to learn how to get help as we facilitate the support meetings.
Getting the Right Amount of Sleep is Critical:
One important aspect of changing our life is that most of us who have learned how to do well with their psychiatric conditions have learned developed a sleep routine. There seems to be a link between becoming psychotic and problems with sleeping. One problem is sleep deprivation. Even if we are not able to sleep well, we strive for extra rest if we need it. Learning how to get extra rest and the importance of rest and sleep can be practiced. We can learn the importance of rest and sleep before we become so stressed out that we are psychotic. If we have not been able to rest we can take some time off for self care. We can call our providers and work with them before we return to facilitating the meeting. This is one of the reasons that we work in pairs, that there we are "co-facilitators." Co-facilitators will be there for the meeting when we need to pull back.
The opposite problem can be that we find that we are sleeping all the time. For example, we may find ourselves sleeping through appointments and fun things, too. In this instance we will need to discipline ourselves to stay in bed for just the hours we have designated to sleep, regardless of whether we sleep then or not. If we miss a night or part of the night's sleep because of this problem, we can decide not to sleep through the day and plan to keep our day very low key, recognizing that we will have some short term sleep deprivation. That night we usually will be able to sleep.
If we still can't sleep, we act, and call our doctors. We may need some chemical support to ensure that we do get some sleep. When we get that chemical support we will want to may sure that our doctor's understand that we are not asking her or him to medicate us for our symptoms only. Rather we are asking them to medicate so that we can function better. Clearly if we are too sedated, this won't help us to be able to function. Sleep is a critical ingredient in both getting better and managing our illness. With proper nutrition, exercising and scheduling we should be able to achieve a good sleep routine.
For Facilitators:
We try to ensure that we are rested and clear headed before we facilitate a peer support meeting. If we are feeling sleep deprived and out of sorts for an extended period then we may need to pull back. We can allow ourselves the luxury of being "just" a participant for a while.
We may need to ask trusted people in our lives to help us make this decision. If we stop for a time, this doesn't mean all is lost. On the contrary we can provide a tremendous power of example to others to respect their own healing process by respecting our own. These support meetings can be very healing.
By our behavior we are saying that each person has his or her own rhythm and pace, we count and that we are worth the extra time and effort it takes to get ourselves 'right' again.
Pacing Ourselves:
Personal pacing is necessary to help maintain stability. Sleep and rest balanced with activity is important to help maintain stability. A certain of level of activity, especially activity that involves physical exercise during the day can also help bring on sleep at night. Balancing work with play is another way in which we can help ourselves to become more organized and healthier.
If are not employed we can think of our 'work' as learning something new every day. If we work all the time, or if all of our time is devoted solely to leisure pursuits, we are out of balance. Being out of balance can lead to being out of sorts with our self and with others in our life.
Knowledge:
We structure our meetings around studying materials related to our mental health or dual diagnoses:
Learning the four S's, how to become Sane, Stable, Safe, and Sober happens through attendance at peer/ mutual support meetings. Sometimes this may take regular attendance over a period of time. Attendance may have to be done slowly until we build up our tolerance to a group or meeting format.
The benefits of meetings are incalculable. For the first time we are probably hearing from people who like ourselves, have one diagnosis or more of a major mental illness. And we are hearing how they successfully cope with it.
What, how and why we study:
A great deal of knowledge can be generated out of our support meetings. That is why Vet to Vet and the Peer Educators Project have an educational foundation. We read material from current thinking about mental illness and then have a discussion. In that way, we gain a strong educational foundation.
We don't sit around and just talk about how we feel, although some of our materials may generate strong feelings. It is important to recognize people need for help with stabilizing and understandings their
feelings. However, the cornerstone of the Peer Educator and Vet to Vet support meetings is education.
Gaining good and useful knowledge is important. The term educational is used here because we are learning from books, magazines, videos and other materials about mental health, then sharing what has helped us in our own experience of mental health services.
How to Select Materials for the Meeting:
When we choose out materials, we will want to choose carefully. There is a lot of misinformation about mental health and addictions out there.
We choose and read our articles and watch visual materials (DVD's etc.) carefully. We can ask a more experienced member or a mental health professional what they think before we share it with the meeting. If an article does not have good scientific evidence, that is, if it reflects only the opinions of various groups or an individual, we may choose to present it or not.
If we do present it, we will want to tell the members that it is just the opinion of the writer or the organization they represent and has not been researched. Most of the time we will be able to find scientifically recognized or researched materials in mental health to use as a starting point for our discussion. In every support meeting we are learning from our materials as well as learning from each other.
So, scientific articles that are well researched and respected by the field of mental health services are good.
Other materials might be poetry, stories, and even songs that express or relate to our mental health and substance abuse recovery.
Some of the materials we have found helpful are:
The Recovery Workbook (Exercises, text)
The Experience of Recovery (First person accounts of Recovery): Published by Boston University by Martin Koehler, LeRoy Spaniol and Dori Hutchenson
Through the Seasons (Poems): Published by Boston University Press by Moe Armstrong These books can be ordered through Boston University Telephone 617-358-1837
The Hip Pocket Recovery Workbook, Koehler, Spaniol and Hutchenson, Boston University Press, and condensed by Moe Armstrong
This can be ordered through the Peer Educators Project 617-277-9799 The Well Recovery Action Plan (WRAP) and WRAP for Dual Diagnosis
An advanced directive, preparation for how to cope with psychiatric crisis, by Mary Ellen Copeland, Published by SAMSHA Telephone 602-636-4445
Co-Occurring Disorders Recovery Plan, (A Series by Foundations Associates)
Creating a Relapse Prevention Plan
Making Medication Part of Your Life
Learning About Relapse
How Medication Can Help You
These publications can be ordered through Foundations by writing or calling:
Foundations, 220 Ventura Circle, Nashville, Tennessee, 615-742-1000 or 888-889-9230
Recovering Your Mental Health (Series) by SAMHSA and CMHS
The books in the series are:
Action Planning for Prevention and Recovery
Building Self-Esteem
Dealing with the Effects of Trauma
Developing a Recovery and Wellness Lifestyle
Making and Keeping Friends
These books are free and can be ordered through Center for Mental Health Services, 5600 Fishers
Lane, Room 15-99, Rockville, MD 20857, 1-800-789-2647
Other books of interest are:
Lives At Risk, (about young people who are dually diagnoses with alcoholism or addiction and major mental illnesses) by Dr. Bert Pepper
Transforming Madness, (about psycho social rehabilitation practice in the framework of the author's brother's recovery) by Jay Nuegeboren
Report to Congress on the Prevention and Treatment of Co-Occurring Substance Abuse Disorders and Mental Disorders: By the U.S. Department of Health and Human Services and Substance Abuse and Mental Health Services Administration
These are some books and materials we have used and still use in our peer support meetings.
Also, the Psychiatric Rehabilitation Journal and Psychiatric Services are two magazines that have good articles that will bring more knowledge about the mental health system and mental illness. Magazines like Newsweek and Time sometimes have articles which might be interesting, and are usually clearly written and easier to understand than some of the journal articles. Keep searching. Keep learning.
A word about non-readers:
Special attention and welcoming must be paid to the member who cannot read or whose reading skills are limited. A member struggling with literacy problems should be encouraged to attend and try to read. If this is not possible for them, suggest that they listen and then participate in the discussion. Because someone can't read does not mean that he or she does not have much to teach. Sometimes people who cannot read or who cannot read with comprehension are excellent orators (speakers) and come from a culture or tradition that values speech more than the written word. Such people are valuable to the meeting because they can articulate knowledge well.
Remember, even the member who cannot either speak or read well is as valued as anyone else. In our meetings everyone is seen as having something of value that they can contribute to the meeting.
Facilitators must be able to read on at least a 10th grade level:
When finding a new facilitator we can ask them to read a couple of paragraphs from an article and then tell us what is in them. Not knowing how to read or how to read well is a very stigmatized problem in our society, yet one in five adult residents of the United States lacks this skill.
Since much of our meeting is based on reading, it is important that the facilitator have good comprehension of written materials. We can check this out easily if we are courteous and considerate. People who have problems with reading are often very sensitive about this topic, so be gentle. When you are recruiting a new facilitator here is how to check for this skill. Get the person to read from one of the articles or workbooks you will be using. Then get them to tell you in their own words what it means. Sometimes people seem to read well but miss the main points or have low comprehension. This is the better part of reading.
Other people, not used to reading out loud, may stumble or read slowly, but have an excellent grasp of the subject matter. By this method you can judge whether or not the person can be a facilitator.
Other Qualities of a Good Facilitator:
Facilitators are made not born. That is, even people who seem very unpromising can learn to facilitate provided they have these qualities:
They are genuinely interested in other people.
They are good listeners.
They speak softy but with authority (they are not afraid to redirect or teach the meeting format/protocol as needed.)
They use democratic process to decide on the format (for instance, no cross talk, no swearing or "purple" language are two that meetings commonly adopt.)
They use the format, not personal likes and dislikes, to guide the meeting.
They make sure that they are getting all the support they need to enable them to be of help to others.
They never let the meeting deteriorate into a monologue or dialogue, recognizing that all members should be given plenty of space to speak out.
They are sensitive to and aware of a member who may be shy, or have trouble sharing.
They are willing to walk the extra mile by reaching out to a member on the verge of relapse or who is simply having a hard time.
They take the time to familiarize themselves with local services (food pantries, shelters, Vet's Centers, etc.) and provide that information to their members as needed.
They understand the difference between a "war story" and a "recovery story" and can gently redirect the meeting towards recovery when it goes off base.
They never use the meeting to humiliate or put any member on the spot.
They are unafraid to ask for help when it is needed.
They never leave a member in need unattended to.
How to Get a Meeting Started:
There are two ways to approach setting up a Peer/ Mutual Educational Support Meetings
1. Inside the mental health system
Principles and theories developed in peer/mutual support meetings have been used in the substance abuse field for some time. Alcoholics Anonymous, Narcotics Anonymous and all the other twelve step programs are some examples of this. Until recently the mental health community has not embraced peers from our mental health programs as meeting facilitators or consumer employees.
There have been some consumer employee initiatives, such as the State of Virginia model or Consumer Case Management in Colorado. This manual expands on that and is aimed at increasing peer support that partner within mental health programs. It will improve our (both peers and providers) ability set up specific support meetings inside programs.
Many mental health programs have life skills classes but these educational classes are generally taught by staff and not taught by the people from the programs. Mental health has missed out on the life experiences that people with mental illness can bring to the mental health system. Such experiences are often more valid to consumers because they have been lived and are not just coming out of a book. Peer facilitators, peer specialists, and peer counselors are beginning to work in the field in many states.
The idea of having the people in our programs be the people to work in our mental health system is an idea whose time has come. . However, many of the jobs like mutual (peer) support meeting facilitators and peer specialists will probably be part time work. These jobs can go a long way towards helping supplement the existing work being done in the mental health system. Some jobs eventually may lead to full time employment.
In peer support meetings with the Peer Educators Project or Vet to Vet, we share educational materials on various topics concerning mental health. Then, there is discussion linking people's life experiences to these topics. We try to stay focused on how to make use of the material we are studying I thus keeping the focus on recovery.
Educational support meetings facilitated by peers are not group therapy sessions. They are more like witnessing or testimonials given by people about surviving and learning to thrive despite their illnesses.
Bringing together people in peer support meetings can give a structure where people start to learn from each other and can stay together to keep learning. Recruiting, training, hiring, and developing peer facilitators who come from our programs can bring about a sense of long term belonging and participation.
People can learn not only the characteristics of Mental Illness but also what the mental health system is about. Language is demystified and put into everyday terms. The diagnosis is seen as a part of each participant’s experience, rather than the sum total of who they are. People begin to feel more comfortable with themselves and the mental health system.
Peer educational support meetings are a chance to learn and teach about mental illness and the mental health system from one another and from providers who wish to participate as recovering people themselves.
2. Outside the mental health system
Peer/ mutual support meetings can also be done outside of the mental health system. Some people feel comfortable with talking about some of their concerns and having that discussion outside of the place where they receive mental health care.
However, educational support requires two elements:
1. People need to read some material which is recognized as valid or emerging as valid by the mental health establishment.
2. We call these get together meetings and not groups so as to distinguish them from group therapy. Support meetings are not group therapy sessions. Mutual help meetings are more like community town hall meetings/discussions rather than therapy.
Some people are not willing to go to the place where they receive those services and have discussions about their lives, strength, hope and knowledge. Having support meetings outside the mental health system is an option that has been going on for awhile. In the future, there will also be more and more educational support meetings both inside and outside the mental health system. However, most people who have serious mental illness live on very tiny fixed incomes. Unlike people in NA and AA they often do not return to competitive employment, at least not right away.
Thus, our meetings are not able to be self supporting and must be subsidized somewhere. Some of us have sought grants for this, some have asked only for a room to meet in. In general we have found the mental health system helpful and receptive if not always understanding of what we are attempting to do.
Outreach:
One of the elements of having a good turn out of participants in the support meetings is learning how to do outreach. The time needed to do outreach needs to be set aside. Many people from our mental health programs have not attended these support meetings in the past. Getting people interested and encouraging them to attend meetings is also a job equal to facilitating support meetings.
The first rule of outreach:
Go where the people are. Find out where the mental health programs are - and go there! Meet people including the people who take care of the people.
Learn to network with the directors of the agency and find out when the community meetings take place. Learn to do public presentations and explain where the support meetings are taking place and what they are all about. Practice public speaking.
Experience in facilitating support meeting can help in doing outreach, also. Facilitating a support meeting requires public speaking. Develop informal relationships with providers who can support your efforts and refer people to the meeting.
Providers need education too! Too many peer support efforts have foundered and failed from leaving out this essential step or doing it half heartedly. When you present/public speak, remember to STAY POSITIVE. After all, you want something and are more likely to get it through praise and gratitude than criticism and anger.
Even if you have had bad experiences in the system (and who hasn't?) don't lead with these. Later, when trust has been established you will find out who is open to change and who has the power to effect that change. Right now, you are just about building relationships. Learn to do a all that can be done to connect with people. Connecting with people is a skill which needs to be learned.
How to Connect:
Connecting requires a genuine interest in the person or people who we are with. We enjoy our time with people. We get to know who they are and what we can do together. We understand where we agree and where we disagree. We don't expect everyone to agree with us. Learning how to connect first before helping is important. So in our connecting with people and learning how to be helpful, we need to:
1. Learn what are people are saying.
2. Learn who they are and what they need and want.
Again, we need take a genuine interest in the person who we are with.
Try to have a regular conversation with people in the support meetings. Don't active listen or therapies the conversation. Just talk. Remember, connecting with another person sometimes also requires some level of self disclosure. Conversation needs to be a two way discussion. Peer support conversational style requires that people share about their hopes, dreams, miseries and struggles. Remember, there can be times when listening to another person is a job. Staying alert and connected with another person can require extra effort.
Being a peer facilitator is a job. Seeing each person fresh and not being judgmental about our past experiences with them is also important and hard work. We have probably spent time with some of the people in the support meetings by our past time together in the mental health system. We can connect with the people who we are with in way that is comfortable to them and to us. We can start fresh connecting with some of the people who we have known. If we don't know them, we know someone who knows them and/or they know some one who knows us. Most of us have been moving together through the mental health and/or substance abuse services for awhile.
Self disclosure isn't a just a choice. Self disclosure is a form of honesty which comes from recognizing why and how we got into the mental health system. We are helping ourselves as well as helping others. We are trying to gain some knowledge of our psychiatric condition and knowledge of our time in the mental health system. We can share that knowledge with each other.
Presence:
Presence is a special state of attentiveness-not only attentiveness to another person and the environment, but also attentiveness to our own personhood. Through presence we can be of better assistance to other people and to ourselves. We can be present and not be intrusive.
Presence is paying attention-paying attention with a sense of sincere interest in the people whom we are with, the environment we are in, and the experiences that we are having. We are aware of our self and others. We pay attention without being intrusive. Again, we can pay attention by having a authentic curiosity about the people who we are with and our time together. Therefore, we pay attention because we enjoy the situation that we are in. We can discover, rediscover, and enjoy the people who we are with.
As meeting facilitators, we understand that we work for the betterment and improvement of others and of ourselves. We have come to terms with the fact that we are good people. We work hard and we make a difference in the lives of other people.
Through realizing that we do make a difference, we gain the strength to continue the work of recovery for others and for ourselves.
All people are recovering at various times from disability, trauma, disappointment, and difficulties in relationships. Because the process of recovery is common to all kinds of situations, we can learn from one another. We can learn new ways to promote our mutual recovery process.
Presence is not seen. Presence is not thought about. Presence is like a breath. It happens even when we aren't thinking about it. Presence is being active when not moving. We are alert and attentive without be hyper vigilant. Presence is being here, right now in the moment, experiencing the moment-without holding on to it. Presence is also taking time to continually learn about others and ourselves.
We need to have personal strength to work in mental health. Presence can strengthen ourselves for the moment when we are needed. Through presence, we learn to have confidence in our stamina. Through presence, we can strengthen our minds so that we find solutions with others as they need them. Presence also requires renewing ourselves, as well as being present.
Presence is understanding ourselves and pacing ourselves so that we aren't exhausted. Presence means that we don't inappropriately try to take over another person's life. We do not know how to lead another person's life better than they do. We do not need to be the expert.
We do need to learn is how to work collaboratively with others-respecting their ability to understand their own life, to change situations that haven't worked for them, and learn how to assist them in their own best interest.
Through presence, we can learn how to pace ourselves both physically and mentally. We can slow down, assist each other to focus on what is happening in the present, and preserve our energy for the collaborative work that real change requires.
Presence means getting out of the office and into the lives of people. Discovering and exploring are also parts of presence. Presence makes discovering. We are focused on what is happening and open to learning.
We are probably born with presence and attentiveness. The natural state of most infants seems to be one of presence and attentiveness. We often lose this sense of presence over the years as we find it more and more difficult to safely be ourselves. Yet, we can learn to accept ourselves, to love ourselves, to feel safe with
ourselves, and to have presence. We can be present and attentive.
Again, presence can only work if we are moving toward physical and emotional health. Attentiveness can only work if we are rested and emotionally healthy. Through presence and attentiveness, we are trying to promote a life which is healthy for us and for other people. And people with mental illness can achieve physical and emotional health-even if the elements of the mental illness continue.
We can pay attention to others only if we can pay attention to ourselves. If we can know who we are and what we can do, we can develop a sense of presence.
We don't mean false pride. We mean humbly going about our work, open to our experiences and the experiences of those people we are assisting. We need to have presence and be .attentive. We need to have presence to assist another person to pull his or her life back together.
We can have presence. We can be attentive. We can assist others because we value them and ourselves. Exercise:
1. Break into small groups. Three to four people in each group. Turn your back the mutual (peer) support meeting facilitators. The facilitators will start to talk and you will have to pay attention. The secret is being attentive without eavesdropping. Be interested without seeing. Turn around and have a discussion about what you heard. Don't analyze. Don't use psychiatric buzz words to define what took place. Just talk with each other.
2. Keep in the small groups. Talk about what you did for the week. Have a discussion with everyone in the group, including the facilitators as part of the conversation. People can also talk about what they would like to do. Practice having mutual conversations. Stay away from anyone being a leader or dominant.
3. Quieting exercise. Sit with hands folded and breathe in quiet for ten minutes. Just think, count your breaths and relax your mind. Breathe in and out. Listen to your breathing. Slow down. Relax.
Assisting and Being Assisted:
First we need to be very clear that assisting another human being is an honor. Assisting another human being is also a valid form of work. Assisting is a job which needs to be done. And conversely, there is nothing wrong in being cared for. There is nothing wrong with being assisted by another human being. We only do our work as long as we can.
When we are tired then we need to rest. We need to stay flexible in the time we spend with people so that we are rested. It is not necessary to burnout in order to help others.
Helping others is not a burden. Our work can nourish us, if we can learn how to have presence and be present we can become nourished from our work because hope, respect, warmth, and clarity are attitudes which can nourish and renew people.
We assist human beings because we are compassionate. We are committed to compassion. Our compassion comes from understanding and accepting ourselves. When we can feel compassion toward ourselves then we can show compassion toward others.
There is nothing wrong in showing compassion or caring for another human being. We show compassion to our fellow workers and the people who receive our services. We are trying to build a community of mental health care and assistance. Therefore, we can mutually share compassion among ourselves, our fellow workers, and the people we are assisting.
We can give compassionate care and we can also receive compassionate care. The frustration of working in today's mental health system can make us forget that all people with mental illness need our presence and attentiveness. By strictly defining rules, regulations and policies, workers and clients from our own agency or department of mental health can create tensions in the lives of people.
The struggle of how much to assist another person is unending. How much assistance a person needs changes daily. We are in a constant flux of trying to understand how much assistance can we give and how much assistance does the person need and want. Also, we get tired. We make mistakes. We say and do the wrong things. We get angry. We get sad. We are not able to do all we would like to do.
Presence and attentiveness are attitudes and practices that need to be regularly reviewed and updated. We think that we are attentive at one time. Then, we find out how to pay better attention to the people who we are with. We learn to better enjoy the companionship of the people who we are with. We are continually learning how to better documents our observations through the conversations that we had - not just on clinical judgment.
Educational peer support meetings require that facilitators not only show up on time and start the meetings. Learning to have presence and assist people is just as important. We need to practice and learn those skills through the following exercises.
Exercise:
1. There can be a treasure hunt. The treasure hunt will go to specific points in the building. The treasure hunt will focus on people opening doors for each other. People will find the information at the sites and
share the information with the people in the group.
At the end of the treasure hunt, there will be some snacks. People will give each other the snacks and eat them. The idea is doing for each other. Inspirational quotes will be the treasures which are being searched for. Maybe, write on a piece of paper and in a colorful box.
The snacks at the end of the treasure hunt will also give people the time to talk about the inspirational messages that they gathered from the treasure hunt. People learn to do tasks with each other while talking.
2. Quiet. Hands folded sitting down and deep breathing for ten minutes. Count our breaths. Keep reminding ourselves to relax, clear our mind, relax.
Assisting:
Our job is to assist other people with a psychiatric condition to go through life. To assist another person, we need to also be healthy and maintain our own health. The concept of assisting comes from the independent living movement. Personal care assistants work with people who have physical disabilities. They are assisting not helping. Assisting give the impression of people working side by side. Assisting also implies collaboration and mutuality. There are fewer interpersonal barriers with assisting.
Assisting can be very practical such as sharing chores around the house. Assisting is doing with someone not doing for someone. Assisting can include conflict resolution. Assisting also involves learning from those whom we assist.
Assisting a person with the use of presence means not using some listening technique. We listen and pay attention because the person means something to us. We listen because we are present and attentive to the person. Assisting becomes a natural response to a person in need.
Presence and assistance also means building relationships and coming to a variety of interpersonal understandings before or when a crisis occurs. We get to know the other person and the other person gets to know us. We share observations and perceptions. We share experiences. We become neighbors in a recovery oriented mental health community-neighbors who are present, attentive, and assist one another.
To be assistive, we drop judgmental language, ie., she's a borderline, he is psychotic. We can start to learn about who we are working with. Labels create distance. Labels create barriers. Labels stigmatize and stereotype people. If we label people, we are not likely to get to know them or to take an authentic interest in them-and we will be less likely to assist them.
Giving assistance is work. Yet, as we spend more time assisting people, we tend to more easily let go of our own prejudices and fears of other people. We get to know other people holistically. We know more about them than momentary conversations. We try to understand them by time spent with them in assistance and presence.
We can learn to enjoy and value the people with whom we are working.
Exercise:
1.Think of an activity that you would like to do. Think about something that you did or thought. Try to explain that to someone. One person listens while the other talks. Then, have both people talk about what was said. What was the reaction? Remember, there is no right or wrong way to listen. One person might listen and get all the "facts." Another person might listen and become inspired. Another person might want to do something or another person might come to a realization. Just find out how some one else thinks. Learn to sit next to the other person, like two people having a conversation on a park bench.
Ask a lot of questions, get a lot of opinions, and keep an open mind:
Many times in mental health, we try to have answers or solutions. People with a psychiatric condition can be suffering emotional anguish. Our heart goes out to the person and we want to do something to lessen the anguish and suffering. We set up these educational support meetings in hopes that we find some solutions. From the support meetings, answers, opinions, suggestions, and solutions seem like they should come right out to stop the suffering.
Asking a lot questions and getting a lot opinions can increase real attentiveness. We are really interested in a person. Therefore, we ask their opinion. We find out about them by asking questions. We really want to know about the person. This is why we ask questions and get opinions. Then, we can make suggestions after having asked questions and gotten opinions-short suggestions but not answers. We want people to come up with their own answers. Sometimes solutions come about over a long period of time. Many people don't object to being reminded. But, being reminded and having someone follow us around prompting us are two different things. We keep it simple. We ask questions. We get opinions because we are authentically interested in other people.
Exercise:
1. Divide into two or threes. Get a broom. One person sweeps the other person asks questions. One person has a dust pan. The two people go around sweeping and talking. People are able to find out about each other and work at the same time.
Remember that the sweeping also has symbolism. The floor looks clean. However, there is still dust to be found. We might think we know everything about a person. We still have a lot to learn. We also learn to keep active with work while we listen. We also pay attention without being intrusive.
Living Attentiveness:
To be attentive to others, we need to learn--be attentive to ourselves. Attentiveness is not over concern. Attentiveness is not being judgmental and paying too much attention to details. Attentiveness comes from enjoying our lives and participating in life.
Attentiveness is a way of caring and of being fully present.
Like tending a garden, we pay attention to what needs to be done. We also take time to smell the sweet air while we weed the garden. We also feel and smell the soil. The soil just isn't seen as dirt that will make our hands dirty. If we can't get all the soil off our hands, we learn to appreciate our soiled hands. We appreciate the whole environment and ourselves in the environment. Through full appreciation, we become attentive.
By being attentive, we can learn to love ourselves and others again. We learn to see how imperfect each of us really is. We try not to get swept away by our judgment of others. We stay balanced because we pay full attention to who we are and what we can do.
We are support meeting facilitators who work with other people. We all have a psychiatric condition. We explore the possibilities of recovery every day. Exploration requires attentiveness. We are attentive because we like ourselves.
We are attentive because we like the people who we work with. We grow to appreciate and like our environment.
We can also understand about and learn from our mistakes because we are attentive. We can develop humor and forgiveness about our mistakes and the mistakes of others.
Through attentiveness, we learn what the situation is and who the people are. We come to appreciate the people and the situation. We can chuckle over our imperfections that we encounter through our contact with people and with situations. This ability to chuckle requires forgiveness.
While we facilitate support meetings, we need to learn to stay attentive.
Exercise:
1. Sit around and explore our dreams. Talk about what we would like to do. Talk about what we hope to do. Listen to each other. Learn to be conversational. Set aside ten minutes to practice attentiveness. Repeat
back what was said and heard. Compare notes and see if other people agree.
Forgiveness:
Forgiveness seems hard to come by. Yet, what goes around comes around. We want to create cycles of forgiveness and not retaliation. Forgiveness means that we see the person and the situation the way they really are. We learn to value a person--blemishes and all. We can forgive because we have come to really understand the person. We suspend judgment even if we don't fully understand the situation the person is in.
The possibility of a place to live and something to do is what we offer in mental health.
We do as much as we can every day to assist people in not losing their opportunities to live a full life. We do as much as we can do, and then go home and rest.
We also have to learn to forgive ourselves when our plans don't go right. We also have to learn to forgive ourselves when the people whom we work with don't take our suggestions. We have to learn how to forgive ourselves, when other people don't follow the plan. We also have to be willing to go back and create a plan that might work.
Forgiveness is hard. Forgiveness requires constant presence and/or attentiveness. We have to keep present in our minds that forgiveness is a must and will be difficult. Forgiveness will be a challenge.
We are taught how to get even. Many of us grew up with the saying, Don't get angry, get even. There are many stories of retribution or getting even. There are very few stories of forgiveness. Forgiveness isn't glorified in popular culture. There are no block buster movies about forgiveness.
Forgiveness can be quiet. Forgiveness can go unnoticed. There doesn't seem to be any social recognition of forgiveness. What does forgiveness do for us?
Forgiveness builds strength of personal character. We discover better who we are and what we can do through the act of forgiveness. We come to understand others through the act of forgiveness. We come to actually have a glimpse into another people's experiences through the act of empathic forgiveness. We come out of ourselves and our judgments through forgiveness. We become flexible through forgiveness.
Still, forgiveness will be hard. Forgiveness is not really part of modern culture or thinking. We are learning to generate and create forgiveness in a cultural, moral, social, or spiritual vacuum.
Forgiveness will be difficult. Like meditation and reflection, practicing a little forgiveness every day might help prepare us for the big acts of forgiveness which we will eventually need. We might start by forgiving ourselves when we make mistakes or we don't do all that we would like to do. We can learn to forgive
ourselves and become repentant when we are not nice to other people.
Through forgiveness, we not only accept the errors of others--we also accept our errors. We forgive other people because we accept that we will also make mistakes--little mistakes, medium sized mistakes, and the occasional huge mistake.
Exercise:
1. Sit around and talk about a situation where we felt awkward and clumsy. Disclose gently some personal frustrations we have with our lives. Try to keep the groups to three or four people. Talk about when other people forgave us. Talk about when we forgave other people. Let the group discussion go on for about fifteen or more minutes. Ask ourselves did we learn a little more about how to forgive.
Forgiveness seems to one of the hardest attributes to learn and retain.
Acceptance:
Acceptance of our vulnerability is demonstrated by accepting assistance. Acceptance can be the extension of assistance. We acknowledge that we are in need of assistance. Through acknowledgment of our need for assistance, we come to the realization of our needs and wants.
However, everyone needs assistance. People and facilitators in our support meetings want acceptance. That is why we attend support meetings. We are looking for some assistance.
Our culture doesn't encourage the idea that sometimes we will all need assistance and care. We are not lone indestructible individuals who have to make it through life on our own. Everyone needs assistance sometime.
We can learn how to accept assistance and care from others. At some times in our lives, some of us will care for others. Some of us will learn how to assist others. And, some of us will be assisted.
Acceptance of our vulnerability brings us to a level where we can accept care and assistance.
By demonstrating acceptance of our personal vulnerability, we set the tone for other people around us to learn how to receive assistance.
Even with acceptance of vulnerability, our minds will continue to have negative thoughts. We will not always think kindly of other people. We will have thoughts like they could do better or we are not good enough. Acceptance will not help us drop the all negative thinking.
We go forward through forgiveness. We can have a temporary relief from this negative thinking. Forgiveness and acceptance opens the door for some moment to moment relief.
The anguish that comes through the conflicts of life can be lifted through forgiveness. Forgiveness is a practical approach to stress caused by conflict. We can forgive because we accept that we are all at times weak and suffering. We accept that we all make mistakes. We accept that everyone at sometime needs assistance. By needing assistance, we accept our vulnerability and the vulnerability of others. Accepting our personal vulnerability will be difficult.
In the support meetings, there will be people who will be disruptive. We need to learn forgiveness. We need to learn to bring people into our support meetings. We need to learn how and when to bring people into our support meetings. Forgiveness can give us the clarity in being welcoming to people.
Exercise:
1. Discuss our failures and disappointments. Talk about what we would have done differently. Talk about acceptance of what we have. How we didn't always get what we want. How to appreciate what we have. Who are we after things didn't work out.
Did people need assistance during those times of failure and disappointments?
Negative Thinking:
Many facilitators have quit because of negative thinking. We don't always think the best thoughts about ourselves. We also don't think the best thoughts about other people. We got discouraged and disillusioned. How can we stop this negative thinking? We have to work at filling up the negative thinking with positive thinking.
Affirmations can sometimes work in a crisis. When our minds are jumbled with horrible thinking, we sometimes need to keep repeating over and over some positive thoughts.
For the long term change in negative thinking, we will need our positive thinking to be internalized through educating ourselves. We need to learn how to write and discuss positive thoughts every day. Again, we can practice positive thinking.
Through practice of positive thinking, we can more easily fall back on the positive thoughts naturally. What are some of the ways to practice immediately changing negative thinking?
1. We make lists of what we have.
2. We learn to appreciate what we have.
3. We write down and discuss who we like.
4. We write poems and/or statements about the beauty around us.
5. We take time to appreciate nature (For instance, we let the wind and fresh air zap us).
6. We practice meditation. (Where we have no thoughts for periods of time).
7. We learn to read books and see things which are inspirational.
8. We restrict the images of the media. (Turn off television for extended periods of time or completely).
9. We accept our imperfections. We accept the imperfections of others.
We also learn to take out time for reflection and/or prayer.
What we are learning is that there are several ways of seeing a situation. We can think that we are either stuck or we can think that we are delayed. Stuck means that we think that we will never get out. We feel trapped. Delayed gives us the opportunity to feel like we are only momentarily caught.
By writing down what we have, what we enjoy, and who we think we are then we can use our time to move forward. Even when we feel that we are delayed. Negative thinking takes up time which we need to renew ourselves. Negative thinking can bring about anxiety.
Negative thinking can become so routine that we don't even realize that we are filling up with negative thoughts.
There will never be complete elimination of negative thinking.
However, we can have temporary relief from negative thinking.
Exercise:
1. Make a list of what we have.
2. Write down and discuss who we like.
3. Write poems and/or statements about the beauty around us.
A. Practice meditation. (Hands folded, quiet deep breathing, sitting in a chair or on the floor and set aside a time period where we reduce our thoughts for periods of time.)
Pause and Check It Out
Pause and check it out is learning how to step back in order to go forward. The goal of pause and check It out is to become closer to people through more thoughtful contact.
This is not an act of distancing.
This is a process for preparing ourselves to be of assistance more fully.
We need to take time out to understand what the person we are assisting expects of us and what he/she really wants us to do. Assistance requires that we understand what needs to be done. Before we give assistance, we need to learn how to pause and check it out. Rushing in to give assistance, we may miss what the person really needs.
Always ask before assisting. Always assume that people can be actively involved in the decisions that affect their lives. Do not assume that you know how to lead someone's life better than they do.
Pause and check it out is a form of renewal. We need to learn how to daily, or sometimes several times a day, get away from the chatter that passes for thinking inside our minds.
We need to rest our minds, refresh our thinking, and then go back into the world of work.
We need to learn how to clear and freshen our mind and the mind of the person we are assisting before we act. We need to learn how to pause and check it out.
We need to build the skills of pause and check it out. One of the best ways learning how to pause during the day is set aside time to mediate, reflect, and/or pray during the day.
For instance, we can get up the same time almost every morning and go to a place in our home. We can engage in quietness and stillness.
We can sit in a chair or sit on the floor for ten or fifteen minutes. We clear our minds. We remind ourselves while we sit that we don't want to think. We want to rest our thinking. We want to clear our mind. We can say, clearing my mind, clearing my mind, clearing my mind, while we are sitting and breathing. We want to get rid of the busy thoughts. Just have the simple thoughts going through our mind.
Focusing on a word or phrase can help us to clear our mind of thoughts. We don't want to focus on what is worrying us. Thoughts can be very persistent and if they reappear, don't be hard on ourselves--simply return to the word or phrase we are using like, clear.
Tech Prep Senior Project Manual
Created for use by Tech Prep High School Instructors in the Northwest Ohio Region including the following Consortia:
Maumee Valley Tech Prep
North Central Ohio Tech Prep
Northwest Ohio Tech Prep
TP Partners of Marion
West Central Tech Prep
Workforce Development Council
Developed Spring 2002
Special Thanks to the following team members who assisted with the development of the Tech Prep Senior Project Manual:
* Sam Bassitt, Director, West Central Tech Prep Consortium
* Karen Bleeks, Director, Maumee Valley Tech Prep Consortium
* LaDoyt Martz, School-to-work, Tech Prep and Job Placement Coordinator, Apollo Career Center
* Nancy Pietras, Director, Northwest Ohio Tech Prep Consortium
* Dr. Jim Piper, Educational Consultant
* Debra Schneider, Project Manager, Northwest Ohio Tech Prep Consortium
Materials found in this document were provided by a variety of sources. We would like to thank the following for sharing their materials with us.
Senior Project Booklet (Draft) from SREB
Handouts from presentation of Heather Boggs, SREB Consultant
Handouts from presentation of Patricia Clark, Consultant
Apollo Career Center
Clay High School
EHOVE Career Center
Lancaster High School
Vantage Career Center
2
Table of Contents
Introduction to Senior Projects
Senior Project Roles
Section 1: Guidelines for Senior Projects
Section 2: Choosing a Senior Project Topic
Section 3: Research Papers Section 4: Selecting a External Business/Industry Mentor
Section 5: The Product/Process Project
Section 6: Oral Presentation 42
Section 7: Assessment Process of Senior Project 54
Section 8: Junior Project Ideas
Section 9: Sample Post-Secondary Capstone Projects
Supplemental Materials in Binder:
CD of Mentoring produced by Lakeland Tech Prep Consortium
CD of Students Presenting Their Senior Projects at EHOVE Career Center Mills, Barbara and Stiles, Mary. “A Rookie’s Guide to Research” Union Creek Communications, Inc. Byrson City, NC. 1999
3
Introduction to Senior Projects
High school seniors are nearing the completion of 12 years of education. They have taken a variety of courses and developed an assortment of skills during those years. Now is the time for seniors to combine their knowledge and skills in a Senior Project to show what they have learned. A Senior Project provides an opportunity for a student to choose an area of interest, conduct in-depth research and demonstrate problem-solving, decision-making and independent learning skills. It contributes to a strong senior year of challenging courses and practical experiences that prepare students for the next step in the workplace or when pursuing further education.
A Senior Project involves several steps.
* First, the student selects a topic, gathers information, writes a research paper and keeps a portfolio of project activities.
* Second, the student produces a product/process that applies some aspect of the research.
* Third, the student makes a formal presentation to a panel composed of teachers, community leaders and peers who know about and are interested in the topic. After the presentation, panel members ask questions about the research and the product/process and find out what the student learned during the project as well as review the student’s portfolio.
The student is not alone during the project. Each student meets individually with his or her Tech Prep Teacher. The student may also work with related academic instructors depending on the design of their Tech Prep program. In addition, the student will work with a Product/Process Mentor who has expertise in the student’s field of study.
Senior Projects should be challenging. They should require considerable effort on the part of the student in showing what he or she has learned. If a student approaches the project step by step and meets the deadlines, she or she will be able to manage the project successfully. There are many benefits involved in a Senior Project such as:
* Foster student as “active learners”, capable of solving complex problems and constructing meaning that is grounded in the world beyond the classroom that encourages collaborative learning with industry and community partners
4
* Organizing curriculum around authentic student projects which serve as a basis for learning from career and academic disciplines in an inter-disciplinary approach
* Developing capacities not conventionally taught, such as, ability to work independently, problem-solving which involves students as expert-practitioners who use and demonstrate their knowledge and skills, etc.
* Engages students in complex, challenging tasks which allow integration of learning, generation of knowledge, reflection, and creation of a product, and provides the student with the opportunity to work with business and industry partners
* Meets rigorous and measurable standards for academic and technical performance that reflect global demands, are required of all students, and make a diploma meaningful
* Serves as a capstone to a sequenced course of technical and academic study
5
Senior Project Roles
Role of the Student:
1. Complete requirements of the project in accordance with the pre-established checklist and timetable.
2. Select External Business/Industry Mentor
3. Maintain copies (on disc and paper) of all work in process, rough draft, final draft, etc.
4. Submit (on disc and paper) rough draft of research paper to Tech Prep Instructor (or appropriate instructor based on design of Tech Prep program)
5. Submit (on disc and paper) final draft of research paper to Tech Prep Instructor (or appropriate instructor based on design of Tech Prep program)
6. Document product/process project selection with Tech Prep Instructor
7. Keep a project log or journal
8. Develop copies of oral presentation outline for evaluation panel.
9. Develop a portfolio to display work
10. Write and distribute teacher approved thank-you letters to mentor and panelist
11. Complete and return a Senior Project evaluation concerning this experience.
Role of the Tech Prep Instructor
1. Assist student with selection of project topic and final approval of project
2. Assist and advise student in the technical portion of the project
3. Oversee student in accordance to project guidelines
4. Review rough draft of research paper and advise student as to where revision may be needed.
5. Evaluate final draft of research paper
6. Allocate related class time to work with students regarding their senior project.
7. Evaluate the product/process project
8. Coach student in preparation for oral presentation and verify technical accuracy of presented materials
9. Create an end of project student evaluation survey form for the purpose of project improvement.
10. If there is no academic instructor connected to the design of the Tech Prep Program, the Career-Technical instructor will need to assist with the roles of the English Teacher that follows.
6
Role of English Teacher (If Tech Prep program design includes the academic connection)
1. Instruct the student in the proper form and process of the research paper
2. Collect and evaluation all written documentation pertaining to the Senior Project
3. Review rough draft of research paper and advise student as to where revision may be needed
4. Evaluate final draft of research paper to pre-established standards
5. Discuss with Career-Technical instructor regarding the content of research paper and integrate evaluations of form and content to final project grade
6. Coach student in appropriate methods of giving an oral presentation
7. Conduct a post presentation discussion with the student for the purpose of project improvement and implementation.
Role of External Business/Industry Mentor
1. Assist and advise student in technical aspect of research paper and/or product/process project.
2. Serve as resource to the student in all stages of the product/process project development
3. Allocate time and make arrangements for student to work on project as needed
4. Sign Mentor Agreement Form
5. Provide accurate and honest verification of student’s work
6. Additional assistance could be provided by:
* Reviewing student research materials
* Evaluate their product/process project
* Advise students in planning their oral presentation
* Attend oral presentation
7
Guidelines for Senior Projects
General:
The Senior Project is an integral part of a student’s final year of high school. It integrates skills, concepts and data from the student’s program of study into one culminating project. Students work on individual projects. Although, the product/process portion may have some joint relationship to other product/process project by other students, each presentation is completed independently.
A Senior Project consists of a written research paper, a major product/process and an oral presentation. Individual instructors will dictate how the projects are graded although this manual contains sample rubrics that might be helpful.
Senior Project Topic Selection:
* Topic must be developed around the student’s career focus
* Initially, the student will select several possible topics for teacher approval. (Sample Senior Project Topic Selection Sheet)
* When topic is finalized, the student will complete the Senior Project Commitment Form. (Sample Senior Project Commitment Form)
Senior Project Components:
Part 1 - Research Paper – a formal paper that encourages students to develop and demonstrate proficiency in conducting research and writing about a chosen topic.
* Must be research-based utilizing at least 3 types of sources of information and a total minimum of 7 individual sources. Types of sources would be periodicals, library references, Internet, personal interviews, technical manuals, etc.
* Documentation of related research information through note cards or other means of note organization develop the initial outline of the paper.
* Paper must be word-processed and conform to MLA or APA standards. A Rookie’s Guide to Research is included in this manual as a reference to MLA standards.
8
* All research papers should be 5-7 content/text pages in length. In addition to the content/text papers, there should be a title page and a bibliography. An Appendix support is optional based on topic.
Part 2 – Product/Process Project– a tangible creation based on choosing, designing and developing an item related to the student’s research topic.
* The student will spend a minimum of 15 hours outside class work on a project related to the research topic.
* The Tech Prep instructor will approve the Product/Process Project. (Sample Senior Product/Process Project Approval Form)
* The student identifies an external mentor to assist with the project. The mentor should be an adult accomplished and experienced in the chosen project area. Student will meet a minimum of three times with mentor.
* All documentation of the Product/Process will be included in a portfolio. The following items may be included
1. A log of the student’s hours, including dates and times in an hour-by hour log and a description of what you did during these times. Travel time, thinking time or time spent practicing your presentation DOES NOT count toward your 15 hours. (sample)
2. Journal entries about each of the student’s experiences (including dates). Entries should note obstacles, challenges, meaningful activities and encounters in a written discussion. (sample)
3. Photographs or other visual documentation
4. Letters of Recommendation from mentor or work-based connection as related to the project
5. Include any notes, papers, flyers, and/or charts that you may have collected and/or created about the project
6. Include a clean copy of your research paper
7. Include a completed mentor sheet. (sample)
8. Other items will also be included in the portfolio. (see Section 5)
9
Part 3 – Oral Presentation– a formal presentation of the project before a panel of judges. The presentation consists of a speech, an explanation of how learning was applied in developing the project and a discussion of lessons learned by the student.
* An 8-10 minute oral presentation (with note cards) given to an audience that might include administrators, teachers, student peers, parents as well as business and industry representatives.
* Speech content should include information about the research paper, the product/process project and how the student personally gains from completing the senior project. (See Oral Presentation information sheet)
* Include at least two types of visual components such as transparencies, computer generated graphics, PowerPoint, posters, etc.
* Student’s portfolio of information should be brought to the oral presentation for review by the audience. Students should be prepared to answer questions from the audience or review panel.
10
Suggested Time Line
Week #
in One
Semester
Week # in a Whole Year
Research
Paper
Product/
Process
Presentation
Portfolio
Item
2
4
Orientation
Orientation
Orientation
3
6
Identify topic
X
4
8
Begin research
Tentatively
identify
product/process and get parental
OK
X
5
10
Begin product/
process
journal/log
6
12
Preliminary
draft
Select
Community or
Business
Mentor
X
7
14
Submit initial product/ process plan
8
16
Review project status and journal
Outline Speech presentation
9
18
Submit final
Draft
X
10
20
Design Visual Aids
X
11
22
Review project status and journal
Journal
12
24
Submit
presentation
outline
X
13
26
Practice
presentation
14
28
Journal/ Product Due
Make
Presentation to Authentic Audience
X
15
30
Orient 11th graders to the Senior Project
16
32
17
34
18
36
Reflection
Reflection
Reflection
11
Student Activity Checklist
Getting Started
All journeys begin with an important first step. The first step in a Senior Project is to choose a topic that interests you, will be fun to study and is worthwhile. Refer to the chart before and begin planning next steps as outlined.
Activity
Due
Date
Date Completed
Comments
Commitment Form
Research Paper
Topic
Outline
First Draft
Revision
Final Draft
Product/Process
Mentor Identified
Plan Submitted
Parent Approval
Journal/Log
Compile Portfolio
Product/Process Completed
Oral Presentation
Speech outline submitted
Note Cards Developed
Visual Aids Developed
Portfolio Completed
Speech/Presentation practiced
Follow-up
Thank-you notes
12
Rubrics
A rubric sets criteria or standards that can be used to evaluate student learning, and gives
observable indicators of various quality levels. Some teachers say, “I know good work when I see it,” but they have a hard time putting those criteria or standards into words. The description of quality levels included in a rubric allows teachers to identify what a specific level of work looks like.
Many teachers use rubrics to assess project-related processes, products and/or student performance. Benefits of using rubrics include the following:
* Rubrics provide clear and accurate definitions of 3-5 quality levels
* Rubrics allow teachers to customize their assessment of projects by choosing the most appropriate criteria.
* Rubrics allow teachers to be accurate and consistent in their assessment of processes, products or student performance.
* Students can understand what is expected of them – before, during and after project participation. (To encourage student buy-in to criteria included in the rubrics, students can be involved in developing criteria.)
* Rubrics accept and do not penalize creativity.
To gain an understanding of the components of rubrics, take a few minutes to examine several of those included in this manual. Notice there are criteria that will be used to access a student’s work. The remaining columns describe different levels of student performance.
Web sites that have useful rubrics:
Rubric Template Visit this page for an all-purpose rubric template. Also check out the examples from
teachers who have used this template to build their own rubrics.
.www.esc20. net/etprojects/formats/webquests/summer99/northside/spu rs/ru bric
Web Publishing Rubric This rubric has ways to evaluate projects, which culminate in the publishing of a web page. www.esc20.net/etprojects/rubrics
Analytical Thinking Process of Problem Solving Rubric: This rubric assumes the difficult task of assessing the though processes behind problem solving. www.iusd.k12.ca.us/schools/ranchosanjoaquin/rsjdocs/analytic/psrubric
Oral Presentation Rubric: A short rubric for teacher and self-assessment of a student’s oral presentation. http://projects.edtech .sandi.net/morse/ocean health/ru brics/oral presentation .html
Problem Based Learning with Multimedia: A rubric for assessing multimedia use, collaboration and
project content. http://pblmm.k12.ca.us/PBLGuide/Mmrubric.htm
The Rubric Bank: Numerous rubrics in pdf format as well as some tips for choosing which one is right for you.
http://intranet.cps.k12.il.us/Assessments/IdeasandRubrics/RubricBank/rubricbank.ht ml
13
Senior Project and Project-Based Learning: Some Helpful website and Samples
Before you start re-creating all your own documents, it is great to surf websites from organizations and schools involved in senior projects and project-based learning that provide information, forms, resources, rubrics and sample projects.
www.bie.org/pbl/index
www.ehs.suhsd.net/seniorproject/seniorproject/contents
www.stemnet.nf.ca/sciencefairs/senior
www.helixcharter.net/seniorproject/rrivas/index
www.serve.org/seniorproject
www.urban.cornell.edu/youthwork/ptools.learn
www.jff.org
www.pltw.org/aindex.asp
www.bigpicture.org
www.nyfa.org/educatebydesign/edassessment
www.iearn-canada.org/guideontheside
www.gsn.org
www.iste.org/research/roadahead/pbl
www.netc.org/presentatons/itsc/2000/pbl/index
www.sreb.org/programs/hstw/publications
www.fsus.fus.edu/m2000/howto/capstone/FLHighcapstone
www.learningspace.org/instruct/lplan/library/entire http
://glef.org
http://schools.ncia.net/gorham/scholastic/seniorproject
http://pc65.frontier.osrhe.edu/hs/science/goals
14
Choosing a Senior Project Topic
A Senior Project is about doing and learning something that you want to do and learn about! This is your chance to choose a topic that will be interesting and worthwhile and will extend your knowledge of your Tech Prep area. However, making the decision may not be easy. Choose carefully, consult with your Career-Technical Instructor and remember to keep your project manageable. Here are some guidelines that might help you.
* Topics must be related to the student’s career and technical area of interest.
* The topic must be broad enough to provide adequate resources and to yield a written report of the desired length and depth of study.
* Topics must be narrow enough to be covered within the time frame of the project.
* Topics must lend themselves to a manageable and affordable product/process project.
15
Sample Senior Project Topic Selection Sheet
Student Name
TOPIC 1
Sources: 1
2
.
3
.
TOPIC 2
Sources: 1
2
.
3
.
TOPIC 3
Sources: 1
2
.
3
Instructoraprovedtopic
InstructorSignature
Sample Senior Project Topic Selection Assessment Rubric
Difficulty
1-Unacceptable
2-Marginal
3-Acceptable
4-Exemplary
Depth of
Thought
Major gaps are evident. Little or no reasoning is demonstrated.
There are major gaps in reasoning. Reasoning is somewhat apparent, but is flawed.
Reasoning is apparent, but a few minor gaps or flaws exist.
Reasoning is clear, concise and effectively demonstrated.
Feasibility
The project solution is clearly not possible within the parameters set forth by the problem.
The feasibility was in question until an explanation was requested and given. The solution may not be possible within the parameters of the problem.
While the solution is valid, it may not be easily replicated.
It is clear that the method of solution is valid and can be readily replicated.
Attention to
Detail
The project is generally characterized by superfluous or surface knowledge.
Only a few questions are answered in detail. The work generally does not attend to the underlying detail required by the problem.
Most of the questions posed by the problem are directly answered in detail.
Questions are anticipated and addressed. All measures scales and other required annotations are documented.
Creativity
This approach to the project is a detail replication of an previous design. No new ideas are demonstrated.
The approach is obviously related to a previous design, but some novelty is shown.
While the design presented my be similar in approach to others, their unique characteristics exist that make this design stand out.
The approach to the design is fresh, novel and unique.
17
Sample Senior Project Commitment Form
My Senior son/daughter and I have read through the Senior Project material. We understand that there are three components of the Senior Project, which include a research paper, 15-hour product/process project and a 8-10 minute oral presentation.
We understand that due dates and syllabi will be distributed throughout the course of the senior year and that these due dates will need to be adhered to. We also realize that this three-part Senior Project must be completed as part of the grading system for the Tech Prep program.
If we have any questions throughout the year about the Senior Project, we understand that we may contact you at (insert phone number).
ParentSignatureandDate
StudentSignatureandDate
Sample letter from Lancaster High School
Dear Seniors, Parents and Guardians:
As you may already know, one of the expectations we at Lancaster High School have of our seniors is that each student completes a Senior Project. The Senior Project was developed through our continuous improvement plan. Its purpose is not only to challenge all students and raise expectations to improve our students’ achievement, but also to provide a culminating activity for seniors to demonstrate the essential skills they all should have before leaving high school.
Senior Projects consist of three components completed throughout the year. The first component of the Senior Project is the actual research paper. This should be finished by January. It will require the student to use specific skills – research, organization, written expression and critical thinking. For most seniors, this paper will be completed in the Senior English Class where the research paper is required to pass the course. Students not enrolled in Senior English will work with a career-technical teacher. The research paper will count as 40% of the final Senior Project grade.
The second component of the Senior Project is the product/process project that is related in some way to the research topic. Each student is required to work with a mentor who can provide expertise and guide, facilitate and document the student’s progress on the project. This project is to be done outside of class time requiring a minimum of 15 documented hours usually between December and March. Students are also required to maintain a portfolio and log to record their time, efforts, thought, etc. while completing the project. Most student find this part of the project the most fun, but is also is invaluable in developing and sharpening their problem solving, communication, time management, and “hands-on” real-life skills. The project/mentorship/portfolio is 40% of the final grade.
The last part of the Senior Project is an 8-10 minute oral presentation of the paper and project hat each student will make to a broad community membership and teachers in May. This is the chance where the students will get to brag and show their pride in what they have accomplished and learned. Although many students, presenting publicly can be stressful, you will agree that oral communication skills are essential for any person to have no matter what career path is chosen. The presentation is work 20% of the complete Senior Project grade.
As you can see, the Senior Project develops and hones many important academic skills. However, many non-academic qualities are also stressed. Students will be expected to meet deadlines, so they will have to learn to plan and budget their time. They will often be required to overcome obstacles, move from their “comfort zones” and learn to cope with the healthy stresses that sometimes accompany that. The student will have to learn to make contacts with other people and find information in places not traditionally applied in school. Still, everything learned from completing the Senior Project process will only help a student become a more independent thinker, learning and problem solver. In turn, it is our hope that they will become a stronger, more independent and more successful member of society.
Each student determines the topic of his or her Senior Project. Of course, we will set some parameters. First, the topic must be related to their Career-Technical field and be appropriate for school. Secondly, the topic for the Senior Project must reflect a true learning experience or “stretch” for the student. This is, it must be something the student doesn’t know much about or hasn’t done before. Finally, there must be a connection between the topic of the
19
paper and the nature of the product. We will not require students to spend money, and we discourage unnecessary physical risk. Those factors are to be determined by the student and their parents. Therefore, the school district and its employees will not be held liable for any injury or harm.
You may ask, “How will the Senior Project be graded?” Every student who is required to complete a Senior Project will be able to earn .5 elective credit which will appear as a second semester class on the grade card. Please note: no student will be enrolled in an additional class; the “course” will appear as an independent study credit. The grade earned for the research paper will be recorded, as the third nine-week’s grade. Then the grade for the actual project will appear for the fourth nine-weeks. Finally, the final exam will reflect the oral presentation grade. The final grade for the Senior Project will then be averaged in the same manner as for any other semester class. However, if a student chooses not to complete one or more components of the Senior Project – the research paper, the project/mentorship/portfolio, and/or the presentation – an F will be recorded for the final grade. Please know that the only part of the Senior Project required for graduation is the research paper; completing it is, and for many years has been required as part of Senior English. In addition, final exam exemptions do not apply to the Senior Project presentations.
While it is expected that each student take responsibility for completing a Senior Project, no student is expected to do it alone. As was mentioned, English teachers will assist with the research paper, and the mentor will serve as a guide in completing the project. In addition each student will be assigned an advisor who is person on the LHS staff who will serve as a coach – someone the student can go to for help, someone who will help keep the student on track in meeting deadlines and finding resources, someone to be there to help students through the stressful times. We are all here to do what we can to make this experience not only a challenging one, but we also want it to be rewarding and even fun. We want all students to experience a true sense of accomplishment by having them achieve something they may have thought they’d never be able to do.
We hope that you as parents and guardians will also take an active role in helping your student and become part of our team. Kathy Rexford serves as our administrator supervising the Senior Project, and Paulette Maravich and Doug Stull are our teacher coordinators. Please so not hesitate to contact any of us with any questions, concerns, ideas or suggestions you have (6817500).
Best of luck as you all completed your senior year!
Sincerely,
The Staff of Lancaster High School
Sample Letter of Intent for Senior Project at Clay High School
Your street Address
City, State, Zip
Date
Mr./Mrs. Teacher’s Name
Clay High School
5665 Seaman Road
Oregon, Ohio 43616
Dear Mrs. Teacher
Paragraph I
Describe the general area of interest. Also explain what you already know about this area and
what experience you have in this field.
Paragraph II
This paragraph should concentrate on the specific area of your paper. What ideas do you hope to
include, and what sources do you plan to use? (Where will you find your information?)
Paragraph III
Begin this paragraph with a transitional sentence that shows the relationship between your research topic and your product/process project. Then describe your project. What will it be? Who is involved? What is the potential cost? Estimate the time you’ll need to spend on it and the possible resources you plan to use.
Paragraph IV
Explain your understanding of plagiarism and its consequences. This is your pledge that you will do
your own work.
Sincerely,
Your signature
Your Name (typed)
Assembling Research Information
Checklist for a Research Paper
* Construct a working bibliography
* Make note cards
* Organize the note cards and make outline
* Write rough draft
* Revise to a final draft
* Complete final paper
Making Note Cards
* Use a separate note card for each source
* Place the source number in the upper right corner of each card
* Use a separate note card for each main idea
* Write a label in the upper left corner of the card
* Write the page number at the bottom of each card
* Use only one side of the card
* Be sure to indicate quoted or paraphrased materials
* Be sure all notes exist within context and are accurately recorded
* Reread the notes to make sure you understand them
Organizing Note Cards
* Use label topics (from upper left corner) to organize the cards
* Discriminate between main and subordinate ideas
* Arrange cards in the order they will appear in the paper
* Write an outline that indicate the order
Writing a Rough Draft
An introductory paragraph is:
* A general statement of information
* A transition sentence using works directly related to the thesis
* A thesis statement and explanation that:
1. Contains a one-sentence statement of the topic being examined
2. States an opinion
3. Narrows the topic so it may be adequately supported in research The body of the paper contains:
* The weakest pro-augment or least important reason/example in support of the thesis
22
* Secondary strongest point
* The strongest pro-augment or most important reason/example in the support of the thesis
* Proper documentation of the sources The conclusion may:
* Restates the thesis
* Paraphrases or summarizes ideas supporting the thesis
* Take the topic one step beyond and predict the future of the problem The Bibliography
* Is an alphabetical list
* Contains all sources used in the paper Miscellaneous
* Follow either MLA or APA style
* Avoid plagiarism – presenting someone else’s words or ideas as your own.
1. Borrowing someone’s words or ideas is acceptable as long as you give appropriate credit
2. Common knowledge does not require documentation
Writing the Final Paper
* Revise and edit the evaluated rough draft
* Make a copy to share with your mentor
23
Sample Research Log
Source
Type (i.e.: book, magazine, interview, computer software, Internet, videotape):
Publication Information (including date)
How will this research be useful?
Source
Type (i.e.: book, magazine, interview, computer software, Internet, videotape):
Publication Information (including date)
How will this research be useful?
Source
Type (i.e.: book, magazine, interview, computer software, Internet, videotape):
Publication Information (including date)
How will this research be useful?
24
Sample Annotated Bibliography Rubric
1. Discussion:
Sufficient content, significant information,
Main ideas, sentence variety, proper length 40 points
2. MLA Documentation:
Alphabetical order, proper punctuation,
Proper format 25 points
3. Mechanics:
Grammar, mechanics, punctuation,
Spelling (minus 1 point per error) 25 points
25
Sample Research Paper Student Editing Sheet
Name Thesis
Statement:
I. Content/Organization
* Lenth aroriate Yes No
* All sections support thesis Yes No
* Clearly written Yes No
* Well-phrased sentences Yes No
* Good transitions Yes No
* Conclusion restates thesis Yes No
* Conclusion offers solution Yes No
II. Use of Sources
* Correct number of sources Yes No
* How many Works Cited
Works Consulted
* Paraphrased information correctly documented Yes No
* No paragraph citations Yes No
* Number of citations in paper Yes No
III. Mechanics
* Underlined book, magazine and play titles Yes No
* Quotation marks around magazine articles Yes No
* Parentheses around citations Yes No
* Periods follow parenthesis Yes No
* End punctuation mark before closing quotation marks in a direct quote
Yes No
* Corrected all spelling errors Yes No
* Eliminated fragments Yes No
* Eliminated run-on sentences Yes No
* Eliminated contractions Yes No
* Mechanical/grammatical errors Yes No
26
Sample Final Copy of Research Paper Grading Sheet Name
Format (65 points)
Title/Cover Page (5)
Thesis Page (15)
One page in length (3)
Thesis sentence in the opening paragraph (5)
Concise summary of content of paper (7)
Works Cited Page/Bibliography (25)
Correct number of sources, in appropriate category (14)
Indentation correct and alphabetically organized (6)
MLA design and punctuation followed (5)
In-Text Citation (20)
Variety of sources and documentation is used (15)
Proper format (5)
Grammar (90 points)
Paraphrases difficult concepts/language (10)
Punctuation including capitals, commas, etc. (20)
Paragraph breaks correctly (10)
Correct spelling (15)
Sentence structure – no fragments/run-ons (15)
Uses non narrative perspective (no 1st or 2n d person pronouns [I, you]) (20)
Organization (45 points)
Follows a logical progression (15)
Includes all components (10)
Required length (20)
Total Points 200 Points Earned Grade
27
Sample Research Paper Rubric
Criteria
Possible
Points
Earned
Points
1. Proper title page heading including name, teacher name, course, date, centered title and proper pagination
5
2. Outline: Formal, includes thesis
5
3. Introduction: universal, bridge, thesis Conclusion: not a repeat of the introduction
5
4. Points of proof
* Your own analysis, supported by sources
* Strong topic sentences
* Elaboration/well developed paragraphs
* Effective use of sources of proof
10
5
10
5
5. Transitions and flow, organization, focus
7
6. Mechanics: grammar, mechanics, punctuation Spelling (minus 1 point per error)
23
7. Internal documentation: form, usage, incorporation of quotes
15
8. List of works cited: MLA form
10
Points 100 Earned Points Grade
28
Sample English Grading Rubric
Rough Draft (typed)
Final Draft (typed)
Format: 20 points
* Correct title page
4
3
2
revise
* Correct outline
4
3
2
revise
* Correct subheadings and titles
4
3
2
revise
* Correct numbering
4
3
2
revise
* Correct margins
4
3
2
revise
Structure 20 points
* Appropriate title sheet
4
3
2
revise
* Thesis statement
4
3
2
revise
* Well developed body (adequate information) and clear method of development
4
3
2
revise
* Adequate conclusion
4
3
2
revise
* Adherence to outline
4
3
2
revise
Documentation 20 points
* Correct parenthetical documentation
4
3
2
revise
* Correct works cited page
4
3
2
revise
* Variety of sources
4
3
2
revise
* All required citations included (lack of Plagiarism)
Any paper where 10% or more has been plagiarized will
receive a ZERO
8
6
4
revise
Mechanics 12 points
* Spelling
4
3
2
revise
* Capitalization
4
3
2
revise
* Punctuation
4
3
2
revise
Usage 28 points
* Parallel structure
4
3
2
revise
* Sentence variety
4
3
2
revise
* Consistency in tone and voice
4
3
2
revise
* Tense consistency
4
3
2
revise
* Pronoun and antecedent agreement
4
3
2
revise
* Transitions
4
3
2
revise
* Subject and verb agreement
4
3
2
revise
Total 100 Points Earned Grade
The English Teacher could utilize this rubric (if the Tech Prep program is thus organized).
29
Sample
Evaluating the Research Paper
Format: Title Page (2) Footnotes (5)
Pagination (2)
Works Cited (11)
Grammar: Spelling (10)
Punctuation (10)
Sentence Structure (10)
Capitalization (5)
Tenses (5)
Formal Tone (2)
Organization: Beginning (thesis statement) (7)
Middle (well-developed paper) (25)
End (conclusion in keeping with the paper’s topic) (6)
Total 100
Evaluating the Product/Process
A high level of technical expertise was evident
1
2
3
4
5
The product represents an appropriate level of challenge / difficulty
1
2
3
4
5
The product clearly applies concepts covered in research content
1
2
3
4
5
The student used appropriate materials to construct the project
1
2
3
4
5
The student showed evidence of problem-solving ability, creativity and originality
The student was conscientious in making an effort
1
2
3
4
5
Total: (x 3) =
30
IT IS TIME TO CHOOSE YOUR PRODUCT/PROCESS MENTOR
A Product/Process Mentor is someone you trust who knows the subject of your project and will take a sincere interest in guiding you to successful completion of your product.
What does the mentor do?
* Provides expertise in the product area
* Gives suggestions and advice
* Supports your efforts
* Gives you feedback
Before you choose a mentor:
* Clearly define your product
* Be specific about what you expect from your mentor, including time commitments
* Be able to show how mentorship will benefit the mentor.
How do you choose a mentor?
* Brainstorm possible experts within your chosen field of study
* Talk to parents, friends and/or teachers about possibilities
* Look for someone who has the expertise and is interested in mentoring you through completion of your product/process.
Once you have a mentor – what should happen?
* Meet with your mentor
* Determine how often you will meet with your mentor
* Decide what areas of your project they will assist with ie: project details, review speech, etc.
* Ask mentor to be part of your review team
31
Sample Senior Project Mentor Agreement Form
SudentName
Project
Mentors: In order for students to complete their Senior Project, they must work with a mentor who has expertise in the area being explored. The mentor must be willing to verify the student’s efforts and time spent. Each student should spend at least 15 hours creating their product/process project and additional hours with the mentor. If you are willing to serve as this student’s mentor, please complete the form below. Thank you for your participation.
I agree to serve as a mentor for the above named student while they complete the Senior Project during 200 ___ - 200 ___ school year.
Mentor Name (print name)
Address
Phone
RelationshiptoStudent
Signature Date
32
Sample Mentor Verification Form
SudentName
Project
You have been chosen to verify this student’s efforts on their Senior Project. Since most of the time spent on the project phase of the assignment has been out of class, verification of the student’s efforts is necessary. Please answer the following questions to help us evaluate/document their 15 hours outside of class. Please keep in mind that this student’s research paper has already been evaluated. This form refers to the physical product/process project.
1. Can you verify that they spent at least 15 hours creating this project?
Yes No
Comments:
2. Have you seen this project at different stages of completion, not just the final phase?
Yes No
Comments:
3. What specific problems did this student encounter and overcome?
4. What success have you seen this student achieve?
Signature
Thank you for your support and contribution to the student. It is appreciated.
33
Choosing a Product/Process for your Senior Project
Carefully choose a product or process that you can build or produce or a community-based service you could provide. Answer these questions as you decide:
* Does the research enhance the product? Is there a clear connection between the research topic and the product / process you want to produce?
* Does the product / process represent significant amounts of time, effort and appropriate complexity? Does it go beyond what you already know how to do?
* Is the project something you will do outside of your regular class or co-curricular requirements? A product / process that you produce for a career-technical student organization will not fulfill the Senior Project requirements unless it goes substantially beyond the parameters of that product.
* Will the product involve tangible evidence of your work – either something physical that can be seen and touched, a community-based service that can be documented as beneficial or something that can be written, produced, taped and presented?
34
Sample Senior Product/Process Project Approval Form
Student Name
Product/Process Title
Product/Process Proposal
Description
Steps for Product/Process completion
Special Equipment/Facilities/Environment
Cost Analysis
Materials List
Approvals
Product/Process Mentor Signature/Date
Tech Prep Instructor Signature/Date
Parent’s Signature/Date
Student Signature/Date
Sample Product/Process Project Proposal
Here are some areas you should cover in your Product/Process Project Proposal.
* Briefly describe your idea.
* What subject(s) or topic(s) do you expect to learn about through your project?
* What skills and understanding will you develop and/or demonstrate?
* What kind of project/process do you envision?
* Name the person(s) with expertise in your area of interest who could serve as your Business/Industry Mentor. If you do not yet know of a person, write down some ideas about how you might find a mentor.
* What do you hope your project will contribute to your school and/or community?
* What do you hope to learn about yourself?
* What most excites you about your proposed project?
36
What it the student needs to modify their Senior Product/Process Project?
As sometimes happens, students may need to modify their original Senior Product/Process Project. If that happens, students should address these questions as they consider modifying their original proposal.
1. How do you wish to change your proposed project?
2. What is the purpose of these changes?
* How will these changes help you more fully explore your area of interest?
* How will these changes allow you to better demonstrate or represent your learning?
3. Have you discussed the proposed changes with others such as your parents, mentor, etc? What response have you received?
4. How will these changes affect your product/process time line?
37
Project Journal Log
Keeping a journal will help you record important questions, ideas, and discoveries as you develop your project.
What should I write in my journal?
1. Project ideas, plans, reflections, questions
2. Informal notes and bits of information
3. Accounts of conversations with your mentor or others
4. Things you are thinking about
When should I write?
1. At least three or four times per week
2. Early in the morning, end of day, or any time
3. Whenever you have a problem to solve, decision to make, etc.
4. Whenever you need to think something out
How should I write?
1. Quickly and freely, getting lots of ideas on paper
2. In any form that makes sense to you
3. In your own voice
Who will read my journal?
1. Your project advisor will review your journal
2. No one will read entries that you prefer to keep private
3. You maybe asked to share some journal entries with the panel at our presentation
Suggestions:
1. Date each entry and start each new entry on a new page
2. Write often: regular writing is what makes a useful journal
3. Whenever you can, write long enough to develop ideas fully
Sample Log Entry
Wednesday, February 1 0th
This was my first meeting with my mentor and I had no idea what to expect. I have had no experience with cars other than driving them and filling them up with gas. I was worried that I would not be able to understand what I was to do. My mentor first had me watch him as he performed an oil change on his truck, a Chevy s10. He talked through each stop, explaining what he was doing. He then ask me to change the oil on the next schedule care, a Ford Tempo. I was very apprehensive, worried that I would mess sup but he reassured me that I could do it and 3watched as I changed the oil. I was very messy and I ended up covered with grease but I was happy with how quickly I caught on. (2 hours, 30 minutes)
Monday, February 1 5th
I went to the library to find information on care maintenance so I could beginning working on my mini-manual. I do not want something very long and in-dept, just a few helpful hints about general and roadside maintenance. (1 hour, 30 minutes)
38
Sample Log Format
Name
Project
Date
Time
Activity
Comments
39
Portfolio Checklist
A portfolio is a good way to strengthen learning. It enables you to reflect on new information and to apply that knowledge in new and creative ways. A Senior Project portfolio should include all forms, references and activities associated with the project: proposals, research information, logs, journals, etc. Portfolio items should be accurate, clean, neat, is sequenced, assembled, labeled and filed in a three-ring finder (or in some other organizer) for future reference.
Recommended items:
* This Checklist
* Topic Selection Form
* Parental Approval Form
* Outline of Research Paper
* Research Paper
* Journal or Learning Log
* Product Agreement Form
* Evidence of work i.e.: pictures, materials collected, newspaper articles, etc.
* Letters of recommendation from Business/Industry Mentor
* Copy of oral presentation
* Other records of learning experience
40
Sample Project Rubric
1. 15 hours minimum (determined by portfolio) (2 points per hours)
2. Project presents an extension of the research
3. Documentation (in portfolio)
* Journal or log entries 6 points
* Photographs 4 points
* Letter of Recommendation 4 points
* Papers/Notes/Flyers/Charts 4 points
* Research Paper 4 points
* Copies of all required forms 4 points
* Reflection/S elf-Evaluation 9 points
4. Signed mentor sheet: minimum of 3 meetings
5. Care and attention to detail, quality product Also includes quality of the portfolio
Total Points 100 possible
Tips for Successful Presentations
Preparation
* Practice your presentation and know your subject well
* Prepare handouts for your audience that highlights the main points of your presentation.
Format
* Think “multimedia!”. Use a variety of formats, including video images, slides, overheads, computer graphics, 3-D models and 2-D presentation boards. Plan carefully to make sure you have all of the equipment you need.
* Use materials such as poster board, markers, colored paper and cardboard to prepare “professional style” presentation boards.
Standards
* Set high standards for your presentation; check all written work for grammar and spelling; create a clear graphic layout; and thoroughly label all products and exhibition board.
* Ask your mentor to review the materials before your presentation
The following will help you organize your presentation. Prepare your speech and practice it many times to build comfort and confidence. Dress properly, check your schedule for your presentation and make notes so that you will feel more at ease in making the presentation.
Speech Organization:
* Introduction: Introduce your topic and try to capture the audience’s attention
* Body: List and explain four key points that you want to make:
* Point 1: How did you become interested in the topic?
* Point 2: What did you learn from the research?
* Point 3: What did you learn from the project?
* Point 4: What did you learn about yourself and how has this information affected your plans for the future
* Conclusion: Connect the introduction, implications for learning and possible plans for the future.
42
Speech Preparation
1. How do your paper and project connect?
2. What emotions did you experience as you worked through the paper and project? (anger, excitement, pride, frustration)
3. What problems did you encounter? (money, time management, skill) Explain.
4. What personal growth did you gain from the paper and project? What self-knowledge did you gain? What knowledge of your topic did you gain?
5. How did the project affect your plans for the future? Explain
6. What project advice would you pass on to next year’s seniors?
Step 2: How am I going to say this? Use the answers to the questions above as references.
1. Organization – jot down your ideas on separate 3 x 5 index cards and arrange them into order that is logical and pleasing.
2. Slip blank cards into spaces where a visual aid is needed or would be appreciated.
3. Add blank cards for the introduction and conclusion.
4. If you have a project that can be displayed, jot down your ideas for that display on another card, i.e. photographs
5. Plan your introduction. The introduction should
* Grab the audience’s attention
* Make the topic thesis clear (be sure to mention both paper and project)
* Take no more than 60 seconds)
6. Plan your conclusion. A good conclusion should
* Restate topic / thesis
* Leave the audience thinking
* Take no more than 30 seconds
7. Plan the display of your project. Will it be an on-going, integral part of your speech, such as a slide show? Part of your introduction? Happen after your conclusion? Will you wear it? Sit on it? Serve up samples? (Avoid passing items during your speech – this causes too much distraction.)
8. Plan your visual aids (Two types should be included)
9. Make sure your Senior Project Advisor knows what equipment you will need.
10. Look at each idea card and fill in details, colorful anecdotes and factual information
11. Place all cards back in order and begin practicing your speech.
43
Step 3: Speech techniques to remember
1. Eye contact – This is extremely important. Practice often enough so that you rarely need to look at your cards. Remember this is a friendly audience. Your peers and your judges are pulling for you. You need the reinforcement that their encouraging expressions will give you. Look at them. True communications happens with the eyes. A speech without eye contact is only half a speech.
2. Posture – Stand Proud. You have a right to be. You have accomplished a great deal. Avoid:
* Gripping the podium – white knuckles are unattractive.
* Locking your knees – you’ve come too far to faint now.
* Twitching, wiggling, shaking
3. Voice – your voice needs to:
* Be loud enough to be heard
* Vary appropriately in pitch and tone
4. Gesture – Use your hands to help make your point. Gestures should be natural and spontaneous, not choreographed and mechanical.
5. Props (audio-visuals) – Plan and practice using any prop or audio-visual you will need during the presentation.
Step 4: Prepare for questions. Of course, there is no way to know for sure what the panel members will ask you, but you can make some educated guesses and that process will provide you with the confidence and clear-headedness to take on any question. Answer the following brainstorm questions.
1. If you were a judge listening to your speech, what would you want to know?
2. What would you like people to ask?
3. What unusual qualities does your project have that might spark interest?
4. What part of your paper might make people curious?
5. What controversial topics, if any, do you touch on?
6. What possessed you to choose this topic?
7. Who helped?
8. How did you finance it?
9. How much time did you spend?
10. Does the project double as credit for another class?
Your Senior Project Presentation should be the peak of your high school career. Your audience will be small and supportive. You will be well prepared. This is your chance to show off, to shine. ENJOY!
44
Presentation Visual Aid Ideas
You must have at least two different kinds of visual aids.
* Computer-generated Presentations
* Posters
* Photographs
* Slide Presentation
* Displays
* Video Tape
* Demonstration
* Art Form: Poetry, Music, Dance
* Role Playing
* Monologues
* Character Sketch
Sample Senior Project Presentation Rubric
Content Low (1) to High (5) 3.5 = C
20%
40%
50%
60%
70%
80%
90%
100%
Introduction – clearly stated or implied: good attention getter
1
2
2.5
3
3.5
4
4.5
5
Main Idea clear and well supported by detail
1
2
2.5
3
3.5
4
4.5
5
Organization – logical flow of ideas, not choppy or random
1
2
2.5
3
3.5
4
4.5
5
Application of learning – “I never knew...” “I learned....” “I plan to use this knowledge....”
1
2
2.5
3
3.5
4
4.5
5
Conclusion – reviews ideas, has memorable statement
1
2
2.5
3
3.5
4
4.5
5
Balanced content between paper and project
1
2
2.5
3
3.5
4
4.5
5
Language Usage – transition, word choice, style
1
2
2.5
3
3.5
4
4.5
5
Challenge – Did the student reach beyond comfort zone?
1
2
2.5
3
3.5
4
4.5
5
Total
Delivery Low (1) to High (5) 3.5 = C
20%
40%
50%
60%
70%
80%
90%
100%
Eye Contact – looks at audience, not just notes
1
2
2.5
3
3.5
4
4.5
5
Poise – appears comfortable
1
2
2.5
3
3.5
4
4.5
5
Posture
1
2
2.5
3
3.5
4
4.5
5
Rate and volume of speaking
1
2
2.5
3
3.5
4
4.5
5
Gestures
1
2
2.5
3
3.5
4
4.5
5
Avoids vocalization pauses
– uh, um, like, and, you
1
2
2.5
3
3.5
4
4.5
5
Appearances - appropriate dress, neatness
1
2
2.5
3
3.5
4
4.5
5
Use of audio and/or visual aides
1
2
2.5
3
3.5
4
4.5
5
Total
46
Sample Presentation Rubric
Points Possible
Use of 3 x 5 index cards
5
Appropriate dress
10
Length of time (8 to 10 minutes)
10
Command of the topic
5
Content: Technically current
10
Organization:
* Materials ready (5)
* Clear and interesting (5)
* Introduction – focus (5)
* Sense of closure (5)
20
Two visual aids – Quality related to topic (must be 2 kinds)
20
Pace: adequate for note taking and interest
5
Voice: Clarity, volume, energy, no “ums”
5
Eye Contact
5
Posture/poise
5
Total Points Possible 100
Sample Evaluating a Presentation
Communication
Verbal
* Volume/tone
1
2
3
4
5
* Grammar
1
2
3
4
5
* Speech pattern/pace
1
2
3
4
5
Non-Verbal
* Appearance
1
2
3
4
5
* Eye Contact
1
2
3
4
5
* Poise/Delivery
1
2
3
4
5
Total: (x 1) =
Content
Subject properly introduced
1
2
3
4
5
Key points made
1
2
3
4
5
Accurate information
1
2
3
4
5
Logical flow
1
2
3
4
5
Visual Aids appropriate
1
2
3
4
5
Visual Aids attractive
1
2
3
4
5
Total: (x 2) =
48
Sample Presentation Rubric
Students will lose 20 points for each day that the presentation is late
Points
Content
* Student demonstrates expertise in topic area
* Student responds appropriately to questions
25
Organization of Presentation
* Students uses introduction and conclusion
* Student shows evidence of preparation
* Student shows originality and creativity
* Student utilizes visual aids (2 types)
25
Communication Skills
* Student speaks clearly
* Student uses proper grammar
* Student chooses words appropriately
* Student shows evidence of practice
* Student uses proper body language, posture, etc.
20
Appearance
* Student dresses appropriately for presentation
* Visual aids are neat, accurate and organized
10
Overall Effectiveness
10
Time Management
* Student completes the presentation in the allotted time of 8 to
10 minutes (excluding questions)
10
49
Guidelines for Judges
1. Be aware that these students have no past – all you know about them will come from their portfolios and their oral presentation only. If the student has a disability or special need, you will be advised of that.
2. Consider the risks these students have taken. Many have stepped out of their comfort zones to try new things – they have taken risks in order to learn something worthwhile to them. The project is successful if the student has learned. The more learning that has taken place, the more successful the project is.
3. All judges need to preview the student’s work by examining the student
portfolio for their designated project. The portfolios are ready for viewing through the Career-Technical Instructor. Remember that the portfolios have already been evaluated. You are examining them in order to become familiar with the student’s work on the Senior Project.
4. The portfolio should include, but not be limited to:
* Project Overview
* A variety of approval and/or verification forms
* Mentor Information
* Research Paper
* Project logs
* Evidence of project (pictures, written materials, etc)
* Project Self-Evaluation Form
We really hope this turns out to be a rewarding experience for all of you who have volunteered to help. Thanks again for your support.
50
Panelist Assessment Sheet
As you review the student’s work, and as you listen to the oral presentation, think about the work in relation to four broad categories: Purpose, Findings, Reflections and Communication.
Performance standards are listed below for each of the categories. Please use the rating scale provided.
I. Purpose: Has the student clearly stated the purpose for doing this project, particularly in terms of its personal and social relevance?
* The student described how they became interested in the topic/project.
1 2 3 4
Exceeds Meets Approaches Needs Attention
* The student clarified the personal meaning of the project.
1 2 3 4
Exceeds Meets Approaches Needs Attention
II. Findings: Has the student arrived at a coherent set of findings/outcomes?
* The student made clear what they learned about the field of inquiry.
1 2 3 4
Exceeds Meets Approaches Needs Attention
* The student described clearly how their work connects to the work of other people in the
field. 1 2 3 4
Exceeds Meets Approaches Needs Attention
* The student identified new questions, ideas or activities to pursue as a result of this work.
1 2 3 4
Exceeds Meets Approaches Needs Attention
51
III. Reflections: Has the student told the story of their project and analyzed what they learned?
* The student provided a coherent narrative of the project, including obstacles and surprises.
1 2 3 4
Exceeds Meets Approaches Needs Attention
* The student reported what they learned about this kind of major independent project.
1 2 3 4
Exceeds Meets Approaches Needs Attention
* The student assessed their own strengths and shortcomings in carrying our the project.
1 2 3 4
Exceeds Meets Approaches Needs Attention
IV. Communications: Has the student presented the work in a way that conveys its meaning and observes the conventions of written and oral communications?
* The student prepared the materials in a clear, concise and well-organized manner.
1 2 3 4
Exceeds Meets Approaches Needs Attention
* The student gave well-organized and thoughtful oral presentation and conveyed a sense of mastery of the material and awareness of audience.
1 2 3 4
Exceeds Meets Approaches Needs Attention
* The student used supporting materials (e.g., visuals) that enhance the presentation
1 2 3 4
Exceeds Meets Approaches Needs Attention
52
V. Comments: This space is for more general responses to the work. The questions below
will guide the responses after the student presents. If you wish, you may use this space to take notes for the discussion.
* What impresses you the most about this project? What do you see as the major strengths of the work?
* What would strengthen the project?
* How does the work of the project connect to other work in the field?
* What would you suggest to the student who wants to pursue this work further?
53
Sample Senior Project Rubric
Name Topic
Evaluation Components
Excellent
Good
Satisfactory
Unacceptable
Research Points 50
4
3
2
1
* Chooses a clear and manageable topic to research
* Uses a variety of sources
* Selects appropriate sources
* Includes current data
* Cites sources correctly using MLA or APA style
* Incorporates up-to-date information that is technically correct
Written Report Points 100
(Mechanics) 50 rough/50 final
4
3
2
1
* Correction of rough draft according to suggestions
* Contains good written expression
* Contains appropriate introduction and summary
* Contain correct grammar and word usage
* Contains correct spelling
Written Report Points 100
(Technical) 50 rough/50 final
4
3
2
1
* Includes adequate information to cover topic
* Provides sufficient background for reader
* Paper is comprehensive and complete
* Examines issues/ideas beyond basic concepts – puts thoughts into your own words
Timeline Points 50
4
3
2
1
* Submits research materials on time
* Submits rough draft on time
* Submits corrected final draft on time
* Submits completed product/process on time
* Completes presentation on time
Product/Product Points 150
4
3
2
1
* Exhibits research beyond knowledge gained through career-technical area
* Demonstrates creativity and originality
* Shows evidence of planning and organizing
* Uses correct terminology
* Worked closely with mentor
* Kept accurate log/journal
Oral Presentation Points 100
4
3
2
1
* Demonstrates expertise on topic
* Speaks clearly
* Uses proper grammar
* Exhibits good eye contact
* Dresses appropriately for presentation
* Shows evidence of practice and preparation for presentation
* Uses appropriate visual aids to enhance presentation
Total Points 550 Student Received Grade Earned
54
Sample Senior Project Evaluation Rubric
Points Possible
Evaluation Areas
1
2
2.5
3
3.5
4
4.5
5
X=
Total Points
5
Completion of Topic Selection Sheet
X1
5
Senior Project Commitment Form
X1
5
Project/Process Project Approval Form
X1
5
Senior Project Checklist
X1
50
Research Paper
X10
10
Mentor Identification Sheet
X2
20
Senior Project Overview
X4
50
Product/Process Project
X10
50
Student Log/Journal
X10
10
Degree of challenge for the student
X2
50
Oral Presentation
X10
50
Portfolio: Physical Evidence of Learning
X10
25
Senior Project Self-Evaluation
X5
335
Total Points
55
Sample Senior Project Self-Evaluation
1. In at least 25 words describe your project.
2. Answer the following:
A. Estimated total hours spend on project
B. Estimated number of steps to your project
Step 1
Step 2
Step 3
Step 4
C. Date Started
D. Date Completed
E. Materials Used:
F. List every kind of resource (including people)
3. Circle the appropriate response
A. Did community resources help Yes No N/A
B. Have you asked for advice/comments
about your project from other people? Yes No N/A
C. Did you utilize task analysis and time
management principles? Yes No N/A
D. Have you ever done a project like this before? Yes No N/A
E. Would you recommend your project area for
future senior project Yes No N/A
Explain:
56
4. Answer in 25 words or less:
A. Would you feel threatened to show your project to an expert for evaluation?
B. List three things that you now know after completing your senior project.
C. List all personal satisfaction you gained from this project experience.
D. Describe what risks you took in completing this project. Consider not only a physical risk, but also an emotional or intellectual challenge.
E. How do you feel your project would compare to other similar projects? Explain
57
READ THIS FIRST
VENTILATOR USER GUIDE MODEL P200S
SMOKE EJECTOR
USER GUIDE
This manual covers the description, cautions, operation, and maintenance of the Super Vac P164S
Please take the time to read this manual before operating the ventilator.
Save this guide for future reference.
CONTENTS
1. BREAKAGE OR DAMAGE DURING SHIPMENT
2. GENERAL DESCRIPTION AND SPECIFICATIONS
3. PRODUCT SAFETY PRECAUTIONS
4. BEFORE OPERATING VENTILATOR
5. OPERATION
6. MAINTENANCE AND REPAIR
7. TROUBLESHOOTING
8. EXTENSION CORDS
9. EXTENSION DUCTS
10. WARRANTY
11. CONTACT INFORMATION
12. PARTS LIST
13. EXPLODED VIEW
1 P200S User Guide Revised 3-2016
SMOKE EJECTOR
1. Breakage or Damage During Shipment
The transportation company is fully responsible for all shipping damage and will resolve problems promptly if you handle it correctly. Please read these instructions carefully.
Examine the contents of all shipping cases. If you find any damage, call your transportation agent at once and have them make a description on the freight or express bill describing the damage and the number of pieces. Then write us and we will send you the original bill of lading. Get a claim blank from the express or truck company. Fill the claim form out. Attach the claim blank to the original bill of lading together with a copy of our invoice. Attach a memo on which you show the value of the damaged goods. Mail or hand these papers to your local transportation agent. They will process your claim with reasonable promptness.
Please note, we cannot and will not enter claims for damages. If we filed claim here, it would be sent to your local freight agent for verification and investigation. This time can be saved by your filing the claim directly. Every consignee is on the ground floor and in contact with the local agent who inspects the damaged goods, and thus, each claim can be given individual attention.
Since our goods are packed to comply with the regulations of all railroad, truck, and express companies, we cannot allow deduction from any invoice because of any damage, however, be sure to file your claim promptly. Our goods are sold F.O.B. factory. We take receipt from the transportation company certifying that the goods were delivered to them in good order, and our responsibility ceases.
It is seldom that any breakage or damage occurs in any of our shipments, and in no case will the customer be out any expense if they follow the above instructions.
Be sure to keep all damaged goods subject to examination of the truck or express company inspector, who may call on you some time later. These damaged goods, of course, will belong to them, and they will inform you what to do with them. If you dispose of these damaged goods, your claim may not be paid.
2 P200S User Guide Revised 3-2016
SMOKE EJECTOR
2. General Description and Specifications
Super Vac ventilators are specifically designed to provide a convenient and portable source of fresh air. Refer to decal affixed to ventilator to determine proper operating voltage requirements. Super Vacelectric ventilators are single speed, fresh air ventilators, for use in extreme conditions.
Specifications
Specifications are subject to change without notification. Refer to model decal affixed to ventilator to determine actual
specifications.
TEAO = Totally Enclosed Air Over motor
This ventilator is designed for high powered air movement. Super Vac’s patented propeller produces a greater intake of air and eliminates turbulence at the tips. The high velocity screw action sweeps air back over the motor, enabling it to run cooler. The patented 4-Blade propeller is cast from aluminum alloy. Each blade is statically balanced for high efficiency and long life.
All Super Vac ventilators are designed to accept optional spiral duct and adapters. Placing adapter and ducts on the output side provides positive pressure ventilation. Place adapter and duct on the input side for negative pressure ventilation.
The square housing design gives the unit added strength and stability. Handles on each corner makes the ventilator easy to hang on windows, doorways or ladders. Multiple units may be stacked using optionally available accessories.
3 P200S User Guide Revised 3-2016
SMOKE EJECTOR
3. Product Safety Precautions
* Do not use the ventilator for purposes other than its’ intended use.
* Do not disassemble or modify the ventilator in any way. Doing so may lead to mechanical failure or personal injury.
* Do not wear loose clothing that could become entangled in the operating ventilator.
* Do not operate ventilator without proper inlet or outlet guards in place. If guards are missing or damaged, contact factory for replacements.
* Do not place fingers or other foreign objects through inlet or outlet guards. Should a foreign object enter the ventilator, immediately disconnect power source. Be certain all mechanical motion has stopped before removing foreign object.
* Do not use with any power voltage other than the indicated voltage.
* Do not place or operate ventilator on unsteady tables, slanted surfaces or other unstable surfaces.
* Never handle energized power cords with wet hands.
* Never let extension cords or plug connection lay in water. Locate the ventilator so that it cannot fall into water inadvertently.
* Never use this ventilator in any explosive environment.
4. Before Operating Ventilator
* Always operate this ventilator from a proper AC power source supplied through a ground fault interrupter receptacle. Test receptacle for ground fault before plugging in the ventilator power cord.
* A complete inspection of the ventilator should be made prior to operation.
* Never use power cords with frayed, cut, or brittle insulation.
* Inspect power cord plug for sound connections to the ground fault interrupter circuit.
* Check all threaded fasteners for proper tightness.
* Inspect inlet and outlet guards for damage or missing pieces. Replace damaged or missing guards before operating ventilator.
4 P200S User Guide Revised 3-2016
SMOKE EJECTOR
5. Operation
Check ventilator inlet and outlet guards for obstructions. Check and remove from immediate surrounding area any objects that could be drawn into the ventilator intake.
Place ventilator in desired location before connecting to power supply.
Extend and place any flexible duct as required. Use caution not to direct toxic fumes into confined spaces with the ventilator.
Please refer to the Super Vac Training Manual for Smoke Ventilation for comprehensive instruction on effective ventilation practices.
Insert power cord plug into 120 VAC power receptacle to begin ventilation.
To stop ventilation remove power cord plug from AC power receptacle.
6. Maintenance and Repair
Maintenance should only be performed by experienced and trained personnel.
Be sure the ventilator is disconnected from the power source before performing any maintenance or repair.
The motor is of heavy duty ball bearing construction. The unit is completely sealed and requires no lubrication for the life of the motor.
The ventilator blade should be specially checked for build-up of material or dirt which may cause an imbalance. Excessive imbalance can lead to accelerated wear on motor bearings and vibration. Remove ventilator guards. Clean blade and inside housing with mild detergent and water. Do not use abrasives, sharp instruments or caustic solvents that may scratch or damage the aluminum alloy blade.
5 P200S User Guide Revised 3-2016
SMOKE EJECTOR
7. Troubleshooting
CAUTION: Motors are equipped with thermal overload protection. When internal motor temperature exceeds design limits, the motor will automatically shut off. Motor will automatically restart when it cools sufficiently after overheating. Remove power cord plug from AC receptacle before examining or working on a motor which has stopped because of overheating. Thermal overload protection sensitivity increases with frequency. Take preventative steps to correct cause of overheating to assure long motor life. Most overheating problems are associated with the use of portable power generators as the ventilator power supply.
Common causes of motor overheating:
* Power generator not providing constant voltage. Ventilators are equipped with AC motors designed to operate on power that falls between ±3% of design specifications.
* Using too small gauge of extension cord.
* Exceeding maximum length extension cord.
* Insufficient unobstructed distance on exhaust side of ventilator. Please refer to the Super Vac Training Manual for Smoke Ventilation for recommended ventilator placement.
* Insufficient wattage generator. When running multiple ventilators on one generator, plug ventilators in one at a time. Wait until first ventilator is running at full speed before connecting next ventilator to generator. Do not exceed generator capacity.
* Using accessory ducting of too small a diameter or excessive length.
* Incorrect amperage circuit breaker for circuit load
* Using reeled or coiled extension cords.
6 P200S User Guide Revised 3-2016
SMOKE EJECTOR
8. Extension Cords
Do not use reeled or coiled cable. Reeled or coiled cable can produce a counter EMF (electromotive force). The counter EMF can reduce the current carrying capability of an extension cord. This counter EMF will result in a voltage drop within the motor instigating a thermal overload condition. Power supply must handle wattage requirements. When conditions require the use of reeled cable, unreel all cable and place on the ground in very loose overlapping non-uniform coils.
Units*
Watts **
Minimum Wire Gauge
Maximum Cable Length
1
2000
14
150 ft
2
4000
12
250 ft
3
6000
10
350 ft
* Multiple units powered by one extension cord
** Minimum running wattage for multiple units. Startup amerage draw is much higher.
9. Extension Ducts
Spiral duct may be used on the intake or exhaust side of a ventilator.
Exhaust Side*
Maximum 40 ft
Intake Side*
Maximum 20 ft
“L” Duct*
Exhaust Side Only
* Adapter required, optional, not included with ventilator.
7P200S User Guide Revised 3-2016
SMOKE EJECTOR
10. Warranty
Super Vacuum Manufacturing Company warrants that the equipment is free from defects in materials and workmanship when used and operated for a period of five years. The responsibility of Super Vacuum Manufacturing Company under this limited warranty is limited to the repair and replacement of any parts which are found defective and which are returned to Super Vacuum Manufacturing Company at 3842 Redman Dr, Fort Collins, CO 80524 with transportation charges prepaid (C.O.D. shipments will not be accepted).
Prior to returning defective parts to SUPER VACUUM MANUFACTURING COMPANY, the original purchaser shall make a claim in writing to SUPER VACUUM MANUFACTURING COMPANY at the above address indicating the model number and type of defect. No parts or equipment will be received by SUPER VACUUM MANUFACTURING COMPANY for repair or replacement under this warranty without specific written authority from it in advance.
Any parts damaged by improper installation, overloading, abuse or accident of any type or cause are not covered by this warranty.
All equipment manufactured by us is pre-run and tested before leaving our plant, and is shipped in good working order and condition. We therefore extend to the original purchasers the following Limited Warranty for the period of five years from the original date of purchase:
1. This warranty does not apply to defects caused by accident, misuse, neglect, or wear and tear, nor can we be held responsible for incidental and consequential expense and loss, nor does this warranty apply to equipment where alterations have been executed without our knowledge or consent. These conditions are readily discernible when the equipment is returned to us for inspection.
2. On all component parts not manufactured by SUPER VACUUM MANUFACTURING COMPANY, their warranty is to the extent that the manufacturer of such component warrants them to SUPER VACUUM MANUFACTURING COMPANY, if at all. Look in your local business phone directory for the nearest repair station for the brand of parts you have or write to us for the address.
3. If equipment received is found to have been damaged in transit, a claim should be made against the carrier within three days, as we assume no responsibility for such damage.
4. Any service other than our Authorized Service voids this warranty.
5. This warranty is in lieu of and is intended to exclude all other warranties, express or implied, oral or written, including any warranties of MERCHANTABILITY or FITNESS for a particular purpose.
8 P200S User Guide Revised 3-2016
SMOKE EJECTOR
11. Contact Information
For parts or service information, contact:
Super Vacuum Manufacturing Company, Inc.
3842 Redman Dr.
Fort Collins, CO 80524
Phone: 800-525-5224
970-297-7100
Fax: 970-297-7099
Email : info@supervac.com
Internet: www.supervac.com
9 P200S User Guide Revised 3-2016
SMOKE EJECTOR
10 P200S User Guide Revised 3-2016
SMOKE EJECTOR
13. Exploded View
9 P200S User Guide Revised 3-2016
IMPROVING SAFETY NET AND WORK SUPPORT
Proposal 14: Smarter, Better, Faster: The Potential for
Predictive Analytics and Rapid-Cycle Evaluation to
Improve Program Development and Outcomes
Scott Cody
Mathematica Policy Research
Andrew Asher
Mathematica Policy Research
Introduction
Public administrators have always been interested in identifying cost-effective strategies for managing their programs. As government agencies invest in data warehouses and business intelligence capabilities, it becomes feasible to employ analytic techniques used more-commonly in the private sector. Predictive analytics and rapid-cycle evaluation are analytical approaches that are used to do more than describe the current status of programs: in both the public and private sectors, these approaches provide decision makers with guidance on what to do next.
Predictive analytics refers to a broad range of methods used to anticipate an outcome. For many types of government programs, predictive analytics can be used to anticipate how individuals will respond to interventions, including new services, targeted prompts to participants, and even automated actions by transactional systems. With information from predictive analytics, administrators can identify who is likely to benefit from an intervention and find ways to formulate better interventions. Predictive analytics can also be embedded in agency operational systems to guide real-time decision making. For instance, predictive analytics could be embedded in intake and eligibility determination systems, prompting frontline workers to review suspect client applications more-closely to determine whether income or assets may be understated or deductions underclaimed.
Rapid-cycle evaluation, another decision-support approach, uses evaluation research methods to quickly determine whether an intervention is effective, and enables program administrators to continuously improve their programs by experimenting with different interventions. Like predictive analytics, rapid-cycle evaluation leverages the data available in administrative records. It can be used to assess large program changes, such as providing clients with a new set of services, as well as small program changes, such as rewording letters that encourage clients to take some action. This type of formative evaluation can be contrasted with the summative program evaluations familiar to many in the policy community. Summative program evaluations often assess whether a program has an impact by comparing program participants with nonparticipants. Rapid-cycle evaluation uses similar techniques, but does not examine the overall impact of the program. Instead, it assesses the impacts of changes to the program by comparing some program participants (with the change) to other program participants (without the change).1 For example, rapid-cycle evaluation can determine whether an employment training program can use text message prompts to encourage more clients to successfully complete program activities. In this way, rapid-cycle evaluation can identify incremental changes that make the program more effective for its clients, increasing the likelihood that a subsequent summative evaluation would identify large impacts relative to individuals not in the program.
The Hamilton Project • Brookings 1
IMPROVING SAFETY NET AND WORK SUPPORT
Proposal 14: Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-Cycle Evaluation to Improve Program Development and Outcomes
We believe that these techniques can be used to help government programs—including social service programs serving low-income individuals—to improve program services while efficiently allocating limited resources. We believe that the use of predictive modeling and rapid-cycle evaluation— both individually and together—holds significant promise to improve programs in an increasingly fast-paced policy and political environment.
We propose that social service agencies take two actions. First, agency departments with planning and oversight responsibilities should encourage the staff of individual programs to conduct a thorough needs assessment. This assessment should identify where predictive analytics and rapid-cycle evaluation can be used to improve service delivery and program management. The assessment should also evaluate whether the benefits of adopting these tools outweigh the costs, resulting in a recommendation of whether and how these tools should be deployed. Second, federal agencies should take broad steps to promote the use of predictive analytics and rapid-cycle evaluation across multiple programs. These steps include investments in data quality and data linkage, as well as measures to support and promote innovation among agency staff.
The Challenge
Our proposal is based on the simple assumption that government programs could do better. This seems self-evident: despite decades of antipoverty efforts, the reality is that unemployment and underemployment, low food security, high poverty rates, and related problems persist. Rigorous evaluations of federal social programs show that many programs have little or even no impact on program participants.
In fact, even those programs held up as examples of proven, evidence-based programs demonstrate that government programs could do better. For example, the Coalition for Evidence-Based Policy identifies top-tier social programs with rigorous evidence of effectiveness, such as the Nurse-Family Partnership, Nevada’s Reemployment and Eligibility Assessment Program, the Transitional Care Model, and other programs (Coalition of Evidence-Based Policy 2012). Multiple randomized controlled trials on each of these programs show positive impacts on client outcomes. But even this positive evidence suggests these programs could be more effective. A systematic review of research on the Nurse Family Partnership program concludes that there is evidence of a positive impact on only seven of the twenty-five measures of child maltreatment, and on only five of the fifty-nine measures of child development and school readiness (U.S.
Department of Health and Human Services n.d.). The Nevada Reemployment and Eligibility Assessment Program increased employment among participants, but only modestly: 52 percent of program participants were employed, which is higher—but not substantially higher—than the rate in the control group, in which 48 percent of participants were employed (Michaelides et al. 2012). In short, even programs highlighted as success stories have room for improvement. They could benefit more clients and they could have a larger impact on the clients they benefit.
The administrators of these and other programs are constantly seeking ways to improve outcomes. Some administrators seek to match clients with the right services. But without the right analytic tools, these administrators cannot determine if their services are targeted as effectively as possible. Other administrators seek to test new procedures aimed at improving program services. But again, without the right analytic tools, these administrators may get biased results, leading them to implement ineffective changes or to dismiss effective ones. In the end, progress toward program improvement is slow, and programs end up spending resources inefficiently and leaving participants underserved.
A New Approach
Because predictive analytics and rapid-cycle evaluation have the potential to improve program effectiveness, we believe that social service agencies should conduct thorough needs assessments to identify, program by program, where these tools can be used. The needs assessments should examine the quality of existing program data to determine whether they are robust enough for use in predictive analytics and rapid-cycle evaluation. The assessments should also examine whether and how programs can deploy predicted outcomes operationally in a way that improves program performance. Furthermore, they should assess whether and to what extent experiments can be conducted to test changes in program operations. In addition to conducting program-level needs assessments, agencies should also take steps to promote the use of these tools broadly across multiple programs. These steps could include investments to improve data systems, improve data governance, and promote a willingness among program staff to test program innovations.
To inform the needs assessment, this section begins with an explanation of how predictive analytics and rapid-cycle evaluation can be deployed in the administration of public programs. These tools are not commonly used at this time. Where possible, we provide real-world examples of the application of these tools. We supplement these examples with a discussion of potential applications. Agencies should
2 Policies to Address Poverty in America
Scott Cody, Andrew Asher
consider these real-world and potential applications when conducting their needs assessments.
PREDICTIVE ANALYTICS
At the individual level, predictive analytics leverages the fact that key outcomes and outputs for program clients are often correlated with the client’s prior behaviors, circumstances, and characteristics, as well as those of the client’s family, associates, service providers, and surroundings. By examining these correlations, predictive analytics methods can be used to rank program clients based on the likelihood that an outcome, whether positive or negative, will occur.
For example, an analysis predicting which participants of a job training program are likely to find employment might leverage existing information about the clients’ education levels and their attendance at job training sessions. The model might tap these factors and other information to rank participants on the likelihood that they will find employment. Using these rankings, program administrators could decide, based on their goals and resource constraints, the exact sub-population that they want to target with their additional services. Depending on their program’s objectives, administrators might focus on individuals most likely to find employment, or might target additional services to individuals less likely to find employment.2
Below we describe two key uses of predictive analytics for policymakers: (1) identifying program participants at risk of an adverse event and (2) predicting the optimal service path for an individual. We then discuss deploying predictive analytics to impact decision making.
IDENTIFYING PROGRAM PARTICIPANTS AT RISK OF AN ADVERSE EVENT
Program administrators can use predictive analytics to identify clients who are at risk of an adverse outcome such as unemployment, fraud, unnecessary hospitalization, mortality, or recidivism. Knowing which participants are most likely to experience an adverse outcome, program staff can provide targeted interventions to reduce the likelihood that such outcomes will occur.
Reducing readmission rates for certain patients discharged from the hospital provides an example of how predictive analytics can be used effectively. Reasons for unplanned readmissions can include clinical and social factors, such as patients’ timely access to quality primary health-care services, their underlying conditions, whether they are homeless, and whether they lack social support and other factors that affect their ability to recuperate at home without incident (Peikes et al. 2012–13). If Medicaid programs could anticipate which
patients are likely to be readmitted, they could intervene to address some of the factors contributing to the higher likelihood of a repeat visit. This would enable the patients to avoid another hospitalization while the Medicaid program would avoid paying for expensive hospital care.
Researchers at New York University have developed such a predictive model to identify a combination of characteristics and circumstances that indicate an elevated risk that a New York Medicaid beneficiary discharged from a hospital will return within one year (Raven 2009; Raven et al. 2009). New York City Health and Hospitals Corporation is using this model within its operational systems to screen admitted patients and identify interventions for those most likely to be readmitted for a preventable reason (Evans 2011).3
A similar approach could be used to prevent recipients of public assistance benefits from letting their eligibility lapse. Assistance programs such as the federal Supplemental Nutrition Assistance Program (SNAP), formerly known as the Food Stamp Program, require beneficiaries to demonstrate eligibility through a periodic recertification process. If clients do not complete the recertification process, their benefits are terminated. Clients often do not reapply for the program until they realize their benefits have been terminated. This creates two problems. First, clients who are eligible for assistance forgo benefits for one or two months until they reapply. Second, the program must bear the costs of processing a new application— which is more expensive than recertification. State agencies that administer the federal SNAP program could use predictive analytics to identify clients at risk of such churning. What would be required, beyond the tested and validated analytics themselves, is that the models be built directly into the case maintenance systems. Identifying these at-risk clients prior to the redetermination would enable program administrators to direct targeted, intensive communication efforts to these clients to prevent churning and help the clients maintain benefits while saving program funds.
Other potential areas for using predictive analytics include enforcement and fraud detection applications. For example, some child support enforcement agencies are developing predictive models to identify noncustodial parents who will not make their child support payments. This information can be used to triage enforcement efforts, making sure fewer resources are devoted to collection efforts against those who will ultimately pay without enforcement and identifying those who are likely to pay in response to more-aggressive efforts.
In addition, predictive analytics can be used to identify provider, client, vendor, and billing entity fraud patterns in health-care and social service programs. In SNAP, for instance, geographic patterns of electronic benefits transfer
The Hamilton Project • Brookings 3
IMPROVING SAFETY NET AND WORK SUPPORT
Proposal 14: Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-Cycle Evaluation to Improve Program Development and Outcomes
redemption and historical investigative data can be used to predict which program clients and retailers may be engaged in benefit trafficking (exchanging SNAP benefits for cash at a discount).
PREDICTING OPTIMAL SERVICE PATHS
Many government programs have different approaches to working with clients to achieve the same outcome. For example, there are multiple approaches to preventing recidivism among juvenile offenders, encouraging preventative health care, and boosting the parenting skills of new mothers. These paths may differ in the services involved or the time at which the services are offered. Under the right circumstances, predictive analytics can be used to determine which approaches are most likely to benefit which clients. Administrators can then identify the optimal service path for a client among the available options.
Consider a caseworker trying to find the right jobs program for a nineteen-year-old unemployed man with no high school diploma. This caseworker can enroll the individual in a low-intensity résumé support and job search program, a more-intensive program teaching specific manufacturing skills, or even a very intensive apprenticeship program. Each path has a different cost, and possibly a different outcome, for this individual. The caseworker’s job is to match the program to the individual’s background and interests. Combining this information—which is readily known at intake—with a prediction, based on which programs are associated with success for similar clients, could yield a better match between client and services, increasing the likelihood that the client will find employment and reducing the likelihood of wasting funds on ineffective training. Many agencies are interested in developing optimal service path predictions, yet in practice few exist. We believe there is an opportunity for optimal service path modeling to benefit the clients of public programs.
RAPID-CYCLE EVALUATION
Rapid-cycle evaluation, another tool that supports decision-making, is increasingly used in public programs with readily available administrative data and the ability to analyze those data in a rapid, cost-effective manner. This type of evaluation uses rigorous experimentation to test changes in agency operations.4 To determine any impacts from the changes, administrators can compare client outputs and outcomes with those for other clients who are included in the evaluation but continue to receive regular services. The evidence from these tests can be more reliable than other sources, such as feedback from staff, complaints from selected clients, or anecdotes from other agencies.
To better understand how rapid-cycle evaluations can be used to test changes, it is useful to consider the three defining terms:
1.
Rapid. The “rapid” means that the impact of the intervention will be identified quickly. To facilitate rapid identification of results, the outcomes of interest should be observable in administrative data. This eliminates the time-consuming process of collecting new data. Additionally, any impacts of the intervention should be observable within a short time frame. For example, it would not be possible to rapidly assess whether an intervention delivered to ninth-grade students leads more of those students to graduate from high school.
2. Cycle. The “cycle” refers to the iterative nature of the tests. Rapid-cycle evaluations can support a formative, continuous improvement model in which an intervention is tested, the results are examined, the intervention is modified if needed, and the modified intervention is tested again or a new intervention is tested.
3. Evaluation. The “evaluation” refers to the use of rigorous research techniques that generate confidence that observed changes in outcomes are due to the intervention and not to other factors (such as differences between the group that received the intervention and the group that did not).
This approach has been used by businesses for years to continuously improve the match between customers and services. For example, Capital One claims it runs more than 30,000 experiments each year to help identify the techniques that cause customers to sign up for new credit cards as well as techniques that encourage customers to pay Capital One back (Davenport and Harris 2007). The company experiments with changes in interest rates, promotional incentives, and even the color of the envelopes used in customer mailings.
Rapid experiments are used in the public sector as well to test a variety of program interventions, including changes in staff procedures, the services provided to clients or customers, and when and where those services are provided. Rapid-cycle evaluations of experiments can assess whether the interventions meet goals such as improving (1) the agency’s ability to serve more clients, (2) the quality of information agencies get from clients, (3) client outcomes, and (4) agency efficiency. It is sometimes possible to test numerous variations of program services simultaneously. Box 14-1 shows how experimentation and rapid-cycle evaluation can fit into overall program operations by presenting applications used by New York City Human Resources Administration.
In some cases it may not be feasible to collect the necessary outcome data. For example, target outcomes may occur too far in the future to be examined in a rapid experiment (e.g., the eventual graduation of ninth graders). It may still be feasible, however, to employ rapid-cycle evaluation by looking at impacts on intermediate outcomes (such as class
4 Policies to Address Poverty in America
Scott Cody, Andrew Asher
BOX 14-1.
New York City Human Resources Administration
Agencies such as the New York City Human Resources Administration (HRA) have deployed rapid-cycle evaluation to improve program services. HRA recently tested new administrative procedures to increase the establishment of child support for children receiving cash assistance (Dinan 2013). Since court appearances are assumed to be a deterrent to establishing child support, the agency tested new procedures that would avoid a court appearance for the noncustodial parent. The agency’s goal was to increase the percentage of these children with established child support orders, reduce the time needed to establish orders, and increase the proportion of noncustodial parents that comply with their established orders. HRA staff developed a simple random assignment process for determining which cases were eligible for the new procedures, and trained frontline workers to administer the pilot. The analysis showed the new procedures were unsuccessful. The rate of child support order establishment for the treatment group (57.3 percent) was essentially the same as the rate for the control group (56.5 percent). Moreover, it took longer to establish those child support orders established through the new procedures.
HRA also used experimentation to test streamlined procedures for investigating Medicaid eligibility fraud (Weinberg 2013). These streamlined procedures were designed to reduce the number of steps needed to investigate potentially fraudulent Medicaid enrollees. They used a four-month random assignment experiment to evaluate the impact of these new procedures. Although the new procedures reduced the time spent investigating cases by 12 percent, fraud investigations conducted with the streamlined procedures were less likely to be successful. Fraud was established for 44 percent of cases investigated through streamlined procedures, compared with 61 percent of those investigated through status quo procedures.
In both of these experiments HRA’s new procedures proved unable to achieve the desired impact. But in each case HRA clearly and quickly established that the procedures were unsuccessful without having to implement these new procedures program-wide.
attendance and grades), as well as program outputs (such as the amount and quality of services provided). Such rapid experiments and rapid-cycle evaluations can often still help improve program services.
Rapid-cycle evaluation also could be used to measure real responses to potential policy changes. For example, programs like SNAP and Temporary Assistance for Needy Families (TANF) have numerous eligibility criteria and other regulations that are often debated by policymakers. These include deduction amounts, certification period lengths, benefit formulas, reporting thresholds for income changes, and even the required number of hours for participation in work programs. When the changes to these regulations are discussed, policymakers debate whether these changes will lead to higher or lower participation rates, and whether they will lead to longer program dependence or encourage employment. Rapid-cycle evaluation has the potential to generate rigorous, reliable information that can take the guesswork out of these policy debates. Regulatory changes can be tested to identify—and quantify—clients’ behavioral response to these changes. This information can ensure that regulatory changes better meet policymakers’ goals.
The greatest benefit of rapid-cycle evaluations to the agencies is the rigorous nature of the evaluation, which can replace other, nonexperimental techniques for assessing programmatic changes. For example, programs may pilot new procedures with all staff in a single location. In such cases, it is often not possible to know whether differences in outcomes are caused by the new procedures or simply by the unique circumstances of that location. This can lead program administrators to the false conclusion that a new procedure has promise, only to learn there is no benefit once it is implemented agency wide. Alternatively, it can lead them to reject a procedure that actually has promise.
COMBINING PREDICTIVE ANALYTICS AND RAPID-CYCLE EVALUATIONS
Predictive analytics and rapid-cycle evaluations can be combined to help program administrators build better interventions. Predictive analytics allow administrators to anticipate which individuals are most (and least) likely to benefit from a program. These predictions can help program administrators guide the formulation and scope of the interventions, and determine the group or subgroups to which they would apply. By creating targeted experiments, program
The Hamilton Project • Brookings 5
IMPROVING SAFETY NET AND WORK SUPPORT
Proposal 14: Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-Cycle Evaluation to Improve Program Development and Outcomes
FIGURE 14-1.
Combining Predictive Analytics and Rapid-Cycle Evaluation: A Simplified Example
Predictive analytics are used to sort program participants by likelihood of benefit from the base program.
A new intervention is developed and tested on participants unlikely to benefit from the base program.
administrators can identify a series of effective, tailored interventions to maximize their ability to make an impact.5
Consider a program administrator seeking to test new approaches for reaching hard-to-serve clients. Initial predictive models could identify which of the program’s current clients are least likely to benefit from the program, and could separate the clients into treatment and control groups. New interventions (or potentially multiple variations of the same intervention) could be tested rapidly and, if effective, could be incorporated into service delivery. After the new interventions operate for sufficient time, the entire cycle could be repeated (see figure 14-1) or applied to a different subgroup of program participants.
The new procedures can also be tested on individuals who are likely to benefit from the program. Such tests can help administrators determine whether new approaches would yield even greater improvements for individuals positioned to benefit the most from program services. Some administrators may view targeting those most likely to benefit as the most effective way to achieve gains for participants and improve the program’s overall success.
For illustration, consider the hospital readmission prediction model mentioned earlier. A predictive model could be used to identify at-risk patients who are most likely to return to a
hospital within one year. If program administrators want to test two different interventions for these at-risk patients, they could randomly assign the at-risk patients to one of three groups—one for each of the two interventions plus a control group—that receive the hospital’s normal discharge planning and other services. The team would then monitor hospital admission rates for three months and assess whether the new interventions cause a significantly lower readmission rate. Any successful intervention could be integrated into program operations; the unsuccessful ones could be discarded.
If multiple interventions prove successful, program administrators could implement all of them or choose one based on cost and potential sustainability. The predictive model could be rerun and follow-up analysis could suggest new, tailored interventions for the remaining at-risk population. These interventions could be formulated and tested as in the previous cycle, evaluated, and either discarded or included in program operations.6
THE POLICY PROPOSAL
We propose that federal social service agencies take two actions. First, agency departments with planning and oversight responsibilities should encourage the staff of individual programs to conduct thorough needs assessment. This assessment should identify where predictive analytics
6 Policies to Address Poverty in America
Scott Cody, Andrew Asher
and rapid-cycle evaluation can be used to improve service delivery and program management.
For predictive analytics, program administrators should assess:
* Whether predictions about specific client and program outcomes could be employed to target program services;
* Whether the program’s current administrative data contain accurate, valid, and reliable measures of those outcomes— as well as valid and reliable measures of information that could predict those outcomes—to support predictive modeling; and
* The magnitude of systems enhancement efforts required to enable frontline workers to use the results of predictive models in real-time when they interact with clients.
For rapid-cycle evaluation, program administrators should assess:
* Whether program changes under consideration would benefit from precise, causally-valid impact estimates generated through rapid-cycle evaluation. The assessment can rely not only on program staff, but also on funders and outside experts to identify program features that they believe would be beneficial to test but were not sure should be implemented permanently without assessment;
* Whether program operations can be modified to facilitate experimentation of these program changes;
* What types of investments in data and systems would be required to deploy predictive analytics and rapid-cycle evaluation together as an integrated strategy;
* What types of programmatic waivers and other policy changes would be needed to facilitate predictive analytics and rapid-cycle evaluation; and
* Whether it would be beneficial to use predictive analytics to subset the program population, and to test program changes on different types of individuals (e.g., those most likely to benefit from current services).
The answers to each of these questions will vary by program. The assessment also should evaluate whether the benefits of adopting these tools outweigh the associated costs. In the end, the assessment should contain a recommendation of whether and how these tools should be deployed.
The second step agencies should take is to promote the adoption of predictive analytics and rapid-cycle evaluation
more broadly across programs. We recommend that agencies take the following steps:
1. Help programs make individual-level data available for analytics. Individual-level data provide the best foundation for predictive analytics and rapid-cycle evaluation. These data can be obtained through internal operational systems maintained by the program, or through integrated data systems that combine administrative data across programs. A broad investment in data can facilitate predictive analytics and rapid-cycle evaluation and can promote the use of these tools across multiple programs.
Federal agencies can help more programs benefit from these analytic tools by facilitating improvements to individual-level administrative data, and ensuring that those data are available for analytic purposes. Some agencies are already taking the lead in this respect. For instance, the Department of Education provided grants to promote the development of statewide longitudinal education data, and the Centers for Medicaid and Medicare Services funded data warehouses to help states manage all aspects of their Medicaid and Children’s Health Insurance Programs. Agencies can also use their expertise to help programs identify the key measures to track for prediction and evaluation on an ongoing basis.
2. Improve data governance and facilitate data sharing. Although high-quality data are necessary, agencies also need strong data governance policies that establish accountability for data quality and that define the terms for how and where data are used (see Digital Services Advisory Group 2012). In addition, as part of data governance efforts, agencies should work to actively support efforts to link data across programs, which involves often-challenging technical and legal considerations. That said, linked data can provide a more comprehensive understanding of the services received and circumstances faced by clients, and provide more-accurate predictions and a more-complete understanding of the impact of rapid-cycle experiments.
3. Encourage analytic decision making. The use of predictive analytics and rapid-cycle evaluation requires an organizational commitment to testing program improvements. This means agency staff must develop program innovations—but be willing to abandon those innovations if they prove unsuccessful. For many program staff, this is a change in mindset from a focus on assessment of their program (and compliance with funder guidelines needed to properly evaluate their programs) to a focus on how to improve the programs and empower program administrators. Federal agencies can help foster innovation by providing performance
The Hamilton Project • Brookings 7
IMPROVING SAFETY NET AND WORK SUPPORT
Proposal 14: Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-Cycle Evaluation to Improve Program Development and Outcomes
based funding opportunities for program improvements.
Predictive analytics and rapid-cycle evaluation can be effective in part because they empower frontline staff to determine the services that best meet their clients’ needs. However, the lessons learned for individual programs can be valuable to other programs serving these same populations. Agencies can further the effectiveness of these tools by ensuring successful efforts are highlighted, and their lessons broadly disseminated.
We believe that by taking these steps, federal agencies can help promote the use of these analytic tools at the federal, state, and local levels.
COSTS AND BENEFITS
Predictive analytics and rapid-cycle evaluation have a number of benefits. In particular, greater use of these tools would increase program effectiveness by reducing wasteful and inefficient spending. Even where the proposal results in an increase in direct outlays in one phase of a program’s intervention, these outlays may generate net savings. Moreover, these analytical innovations would allow programs to fulfill their missions more effectively by better targeting their intended beneficiaries and helping them continually identify and implement cost-effective interventions.
That said, adopting these tools can require significant investments at a time when government budgets are under pressure. Developing the data and technology infrastructure necessary to deploy these analytical capabilities—if they are not already present—is expensive, as are, to a lesser extent, the resources needed to perform these analytics. For example, what may be considered to be the gold standard for data infrastructure—a full-featured, enterprise-wide data warehouse that integrates data across programs and is refreshed on a weekly basis—can cost several million dollars to build, and millions more annually to staff with dedicated maintenance and analytical personnel. Less-expensive data systems, such as purpose-specific analytical datamarts within existing warehouses or standalone databases focused on specific questions, may be more feasible and more appropriate in some cases.
As part of their needs assessments, agencies should assess the costs of any changes needed to deploy predictive analytics and rapid-cycle evaluation. In addition to data infrastructure costs, agencies should examine the costs associated with training staff, as well as the costs of altering program operations to incorporate predictive analytics and to implement rapid-cycle experiments and evaluations.
Agencies should compare these projected costs with the potential benefits obtained from these tools. In many cases, the benefits will include long-term savings in program administrative costs because the tools render the program more efficient. Other important benefits, however, such as improvements to the quality, availability, and access to services, should also be considered.
In the end, we believe that the benefits of predictive analytics and rapid-cycle evaluation will be substantial for many programs. We believe that this potential may warrant significant investment in these tools for many programs. For virtually all programs, however, we believe that this potential warrants the costs of conducting a needs assessment.
Questions and Concerns
In this section we examine some of the factors that could affect an agency’s ability to adopt predictive analytics and rapid-cycle evaluation by posing and addressing a series of key questions and concerns.
What data resources are necessary for the use of these tools?
Programs with advanced information systems that contain individual-level administrative data are better suited to deploy predictive analytics and rapid-cycle evaluation with minimal investment. For example, in many states sophisticated cross-program data warehouses have been developed to support a wide array of Medicaid and social service program monitoring needs. These systems are rapidly updated and could be easily used for both predictive analytics and rapid-cycle evaluation. For other programs, administrative data obtained from transactional systems can provide an important source of information. These programs would require additional investment to create analytics-ready data repositories through the extraction, transformation, and storage of the data.
It is important to note that predictive analytics require historic observations of key outcomes. This means that programs developing new systems may not be able to perform predictive analytics until the system has captured enough history. Similarly, programs extracting data from transactional systems would need to extract a sufficiently large volume of historical data in order for predictive analytics to be effective.
Who would implement these tools?
The program managers and staff in agencies directly responsible for program delivery would implement these tools. To be successful, the implementation of these tools requires a division of labor. Program administration staff— both program operators and those working in support of
8 Policies to Address Poverty in America
Scott Cody, Andrew Asher
them at federal and state agencies—need to determine which interventions are worth implementing and figure out how to do so. These program experts need to be supported by analytical specialists who are charged with designing the predictive analytics and assessing the results of the experiments through the rapid-cycle evaluations. Such a partnership allows this approach to become feasible and avoid burdening those with the pressing responsibility of running programs.
Can predictive analytics be wrong?
Yes. Predictive models detect patterns, but not every individual will follow that pattern. This can lead to incorrect predictions. Administrators can take several steps to minimize problems stemming from inaccuracies in predictive models.
First, predictive models should be subjected to extensive validation. For adverse event situations, such as hospital readmissions or fraud, models should be deployed historically so that their ability to predict known outcomes can be assessed. Through repeated retrospective testing, use and learning, the models can be improved, often to the point where they can be used prospectively.
Second, it is important to ensure model predictions are followed up by human judgment. Whether it is identifying clients who should receive a caseworker visit or those who may be defrauding the government, predictive analytics should be used to prioritize cases; staff should make the final determination. Similarly, even after optimal service paths are predicted, clients should still have a say in the services they receive.
Conclusion
As integrated data repositories become common in government agencies, program administrators have become comfortable using these data to monitor their programs. Now administrators are poised to expand the use of analytics to better decide what to do next. Predictive analytics and rapid-cycle evaluation, if used individually but especially if used together, can help agencies provide services where they are needed and develop more-effective approaches for improving program outcomes.
The Hamilton Project • Brookings 9
IMPROVING SAFETY NET AND WORK SUPPORT
Proposal 14: Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-Cycle Evaluation to Improve Program Development and Outcomes
Authors
Andrew Asher
Senior Fellow, Health Policy Division, Mathematica Policy Research
Andrew Asher is a senior fellow at Mathematica and with extensive experience in health care management and data analytics—the use of data to help agencies and health care payers strengthen the integrity of their programs, realize cost savings, and improve program outcomes. He is one of the leaders of Mathematica’s data analytics initiative and his expertise includes program integrity, health policy, health program development, big data, and developing strategies to help agencies improve their performance. He is nationally recognized for developing and implementing innovative data-driven approaches to help health care purchasers and insurers reduce fraud and abuse and manage their costs effectively. He rejoined Mathematica in 2013 after serving as vice president and director of government solutions at OptumInsight from 2007 to 2013 and in various leadership roles with the state of Illinois. From 2001 to 2007, he served as the manager of the Fraud Science Team within the Illinois Department of Healthcare and Family Services’ Office of Inspector General, a group whose innovative fraud detection and prevention work was featured in Government Accountability Office and Center for Medicare and Medicaid Services best practice reviews. Asher was a health researcher and analyst at Mathematica from 1989 to 1994 and holds a master’s in public policy from the University of Michigan.
Scott Cody
Vice President and Director of Human Services Research, Mathematica Policy Research
Scott Cody is a vice president and director of human services research in Mathematica’s Cambridge, Massachusetts office. He is a national expert in the Supplemental Nutrition Assistance Program (SNAP). He led the first major in-depth study of state efforts to modernize enrollment procedures for programs like SNAP, Temporary Assistance to Needy Families (TANF), and Medicaid. His research also has examined the determinants of SNAP participation. He led an impact evaluation of efforts to increase program participation among the elderly, and multiple studies to identify the circumstances that lead individuals to enter and exit SNAP. Cody played a central role in adapting Mathematica’s SNAP policy simulation model to reflect major changes in program policy. In addition to research on SNAP, Cody is the deputy director of the What Works Clearinghouse (WWC). The WWC is an initiative of the U.S. Department of Education’s Institute of Education Sciences. The WWC assesses the quality of thousands of studies of education curricula, practices, and policies, and then summarizes the findings of well-designed studies in reports for educators, policymakers, and the general public. Cody holds a master’s in public policy from the Johns Hopkins University.
10 Policies to Address Poverty in America
Scott Cody, Andrew Asher
Endnotes
1. We recognize that there are other definitions of rapid-cycle evaluation that will not utilize a comparison group. In this paper, we focus on the assessment of rapid experiments using comparison groups.
2. It is important to note that the performance of predictive analytics can vary. A number of considerations, including the extent to which strong predictors are available and the quality of the data, can affect performance. The strength of the underlying predictive models should be assessed before deploying predictive analytics in high-stakes situations.
3. The estimated equation generated by this model is used as part of an automated algorithm to find at-risk patients for intervention among those newly admitted to the hospital. In one early pilot, inpatient readmissions declined by 45 percent (Raven 2009; Raven et al. 2009).
4. Rigorous experimental techniques include randomized controlled trials and orthogonal research designs, and rigorous quasi-experimental designs include regression discontinuity research designs. These designs can be used to determine whether an intervention caused an outcome. Like a clinical drug trial, randomized controlled trials create randomly formed treatment and control groups, each receiving a different intervention. Orthogonal research designs use a similar approach but test variation in the components of an intervention. Regression discontinuity studies create a treatment group with individuals above (or below) a certain eligibility threshold (with individuals on the other side of the threshold forming the control group), and use analysis techniques to control for the eligibility score in the assessment of the program.
5. While we are not aware of specific, published examples of the use of these methods together, we believe the integration of these approaches is powerful and compelling, as the discussion that follows demonstrates.
6. It is noteworthy that examining multiple groups requires additional sample observations if the same level of precision is to be obtained. Generalizing beyond one hospital or program likewise requires additional sample observations.
The Hamilton Project • Brookings 11
IMPROVING SAFETY NET AND WORK SUPPORT
Proposal 14: Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-Cycle Evaluation to Improve Program Development and Outcomes
References
Coalition of Evidence-Based Policy. 2012. “Social Programs that
Work.” Coalition for Evidence-Based Policy, Washington,
DC.
Davenport, Thomas, and Jeanne Harris. 2007. Competing on
Analytics: The New Science of Winning. Boston: Harvard
Business School Publishing.
Digital Services Advisory Group, Federal Chief Information
Officers Council, and Federal Web Managers Council.
2012. “Digital Service Governance Recommendations.”
http://www.whitehouse.gov/digitalgov/digital-services
governance-recommendations.
Dinan, Kinsey. 2013. “Local Agency Lessons On Implementing
Random Assignment: An Example from NYC’s Child
Support Program.” Presentation to the Association for
Public Policy Analysis and Management, Washington, DC.
Evans, Melanie. 2011. “Healthcare’s ‘Money ball’: Predictive
Modeling Being Tested in Data-Driven Effort to Strike Out
Hospital Readmissions.” Modern Healthcare 41 (41): 28–30.
Michaelides, Marios, Eileen Poe-Yamagata, Jacob Benus, and
Dharmendra Tirumalasetti. 2012. “Impact of the
Reemployment and Eligibility Assessment (REA) Initiative
in Nevada.” Impaq International LLC, Columbia, MD.
Peikes, Deborah, Rebecca Sweetland Lester, Boyd Gilman, and
Randall Brown. 2012–13. “The Effects of Transitional
Care Models on Re-Admissions: A Review of the Current
Evidence.” GENERATIONS—Journal of the American
Society on Aging 36 (4, Winter): 44–55.
Raven, Maria C. 2009. “Improving Care and Cutting Costs for
High-Risk Medicaid Patients.” New York State Health
Foundation, New York, NY.
Raven, Maria C., John C. Billings, Lewis R. Goldfrank, Eric D.
Manheimer, and Marc N. Gourevitch. 2009. “Medicaid
Patients at High Risk for Frequent Hospital Admission:
Real-Time Identification and Remediable Risks.” Journal of
Urban Health 86 (2): 230–41.
U.S. Department of Health and Human Services (HHS).
n.d. “Home Visiting Evidence of Effectiveness.” U.S.
Department of Health and Human Services, Washington,
DC.
Weinberg, Stephen. 2013. “Using RCTs to Test Procedural
Changes: Evidence From Medicaid Fraud Investigations.”
Presentation to the Association for Public Policy Analysis
and Management, Washington, DC.
12 Policies to Address Poverty in America
Using Computer-Assisted Qualitative Data Analysis Software
(CAQDAS) to Develop a Grounded Theory Project
J. D. Bringer
Sports Council for Wales
Sophia Gardens
Cardiff, CF11 9SW
United Kingdom
Tel: +44 (0)29 2033 8303
Fax: +44 (0)29 2030 0617
e-mail: Joy.Bringer@scw.co.uk
L. H. Johnston
Sheffield Hallam University
Collegiate Hall, Sheffield, S10 2BP
United Kingdom
Tel: +44 (0)114 225 5863
Fax: +44 (0)114 225 4341
e-mail: L.Johnston@shu.ac.uk
C. H. Brackenridge
School of Sport and Education
Brunel University
Uxbridge, UB8 3PH
United Kingdom
Tel: +44 (0)1895 267160
Fax: +44 (0)1895 269769
e.mail: celia.brackenridge@brunel.ac.uk
Authors’ Bios
This paper is based on Joy D. Bringer’s doctoral research into coaches’ perceptions of sexual relationships in sport, completed in 2002, at the University of Gloucestershire, UK. She is a Chartered Psychologist (British Psychological Society) and is accredited by the British Association of Sport and Exercise Sciences (BASES) for Sport Psychology support. She is now a Senior Sport Psychologist at the Sports Council for Wales in the United Kingdom where she works with national level coaches and athletes. Her research interests surround aspects of coach and athlete welfare.
RECENT PUBLICATIONS
Brackenridge, C.H., Bringer, J.D. and Bishopp, D. (2005) ‘Managing cases of abuse in sport’, Child Abuse Review 14(4):259-274.
Bringer, J. D., Brackenridge, C. H., & Johnston, L. H. (2001). The name of the game: A review of sexual exploitation of females in sport. Current Women's Health Reports, 1(3), 225-231.
Bringer, J.D., Johnston, L.H. & Brackenridge, C.H. (2004). Maximising transparency in a doctoral thesis: The complexities of writing about the use of QSR*NVIVO within a grounded theory study. Qualitative Research, 4 (2) 247-265.
Dr Lynne Johnston is a Reader at the Centre for Sport and Exercise Science, Sheffield Hallam University. She is a Chartered Psychologist (British Psychological Society) and has worked in research training and consultancy since 1997. She has extensive experience of
2
teaching research methodology and methods to postgraduate students and is one of a small group of UK based trainers who have trained with QSR International (software developers) in the UK and in Australia. Lynne is a member of the QSR International Conference Organising Committee (http://www.qual-strategies.org/
) and has presented internationally on training issues associated with the use of QSR software. She has a particular interest in the use of QSR software within Doctoral Research and has supervised and examined several PhDs which have utilised QSR software.
RECENT PUBLICATIONS/ PRESENTATIONS:
Johnston, L. H. (in press). Technical and methodological learning curves: Reflections on the use of QSR NVivo in doctoral research. International Journal of Social Research Methodology.
Richards, L, Johnston, L.H. Di Gregorio, S. Bazeley, P. (2004). Evaluating the performance of NVivo users: What demonstrates skill with NVivo and why is it important to do so?. Invited expert panel, facilitated by Davidson J. at Strategies in Qualitative Research, 1-3rd September, University of Durham, UK.
Coupland, C. & Johnston, L. H. (2004). Using QSR NVivo in Phenomenological Research: The experience of recovering from Psychosis through Groupwork. Paper Presented at Strategies in Qualitative Research, 1-3rd September, University of Durham, UK.
Professor Celia Brackenridge is Chair in Sport Sciences (Youth Sport) in the School of Sport and Education at Brunel University, West London. She is responsible for developing research and consultancy in the social sciences applied to sport, with a special emphasis on youth sport, child protection and welfare issues. Celia has carried out major studies of women and leadership in leisure management and sports coaching and has been investigating sexual abuse and child protection in
3
sport for the past 16 years. She serves on the Research Committee of the National Organisation for the Treatment of Abusers and also chairs the Research Task Force of the NSPCC/Sport England Child Protection in Sport Unit.
Recent Publications:
Brackenridge, C. H. (2001) Spoilsports: Understanding and Preventing Sexual Exploitation in Sport. London: Routledge.
Brackenridge, C.H., Pawlaczek, Z., Bringer, J.D., Cockburn, C., Nutt, G., Pitchford, A. and Russell, K. (2005) ‘Measuring the impact of child protection through Activation States’, Sport, Education and Society 10(2): 239-256.
Fasting, K. and Brackenridge, C.H. (2005) ‘The grooming process in sport: Case studies of sexual harassment and abuse’, Auto/Biography 13(1): 33-52.
Fasting, K., Brackenridge, C., and Sundgot-Borgen, J. (2004) ‘Prevalence of sexual harassment among Norwegian female elite athletes in relation to sport type’, International Review for the Sociology of Sport 39: 373-386.
4
Abstract
The promise of theory and model development makes grounded theory an attractive methodology to follow. However, it has been argued that many researchers fall short and only provide a detailed description of the research area or simply a quantitative content analysis rather than an explanatory model. This article illustrates how the researchers used a computer-assisted qualitative software program (CAQDAS) as a tool for moving beyond thick description of swimming coaches’ perceptions of sexual relationships in sport to an explanatory model grounded in the data. Grounded theory is an iterative process whereby the researchers move between data collection and analysis, writing memos, coding, and creating models. The nonlinear design of the selected CAQDAS program, NVIVO, facilitates such iterative approaches. Although the examples provided in this project focus on NVIVO, the concepts presented here could be applied to the use of other CAQDAS programs. Examples are provided of how the grounded theory techniques of open coding, writing memos, axial coding, and creating models were conducted within the program.
KEY WORDS: grounded theory, qualitative, QSR NVIVO, child protection, sport
5
Using Computer-Assisted Qualitative Data Analysis Software
(CAQDAS) to Develop a Grounded Theory Project
The constructivist revision of grounded theory (Strauss and Corbin 1990, 1998; Charmaz 2000) is a process designed for systematic theoretical development where a theory relevant to the specific topic and population of study does not exist. Theory, in grounded theory, is not intended as an all-encompassing grand theory, rather it is a methodology to assist in the development of an explanatory model grounded in empirical data (Glaser and Strauss 1967).
There was much debate in the 1990s about how to achieve grounded theory (Glaser 1992; Becker 1993; Charmaz 1995; Annells 1996; Melia 1996; Wilson and Hutchinson 1996), including comments by Glaser (1992) vehemently disagreeing with Strauss and Corbin (1990). Glaser and Strauss’s (1967) first text explaining grounded theory provided guidelines for conducting qualitative research, a task that had previously been generally passed down orally from supervisor to student (Charmaz 1995). However, this first text has been criticized as being too abstract (Charmaz 2000). Strauss and Corbin’s (1990) introductory text for novice users of grounded theory was written with this critique in mind. Glaser’s (1992) greatest objection to Strauss and Corbin’s (1990) book was that it was too prescriptive and thus forced theory to emerge (see Fielding and Lee [1998] for a summary of Glaser’s objections). This concern echoed a broader concern in the research community that the introduction of computer analysis programs (for both quantitative and qualitative data) allowed users to do complex analyses without understanding the principles of the analysis (Lee and Fielding 1991; Kelle 1995; Richards and Richards 1995; Seidel and Kelle 1995; L. Richards 1998; Weitzman 2000; Bringer, Johnston, and Brackenridge 2004).
6
Clearly stating methodological assumptions and evaluative criteria helps address concerns about the potential misuse of computer-assisted qualitative software programs (CAQDAS). Specific criteria for evaluating research based on grounded theory can be divided into two elements: the research process and the research product. Issues of credibility, plausibility, and trustworthiness are evident in both evaluations. The criteria presented by Strauss and Corbin (1990, 1998) for evaluating process are: detail about sampling, events leading to emerging categories, identification of major categories, relationships between categories, theoretical sampling, negative cases, and the emergence of the core category. In terms of product, concepts should be generated from the data, be systematically related to create categories, and have conceptual depth.
I followed the constructivist revision of grounded theory because I was studying an area that had received little research attention: the attitudes of coaches toward sexual relationships in sport. I used CAQDAS to help with the organizational aspects of managing qualitative data. I was also aware of the possibility that the tools in CAQDAS might help me move beyond description to theorizing, while meeting the evaluative criteria of grounded theory. Another benefit of consistent use of CAQDAS is that it doubles as an audit trail. (Qualitative researchers kept audit trails long before CAQDAS, but because the software include features designed to assist with record keeping, it is easier to consistently maintain this process.) Excerpts from the program can be included in written reports to demonstrate rigor and allow others to more accurately evaluate the research (Bringer, Johnston, and Brackenridge 2004).
Interestingly, in a review of the QUAL-software e-mail list, MacMillan and Koenig (2004) concluded that CAQDAS was predominantly used by those claiming to follow the
7
grounded theory methodology. Fielding and Lee (1998) also reviewed a set of studies using CAQDAS and found that 30% claimed to be using grounded theory. However, the remaining 70% referred to a range of other methodologies, suggesting that CAQDAS use is not dominated by grounded theorists. It is imperative to understand that although a program may facilitate the user’s development of theory, this does not suggest that the program can guarantee theory development, nor coherence with a particular methodology. Program users are ultimately responsible for analyzing the data and developing theory (Fielding and Lee 1998; MacMillan and Koenig 2004).
My aim was to find a program that could help with organization and offer flexibility that would complement the analysis methods within grounded theory. A number of computer programs are available as shareware1 and commercially that may have been suitable (see Lewins and Silver [2004] for a comparison of programs). I selected QSR NVIVO (2000) (hereafter referred to as NVIVO) because it met my requirements.
Several books discuss the use of NVIVO (Bazeley and Richards 2000; Bazeley 2002; Gibbs 2002; Morse and Richards 2002; L. Richards 2005), but not specifically within a grounded theory methodology. Therefore, the main purpose of this article is to explain how and why CAQDAS can be used to facilitate a grounded theory analysis. Although the examples focus specifically on the tools I used in NVIVO, the principles could be applied to other CAQDAS programs. Details about the methods for data collection and the overall rationale for the research topic are explained in detail elsewhere (Bringer 2002; Bringer Brackenridge, and Johnston 2002a).
Nineteen swimming coaches participated in either an elite, national, or county-level focus group to examine their constructions of appropriateness about coach/swimmer sexual
8
relationships. Coaches discussed the appropriateness of relationships presented in vignettes. After completing the initial analysis, the emergent themes were further explored in individual unstructured interviews with three purposively selected coaches. This additional data collection and analysis revealed that the themes about appropriateness relate to the broader issue of coaches’ attempts to resolve perceived role conflict and ambiguity that has arisen from increased awareness of child protection.
The Iterative Process of Grounded Theory and NVIVO
The software program NVIVO facilitated the iterative process of grounded theory in a number of ways. I started my project in NVIVO before collecting any data by recording initial thoughts in memos and in my research journal. As the data were integrated into the project, memos were attached to focus group documents and coding categories. The program allows for open coding, axial coding (making links between codes), hyperlinks to nontextual data such as audio clips or photographs, coding according to demographic information, and the exploring of ideas visually with a modeler. Rather than requiring that all of the data be collected before analysis can start, the program has been intentionally designed to encourage researchers to analyze data as they are collected. The program facilitates and allows text searches, ideas to be linked, data coded and searched, and models to be drawn while always being able to instantly access the original data behind the concepts. However, this does not imply that the computer is doing the analysis. The researcher still must ask the questions, interpret the data, decide what to code, and use the computer program to maximize efficiency in these processes.
9
Data Documents in Grounded Theory and NVIVO
The initial inductive nature of grounded theory usually leads researchers to select qualitative sources of data. Grounded theory does not, however, preclude the use of quantitative data such as survey data that can be used at the later stages of a project to support or further explore the initial analysis (Glaser and Strauss 1967). In this project, focus groups and interviews were the main sources of data and were saved as individual documents in NVIVO. Newspaper reports and policy documents also influenced the analysis and were saved in NVIVO as document memos. Classifying the media reports as memos allowed them to be easily distinguished from the coaches’ comments in interviews and focus groups, which were the main focus of my analysis. Separating out the media reports made it easier to examine coding specific to what participants said about abuse in swimming, as compared to journalists who wrote about abuse more generally. It is possible to achieve the same result by creating sets of documents (e.g., a set of media reports, a set of interviews).
An advantage of using a program like NVIVO is the ability to transform the way data are viewed (from static to dynamic) in a way that makes relationships between categories more visible by using text formatting and hyperlinks to other documents and categories (Weaver and Atkinson 1994). Internal annotations and external files can be attached to any piece of text in a document to record referential information that may be important for context but which would interrupt the flow if placed as text in a transcript. Internal annotations are brief and conceptually similar to footnotes. For example, when a coach referred to a celebrity, an internal annotation was used to note why he was significant to the conversation. External files can be attached in a similar manner, but are intended for larger files. These might include pictures, audio files, video clips, or web pages. Audio clips from
10
the focus groups were inserted as external files when it was unclear what was being said. Often, after rereading the transcript several times and listening to the clip, the section was deciphered. Textual contextual information can also be linked to the document text in the form of a NVIVO document or memo. Unlike internal annotations and external files, linked documents and memos can be coded directly.2 Compound interlinking documents were created using color, formatting, and linking annotations, memos, documents, and nodes. This helped me think about the conceptual links and associations in my data, a key element in grounded theory (Weaver and Atkinson 1994; Fielding and Lee 1998).
Strauss and Corbin assert the value of researchers using their experiences to inform the development of a grounded theory project. The research journal is an important tool for reflection on the research process as it includes the reciprocal influence of the research on the researcher. This is especially important when researching sensitive topics like sexual abuse (Brackenridge 1999). Writing the research journal within NVIVO had many benefits over a hardbound copy, including being able to code thoughts within the journal and creating live links to specific documents, nodes, and media reports (Bringer, Johnston, and Brackenridge 2004).
Prior Knowledge in a Grounded Theory Project
One of the main tenets of grounded theory is that coding should emerge from the data. That is, any concept in the analysis should be supported from the data, rather than from preconceived models, theories, or hypotheses. Depending on which thread of grounded theory is followed, the use of prior knowledge has different applications. Strauss and Corbin (1990, 1998) encouraged the use of discipline-based knowledge as long as the knowledge fits the data and is not inappropriately applied to it. Glaser specified that: “The analyst’s
11
assumptions, experiences and knowledge are not necessarily bad in and of themselves. They are helpful in developing alertness or sensitivity to what is going on in the observational-interview data, but they are not the subject’s perspective” (1992:49).
He further advised: “These sources of theoretical sensitivity must be put aside. Indeed, the analyst should just not know as he approaches the data, so he does not even have to waste time correcting his preconceptions” (Glaser 1992:50). Whereas Glaser accused Strauss and Corbin (1990) of forcing data, Charmaz acknowledged that with the constructivist revision of grounded theory, she “generates data by investigating aspects of life that the research participant takes for granted” (Charmaz 1995:36). This is the perspective that I followed.
Glaser (1992) and Strauss and Corbin (1990, 1998) agreed that writing the literature review too early in a grounded theory study may unduly influence the data collection and analysis and may be a waste of time if the data lead the analyst in a different direction. However, Strauss and Corbin (1990, 1998) are more pragmatic with their admission that it may be impossible to delay the literature review completely. As was the case in this study, researchers are often required to present research proposals to funding boards, supervisors, and ethics committees before any data are collected. Strauss and Corbin (1990, 1998) advised that, although some literature review is necessary, an exhaustive literature review might be inhibiting. Ultimately, it is a balance between reading enough to be aware of and understand possible factors that could influence the area of study, while still remaining open minded to what the participants have to say.
I imported the literature review notes into NVIVO to facilitate access to them. A search could be conducted for all of the articles containing a particular method, such as focus
12
groups. The “reading notes” document was automatically coded into different nodes by using a coding function called “Code by Section.” The reading notes here contained two headers. The top-level header referred to the cited reference and the remaining reading subheads (e.g., method, findings) were second-level headers. Reference management software programs (e.g., Reference Manager, EndNote) also include searchable categories and are useful for inserting references when writing journal articles. The benefit of importing the literature review notes into CAQDAS was that, in addition to searching, I could also code them or link them to other documents (e.g., memos, research journal) integrating the literature more closely with the research process (di Gregorio 2000).
Recording the Social Milieu
Grounded theory is aimed at creating theory based in reality, not a social vacuum. It is, therefore, important to note the social milieu at the time of a focus group, interview, or analysis. Media reports are one reflection of the social context. When child abuse or sexual harassment cases were reported in the media, these were imported into NVIVO as memos. These news reports contributed to a description of the overall contextual background to understanding sexual abuse in sport. Recording these notes allowed me to reflect on the situation at the time of the data collection, thus keeping the information in perspective rather than allowing the reports to dominate my interpretation of the participants’ comments. A key tenet of grounded theory is that by focusing on the participant’s view, the developing theory will be relevant to the participants. Context is still important, and having my reflections easily accessible reminded me of the social context at the time of the interviews and focus groups.
13
Grounded Theory Data Analysis in NVIVO
The analyst following grounded theory is encouraged to oscillate between open coding, writing memos, axial coding, and modeling. Just as CAQDAS can facilitate the overall iterative process of data collection, analysis, and theorizing, CAQDAS programs are generally designed to facilitate iterations within data coding and analysis (see also Seidel 1998). I was able to move quickly from open coding to focusing more specifically on coding for distinctions within a category and back again to open coding as necessary, all the while writing conceptual and theoretical memos. Creating links between nodes, memos, and documents facilitated this iterative process as I developed the analysis interwoven from data and ideas (Weaver and Atkinson 1994; L. Richards 1999).
Coding in NVIVO
Bazeley and Richards (2000) emphasized the analytical and organizational functions of coding as they described the process of coding in NVIVO. The organizational step is the systematic process of coding that Glaser and Strauss (1967) referred to as a necessary process in reaching the more abstract goal of theorizing. Tools within NVIVO also facilitated the continual oscillation between the open coding phase of analysis (e.g., labeling age of consent as a category) and deeper analysis (looking at how age of consent related to the overall concept of appropriateness). These tools included nonhierarchical listing of categories (free nodes), hierarchical organisation of categories (tree nodes), memos, models, and search tools.
The coding processes in grounded theory starts with open coding or dissecting the data into discrete parts, examining the data for similarities and differences, and grouping together conceptually similar data to form categories. Strauss and Corbin (1998) describe conceptualizing, or giving a conceptual name to categories (represented in NVIVO by
14
nodes), as the first step in theorizing. When possible, node names were active to encourage me to think about processes rather than mere description (Glaser 1978). For example, the node name developing rapport was selected over rapport, and socializing was used instead of social contact. Using participants’ own words in category names, known as in-vivo coding, is also encouraged as a method of staying true to the data (Glaser 1978).
Monitoring consistent use of codes can be achieved through two functions in NVIVO: one that records the researcher-defined description of a node and one that allows the researcher to attach a memo directly to the node. Both functions were used in this project. CAQDAS programs generally have a feature to create a list of nodes and their descriptions, similar to a codebook (Bazeley and Richards 2000), that can be printed at any stage in the analysis. These descriptions, along with the node memos, enabled (but, of course, did not guarantee) consistent use of the nodes.
The Importance of Writing Memos
In grounded theory, memos are essential to the development of theory. Through writing memos, I moved from a descriptive mode of placing conceptually similar passages together to thinking analytically about the emerging concepts. Following recommendations by Strauss and Corbin (1990, 1998), different types of memos were created to facilitate thinking at different levels. The memo name began with the memo-type prefix so that the memos were automatically sorted in NVIVO’s document browser. In addition to being alphabetically sorted, they could be sorted by size, the number of linked nodes, and creation or modification date. NVIVO also allows documents, such as memos, to be color coded and stored in sets. The seven memo types, their prefix, number written, and their purposes are described in
15
Table 1. Memos corresponding to each focus group were named after the transcript name and linked directly to the transcript.
TABLE 1 ABOUT HERE
Memos serve multiple purposes within a grounded-theory project, including clarification, category saturation, theoretical development, and transparency. A memo was attached to each node in this project to justify the selection of passages and the naming of a node. Thus, the category name was clarified, passages compared, and categories renamed, merged, or dropped accordingly. Strauss (1987) encouraged researchers to discuss ideas conceptually in memos, rather than the actions of individual participants, thus pushing the researcher to think more broadly about possible properties and dimensions.
As writing within memos is less structured than for a formal document, there is space for ideas to develop freely without the constraints of rigid conformity to sentence structure (Glaser 1978). To develop higher-order categories and investigate links between categories, memos were physically sorted into similar categories (Strauss and Corbin 1998). In NVIVO, sets of memos can be created for this purpose. Memos also provided a record of how the project developed. Without memos, a project is likely to “lack conceptual density and integration” and transparency is minimized (Strauss and Corbin 1998:218). In addition to the organizational benefit of writing the memos within the computer program, there is the added benefit of being able to code within a memo and make links to other memos, documents, and nodes.
More than just Categorizing
Analytical techniques, such as questioning, detailed word-by-word, or line-by-line analysis, comparing extreme examples or examples from outside the area of focus, and being aware of
16
implicit assumptions, are a few tools suggested by Strauss and Corbin (1998) that can be used to move the analyst from mere description to developing theory. Focused coding can take place after codes start to continually reappear in the coding (Charmaz 1995). This inaccurately suggests that the frequency of emerging codes is related to relevance. Analytic techniques in grounded theory are designed to avoid the false assumption that frequency implies importance. CAQDAS can be used for frequency counts, but the programs also have design features that assist the researcher in recognizing gaps in coding and bringing recognition to the salient but perhaps less voiced viewpoint.
Coding can also be automated through the use of sections, as was done to code each participant’s text into a single node (or category; see Attributes section). For example, coaches who had received child protection training could be compared with those who had not. Or, I could compare all of one coach’s comments to another coach in the same focus group. This process helped me expand on the concept of age difference by examining the different responses to how age difference influenced acceptability. I could also further code directly from a node, and immediately link back to the original context, if necessary. This is important because many qualitative methods emphasize the importance of context.
Most CAQDAS programs include a variety of search operations to assist the researcher in examining the data. In NVIVO, these include the ability to search for text and combinations of text, refine a search to a node, a document, an attribute or even proximally coded items. In later stages of the project, the search tool was used to run matrix searches to examine potential links between categories. (A matrix search allows one set of items to be searched with another set, resulting in a table of paired results such as A and B; A and C; B and C.)
17
In the early stages of coding, the text search tool was used to search previously coded documents for instances of a newly developed category. This allowed me to ascertain quickly if there were instances of the concept that I had missed in the early transcripts that could add to my understanding of the node. Although participants do not always use the same words to speak about a concept, the text search tool can be used to search for sets of synonyms. For example, nine months after creating the node, law, to record instances of where the coaches referred to the law or legal aspects of sexual relationships, I used the search tool to create an assay report on the node law to examine if this concept occurred in all of the focus groups. (The assay report is a table that illustrates in which documents, or nodes, the selected item occurs.) This revealed that only three of the four focus groups had references in this node, so I then ran a text search for “law or legal or court” and found that these concepts did exist in all of the focus groups, but I had not coded it throughout.
In addition to focused coding, I “coded-on” (Bazeley and Richards 2000) from a category as a method of developing dense categories and exploring links to other categories. This is achieved by viewing all of the text in a category (by opening a browser for the corresponding node) and coding it into additional categories. Search results can also be saved as nodes, allowing the researcher to continue to code-on from the results of a search.
Not surprisingly, the coaches in this study agreed that sexual relationships with athletes below the age of consent were completely inappropriate. What I was interested in was what contributed to the variations in opinions about appropriateness of relationships with athletes above the age of consent. Therefore, I opened a browser for the node law, allowing me to see all the text coded into that category and looked for anything that might explain coaches’ differences in perceptions about legal relationships. Several new nodes (and
18
associated memos) were created to capture the influence of legal standards, instances where coaches wanted more guidance than what the law offered and where coaches disapproved despite a relationship being legal. The results of such coding can be viewed or searched at any time to examine links between categories. Thus, I was able to recognize when an incident did not fit with the rest of the category. In such cases, the text was recoded. In a few cases, categories were merged when it was discovered that the categories were essentially the same.
A node can be viewed separately, but what is more interesting is to view nodes in comparison with each other. “Coding stripes” facilitate the task of comparing categories. This is illustrated in Figure 1, where all of the text, from the interviews and focus groups, coded at the node “degrees of appropriateness” is displayed in the “node browser.” Directing the program to “show coding stripes” (as seen in the right-hand side of Figure 1) allows the researcher to see which text coded at degrees of appropriateness is also coded at other nodes.
FIGURE 1 ABOUT HERE
Glaser and Strauss (1967) describe the analysis within grounded theory as the constant comparative method. This included comparing incidents within each category, comparing categories to each other, clarifying the developing theory, and writing a coherent theory. Comparing the incidences allowed me to notice subtle differences that resulted in developing the properties and dimensions of the categories. The analysis does not actually occur in discrete stages; it is an iterative process whereby the researcher returns to various methods of coding throughout the project.
Making comparisons between nodes is made easier when the nodes are organized in a hierarchical structure (T. Richards and Richards 1995). The tree structure is an infrastructure
19
designed to help interrogate, not represent, the data. This structure evolved throughout the analysis process from the initial free nodes into a more ordered structure of nodes. Hierarchical structures are designed to make finding nodes easier, to assist in viewing categories in relation to other categories, to run complex matrix searches, and to facilitate generic higher order coding (T. Richards and Richards 1995; T. Richards 2004).
Some CAQDAS programs have a model feature so that the user can explore ideas in a visual format without changing the database of the project. The modeler was one tool used in the early conceptual development of the hierarchical structure. In an attempt to make sense of the sixty-two free nodes that were created in the early stage of the project, with one click of the mouse I imported all sixty-two nodes into the modeler. The nodes were moved around the screen into related clusters. When it was necessary to review the node, the node could be browsed simply by right clicking on the relevant node icon. I could thus oscillate between being close to the data and gaining distance for analytical purposes. In addition to viewing nodes, links to documents and memos can also be added to models. In this project, the modeler was used to design a conceptual model for the core category and to explore the overall structure for the research report.
Making comparisons at the category and subcategory level is what Strauss and Corbin (1990, 1998) refer to as axial coding. This is where the analysis moved from thick description to explaining the phenomenon of interest. Coding stripes, as seen in the right side of Figure 1, were instrumental in developing links between categories. The researcher can quickly ascertain similarities and gaps in coding. This is where the hierarchal system of coding can facilitate making comparisons. The term “parent node” is used to denote a higher
20
order storage category, whereas “child node” refers to a mutually exclusive subcategory of the parent node.
The node browser for the node degrees of appropriateness is shown in Figure 1 with coding stripes illustrated on the right side of the figure. To systematically explore the peculiarities and subtle differences between the subcategories (children) of this node (all wrong, some wrong, high morals, some OK, all OK), all of the text from each child node was gathered up and stored at the parent node for cross comparison.
The next step involved mapping coding stripes, representing each of the child nodes, onto the parent nodes (in the document browser.) This allowed a visual display of co-occurrences. Viewing the coding in this manner can highlight discrepancies in coding which might indicate the need to refine coding. Alternatively, the co-occurrences and differences may suggest meaningful relationships between categories.
Examining the coding stripes throughout the category made it apparent that age (especially age of consent) co-occurred frequently with all OK (which represented instances where all of the coaches agreed the scenario was appropriate). In contrast, fear of judging co-occurred in instances of some OK (indicating that only some of the coaches thought that the particular relationship was appropriate). Examining coding stripes was one way that I started to explore the links between categories that would lead to the development of an explanatory model.
In the above example, age has the properties age of consent and age difference, where age difference ranges from no difference to many years difference. As might be expected, age difference is one category that emerged in relation to perceptions of appropriateness. I was able to further expand categories by comparing responses from someone who did not
21
feel constricted by professional standards to those who did. Such comparisons led to new categories, confirmed in the data, that influenced perceptions of appropriateness including: what is the priority (the coaching relationship versus developing a romantic relationship), relationship boundaries, and sacrifices (sacrificing the possibility of a romantic relationship in order to maintain the coaching relationship). These became subcategories of professional manner when I performed what Glaser and Strauss labeled the second stage of comparisons, “integrating categories and their properties” (1967:105).
Attributes
Exploring demographic information may lead to a preliminary explanatory understanding of the relationship between categories. The aim is not to provide statistically relevant predictions but rather to explore preliminary relationships. Demographic information is stored as “attributes” and can be used to search data and compare responses. Attributes are essentially variables, or fields, worked within a spreadsheet-like view. The attribute data can be designated as text, numeric, or date information.
Attributes can be attached to documents or nodes, depending on the structure of the project. Demographic information for interview participants were recorded as document attributes, thus attaching the participant’s information to his entire interview. This strategy does not make sense for focus group transcripts, however, because there are multiple participants in each document. If all of what John said is coded in a node called John, and all of what Elliot said is coded to his own node, then it is possible to attach the attributes to the nodes. The researcher can search, for example, for all references to civil liberties (as tracked by coding) by coaches with no child protection training (as indicated by the attributes attached to speaker-name nodes). During transcription, each speaker’s pseudonym was set at
22
heading Level One, and the speaker’s words remained as normal text. This divided the documents into sections by speaker and allowed the text to be auto-coded by section into the respective nodes.
Figure 2 illustrates in three windows how cases and their attributes are displayed in NVIVO. The tree structure of the case nodes is seen in the left pane. Clicking on each individual case will reveal all of the text for that particular coach. The middle window is a partial view of the node attribute explorer with attributes for child protection training, level of education, and ethnicity, among other variables viewed as column headers. Each row is a case and represents one participant. The third window, in the lower-right pane, illustrates how attributes are selected for use in searches. In this instance, the attribute for child protection training was selected with the values “yes” and “no.” This type of tool can assist in developing categories and exploring relationships between categories.
FIGURE 2 ABOUT HERE
Moving from Describing to Theorizing
The conditional/consequential matrix was one of several grounded theory tools (Strauss and Corbin 1998) used during axial coding to explore the relationships between categories. The matrix is simply a heuristic diagram to assist the researcher in identifying conditions and consequences of the core category. The core category is the central theme or problem of interest that emerges from the data (Strauss and Corbin 1998). In this study, it was “role conflict and role ambiguity.” Conditions are the contextual and predisposing factors in which the core category occurs (e.g., the extent to which coaches experienced, or did not experience, role conflict and role ambiguity, was influenced by a coach’s acceptance and awareness of child protection issues, his own coaching behaviors, and whether or not there
23
was congruence between his coaching behaviors and the child protection guidelines). Consequences are the result of the actions/interactions with the core category (e.g., coaches’ attempts at managing the ambiguity).
The conditional/consequential matrix was used to explore “relationships between macro and micro conditions/consequences both to each other and to process” (Strauss and Corbin 1998:181). This tool was first used to explain differences in perceptions of appropriateness as I struggled to make sense of the data. I chose to work away from the computer and sketched fifteen different models and conditional/consequential matrices before any coherent results were evident. I identified concepts on the micro (or individual level) that seemed to influence perceptions. These included age and personal experiences of being in a coach-athlete intimate relationship. Moving away from micro issues, the club environment and scrutiny from others (e.g., parents or other club members) also appeared to influence perceptions of appropriateness. On a more macro (or societal) level, new sport guidelines and changes in public awareness about abuse seemed to influence perceptions. Although I explored these concepts on paper, I returned to the computer frequently to memo my thoughts and to explore the emerging concepts in the data.
A second grounded theory heuristic tool called the paradigm model (which builds on the conditions and consequences by considering the structure and process of actions and interactions) was also used to structure the data in a more systematic manner. Where the conditional/consequential matrix focuses attention on conditions that need to be present for the core category to exist and consequences of the core category, the paradigm model has a broader focus that includes actions/interactions (what the coaches did to manage the role conflict and ambiguity). Actions/interactions are the responses that are taken to the core
24
problem (e.g., a coach might change his definition of effective coaching or he might assess the risks of not changing). Actions/interactions are not static, rather they “evolve over time as persons define or give meanings to situations” (Strauss and Corbin 1998:134).
The paradigm model was used to help guide my understanding of the main issues for the coaches. Initially, my research was focused on perceptions of appropriateness. But, as I analyzed the data and used the elements of the paradigm model to try to figure out how the concepts were related to each other, I kept noticing that the child protection policies appeared to be forcing coaches to reevaluate their coaching and perceptions of “good coaching.” I noted in a memo that the coaches’ perceptions of appropriateness are a reflection of their attempt to resolve the conflict between child protection policies and good coaching. The main issue for concern for the coaches seemed to be how to resolve the role conflict and ambiguity brought about by child protection issues. Once I had identified role conflict and role ambiguity as the core category, I further examined the links between the conditions leading to role ambiguity and role conflict, how coaches managed the ambiguity and conflict (through actions and interactions), and the consequences of those actions and interactions (see Bringer 2002; Bringer, Brackenridge, and Johnston 2002b). CAQDAS was useful in this stage because it provided easy access to the memos and nodes of interest as well as the original transcripts.
The modeler within NVIVO could have facilitated this stage by allowing me direct access to the data. However, the size of my computer screen limited how much of the model I could see at once (see Limitations section). The layer function can help overcome the limitation of screen size by allowing the user to expand or collapse the model.
25
In addition to the contextual/consequential matrix and the paradigm model, I used the idea of writing a descriptive storyline (Strauss and Corbin 1998) to further understand how the concepts in the data fit together. The purpose of this is to help verbalize the main concepts and their relationships to each other. Working through these grounded theory tools, I developed a preliminary model of concepts influencing perceptions of appropriateness and how these differed according to whether the coach was evaluating his own beliefs and actions or that of other coaches. Public scrutiny and avoiding false accusations were concepts that influenced coaches’ own beliefs and actions. When commenting about other coaches, the concepts of power imbalance (between coach and athlete), evaluating consequences (of the relationship), and a reluctance to interfere were the main concepts influencing their perceptions of appropriateness (see Bringer, Brackenridge, and Johnston [2002a] for a more detailed discussion).
In grounded theory, it is important to examine how each case fits with the emerging theory, to see whether it is an extreme dimension of a concept or a contradictory case. For example, there were a few coaches who represented negative cases in that they did not seem to be experiencing role conflict and ambiguity. I returned to the data to look at what conditions were present and how the coaches varied on these conditions compared with those who did experience role conflict and ambiguity. These negative cases helped me further develop my emerging theory by forcing me to examine how they differed from the other coaches.
26
Limitations
Like any approach to processing data, CAQDAS has limitations. Researchers who are not confident using computers and new programs may take longer to learn how to use CAQDAS than learning a particular strategy of managing the data by hand (Fielding and Lee 1998). Weitzman (2000) advises CAQDAS users to answer questions about their own computer literacy, the type of project, the type of data, and the anticipated type of analysis before selecting a software program. Users may also be tempted to use all of the program’s functions (Mangabeira, Lee, and Fielding 2004), rather than just the ones that would be beneficial in answering their research question from their particular methodological standpoint. Equally, users could be enticed into thinking that they must use the computer for every stage of the analysis. Agar (1991) asserted that analysis involves thinking, which for him is facilitated by space to see more data and concepts than fit on a computer screen. In a similar manner, I found myself drawing diagrams by hand, printing memos, and using a whiteboard to discuss concepts with colleagues. Creating the time and space to think about ideas, as well having discussion with fellow researchers is advocated as a necessity in grounded theory. Use of a computer program need not change these valuable activities.
Over the years, much has been written about the pros and cons of CAQDAS as well as speculation about how the research might have been different if CAQDAS had or had not been used. Some research supervisors and examiners are sceptical of CAQDAS, perhaps because they do not understand the software (Delamont, Aitkinson, and Parry 2004) or they believe in a false dichotomy between research tool and process (Johnston 2004). Inherent in questioning how the research might have been different is the implication that the tool (manual versus computer) is the main determinant in the research outcome. Arguably, how
27
well the researcher follows the chosen research methodology and applies the selected research methods will have a greater impact than whether or not CAQDAS is used. In answering questions about the impact of the use of CAQDAS on a research project, the researcher’s knowledge of qualitative research and qualitative computing must be taken into account (Fielding and Lee 2002; Mangabeira, Lee, and Fielding 2004).
28
Acknowledgments
This article is based on Joy D. Bringer’s doctoral thesis, therefore, use of the first person pronoun has been maintained to refer directly to Dr. Bringer. The authors would like to thank conference attendees who commented on an earlier version of this paper at the Strategies in Qualitative Research: Methodological Issues and Practices Using QSR NVIVO and NUD*IST conference hosted by the Institute of Education, University of London (May 8–9, 2003). We are also grateful to the Amateur Swimming Association and the Institute of Swimming Teachers and Coaches that funded the research and the British Association for Swimming Teachers and Coaches that provided logistical support.
Notes
1. Shareware is software that is available on a trial basis at no cost. The user is generally expected to pay a fee for use beyond the trial period.
2. Internal annotations and external files can still be coded, but this is by coding the anchored text.
29
References Cited
Agar, M. 1991. The right brain strikes back. In Using computers in qualitative research, N. G. Fielding and R. M. Lee, eds., 181–94. London: Sage.
Annells, M. 1996. Grounded theory method: Philosophical perspectives, paradigm of inquiry, and postmodernism. Qualitative Health Research 6 (3): 379–93.
Bazeley, P. 2002. Computerized data analysis for mixed methods research. In Handbook of mixed methods for the social and behavioral sciences, A. Tashakkori and C. Teddlie, eds., pp. 385-422. London: Sage.
Bazeley, P., and L. Richards. 2000. The NVIVO qualitative project book. London: Sage. Becker, P. H. 1993. Common pitfalls in published grounded theory research. Qualitative Health Research 3 (2): 254–60.
Brackenridge, C. H. 1999. Managing myself: Investigator survival in sensitive research. International Review for the Sociology of Sport 34 (4):3 99–410.
Bringer, J. D. 2002. Sexual exploitation: Swimming coaches’ perceptions and the
development of role conflict and role ambiguity. Unpublished doctoral thesis, University of Gloucestershire, Cheltenham, UK.
Bringer, J. D., C. H. Brackenridge, and L. H. Johnston. 2002a. Defining appropriateness in coach-athlete sexual relationships: The voice of coaches. Journal of Sexual Aggression 8 (2): 83–98.
. 2002b. Role ambiguity among swimming coaches: The impact of child protection
policies on coaching practice. Paper presented at the Annual Meeting of the
30
Association for the Advancement of Applied Sport Psychology, Tucson, Arizona, October 31.
Bringer, J. D., L. H. Johnston, and C. H. Brackenridge. 2004. Maximizing transparency in a doctoral thesis: The complexities of writing about the use of QSR*NVIVO within a grounded theory study. Qualitative Research 4 (2): 247–65.
Charmaz, K. 1995. Grounded theory. In Rethinking methods in psychology, J. A. Smith, R. Harré, and L. V. Langenhove, eds., 27–48. London: Sage.
. 2000. Grounded theory: Objectivist and constructivist methods. In Handbook of qualitative research. 2nd ed., N. K. Denzin and Y. S. Lincoln, eds., 509–35. London: Sage.
Delamont, S., P. Aitkinson, and O. Parry. 2004. Supervising the doctorate: A guide to success. Maidenhead, UK: McGraw Hill Education.
di Gregorio, S. 2000. Using NVivo software for literature reviews. Paper presented at the Strategies in Qualitative Research: Issues and Results from Analysis Using QSR NVivo and NUD*IST, The Institute of Education, University of London, September 30. Full paper available at http://k1.ioe.ac.uk/conferences/nvivo/conf2000_sess1.htm.
Fielding, N. G., and R. M. Lee. 1998. Computer analysis and qualitative research. London: Sage.
. 2002. New patterns in the adoption and use of qualitative software. Field Methods
14 (2): 197–216.
Gibbs, G. 2002. Qualitative data analysis: Explorations with NVivo. London: Open
University Press.
31
Glaser, B. G. 1978. Theoretical sensitivity: Advances in the methodology of grounded theory. Mill Valley, CA: The Sociology Press.
. 1992. Basics of grounded theory analysis: Emergence vs. forcing. Mill Valley, CA:
Sociology Press.
Glaser, B. G., and A. L. Strauss. 1967. The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter.
Johnston, L. H. 2004. Technical and methodological learning curves: Reflections on the use of QSR NVIVO in doctoral research. Paper presented at the Strategies in Qualitative Research: Methodological Issues and Practices in Using QSR NVIVO and NUD*IST, Durham, UK, September 1–3.
Kelle, U. 1995. Introduction: An overview of computer-aided methods in qualitative
research. In Computer-aided qualitative data analysis: Theory, methods, and practice, U. Kelle, G. Prein, and K. Bird, eds., 1–17. London: Sage.
Lee, R. M., and N. G. Fielding. 1991. Computing for qualitative research: Options, problems and potential. In Using computers in qualitative research, N. G. Fielding and R. M. Lee, eds., 1–13. London: Sage.
Lewins, A., and C. Silver. 2004. Choosing a CAQDAS package. Retrieved April 4, 2005, from http://caqdas.soc.surrey.ac.uk/
MacMillan, K., and T. Koenig. 2004. The wow factor: Preconceptions and expectations for data analysis software in qualitative research. Social Science Computer Review 22 (2): 179–86.
32
Mangabeira, W. C., R. M. Lee, and N. G. Fielding. 2004. Computers and qualitative
research: Adoption, use, and representation. Social Science Computer Review 22 (2): 167–78.
May, K. A. 1996. Editorial: Diffusion, dilution, or distillation? The case of grounded theory method. Qualitative Health Research 6 (3): 406–28.
Melia, K. M. 1996. Rediscovering Glaser. Qualitative Health Research 6 (3): 368–78. Morse, J., and L. Richards. 2002. Read me first for a user’s guide to qualitative methods. London: Sage.
QSR. 2000. QSR NUD*IST VIVO (Version 1.3). Melbourne, Australia: QSR International Pty Ltd.
Richards, L. 1998. Closeness to data: The changing goals of qualitative data handling. Qualitative Health Research 8 (3): 319–28.
. 1999. Data alive! The thinking behind NVIVO. Qualitative Health Research 9 (3): 412–28.
. 2005. Handling qualitative data. London: Sage.
Richards, T. 2004. Unleash the power within! What node hierarchies are really all about, why they are the heart of powerful research techniques, and where they can take us. Paper presented at the Strategies in Qualitative Research: Methodological Issues and Practices in Using QSR NVIVO and NUD*IST, Institute of Education, University of London, May 8.
Richards, T., and L. Richards. 1995. Using hierarchical categories in qualitative data analysis. In Computer-aided qualitative data analysis: Theory, methods, and practice, U. Kelle, G. Prein, and K. Bird, eds., 80–95. London: Sage.
33
Seidel, J. V. 1998. Qualitative data analysis. In The Ethnograph v5 user’s manual, Appendix E. Denver: Qualis Research Associates. Accessed March 2005 from www.qualisresearch.com
Seidel, J. V., and U. Kelle. 1995. Different functions of coding in the analysis of textual data. In Computer-aided qualitative data analysis: Theory, methods, and practice, U. Kelle, G. Prein, and K. Bird, eds., 52–61. London: Sage.
Strauss, A. L. 1987. Qualitative analysis for social scientists. London: Sage.
Strauss, A. L., and J. Corbin. 1990. Basics of qualitative research: Grounded theory procedures and techniques. London: Sage.
. 1998. Basics of qualitative research: Techniques and procedures for developing grounded theory. 2nd ed. London: Sage.
Weaver, A., and P. Atkinson. 1994. Microcomputing and qualitative data analysis. Aldershot, UK: Avebury.
Weitzman, E. A. 2000. Software and qualitative research. In Handbook of qualitative
research, 2nd ed., N. K. Denzin and Y. S. Lincoln, eds., 803–20. London: Sage. Wilson, H. S., and S. A. Hutchinson. 1996. Methodological mistakes in grounded theory.
Nursing Research 45 (2): 122–24.
34
Table 1: Memo Types Used in This Project
Prefix Type and number Purpose
Mcn Code note memos Define the node and record analytical
(n = 110) thinking about the node; include links
to other nodes and memos
MTH Theoretical memos Higher-order memos for evolving
(n = 5) theory at a more abstract level;
summary memos and thoughts about selective sampling
MOP Operational memos Notes about procedures, what
(n = 4) questions to ask in the next interview
(no prefix; electronic drawings Diagrams Visual representations of relationships
are stored in NVIVO’s modeler, (n = 40) among categories hand drawings are in a folder)
nCm News, contextual memos (n = 15) News articles (or memos on reports)
that influence, or illustrate, the context of child abuse and sexual harassment
nVa NVIVO memos Technical notes about using NVIVO
(n = 3)
mEM Executive meetings Notes from meetings with governing
(n = 2) body officials to discuss thesis results
35
Browser Node Document Edit view Tools
q!. /Actor Charactermtic0Age -"1 C7 0:!. A A IrB
0 15,05 11,„ /Degrees of appropriateness
Figure 1: Coding stripes illustrate where text coded at one node co-occurs with other coding.
36
Figure 2. Viewing and searching participant attributes in NVIVO. (Note: Participant names are pseudonyms.)
37
APPENDIX C.
Tidak ada komentar:
Posting Komentar