THE HOUSING MARKET CRASH
By Todd J. Zywicki and Gabriel Okloski
The ideas presented in this research are the authors’ and do not represent official positions
of the Mercatus Center at George Mason University.
Mercatus Working Paper – The Housing Market Crash
By Todd J. Zywicki and Gabe Okloski
Introduction
Beginning in 2007 and continuing into 2008 and 2009, the residential real-estate market was roiled by tumult unprecedented in recent American history. Widespread foreclosures and a collapse in home prices in many areas of the country spawned an ongoing global financial crisis. Although home prices have fallen precipitously in many areas of the country and foreclosures have risen to all-time highs, the end of the crisis still may not be in sight. The United States government has engineered a series of unprecedented market interventions designed to stabilize the housing market and the financial markets dependent on mortgage-backed securities.
Consumer borrowing secured by residential real estate grew substantially over the past several years, due to a number of factors that tend to increase the value of housing by increasing the willingness of purchasers to pay higher prices for the houses. Standard economics thus provides a compelling explanation for much of the increase in household mortgage obligations—low interest rates, high effective tax rates, and the increased capital value of residential real estate. Other factors, moreover, are more difficult to explain by fundamentals, such as the prevalence of a large number of speculative investors in some of the major boom and bust markets.
This working paper focuses on underlying questions related to consumer behavior and looks at the impact of these developments in the housing market on household financial condition. Specifically, this paper looks at the factors that drove consumer demand during the “bubble” market that later popped with such widespread effects. It
particularly investigates foreclosure as a rational decision on the part of consumers who responded to various incentives put in place before the eventual fall of housing prices and crash in the housing market.
The Rise and Fall of the Mortgage Market
In contrast to this relative stability in the overall housing and residential mortgage market between 1980 and 2000, recent years have seen an unprecedented rise and implosion of the residential mortgage market, especially the rapid development of the subprime mortgage market. These events have given rise to major woes in the American real-estate market and overall economy and have major implications for thinking about consumer lending markets generally.
The mortgage market collapse can be studied from many different perspectives. Many commentators have focused on the impact of mortgage lending crisis on the financial side of the issue, such as the collapse of leading investment banks and governmental efforts to stabilize the financial economy. The discussion in this working paper, by contrast, will focus on the consumer side of the market, examining the real-estate and mortgage market to determine what it can tell us about consumer behavior more generally.
Homeownership and Economic Welfare
Homeownership can be a transformative life experience, both economically and psychologically. Homeownership historically has been an important source of wealth for American households and the primary method of wealth accumulation for low and
moderate-income people. 1 According to the 2004 Survey of Consumer Finances, a family that owns a home on average has $624,900 in average wealth (median of $184,400) and the average renter family has $54,100 ($4,000 median). The impact of homeownership on increasing the wealth of lower-income families is especially important, as low-income families generally do not own financial assets. In 2001, for example, the average low-income homeowner (annual income is less than $20,000) had nearly $73,000 in net wealth, compared with a similar renter with only $900 of net wealth.2 In fact, homeownership has been such a potent vehicle for wealth accumulation that the polarization of wealth between homeowners and renters has risen dramatically, even as the wealth polarization among different income classes has decreased.3 Low-income and even middle-class homeowners rely on homeownership for the majority of their net worth—almost 80 percent of the wealth of low-income households is in residential real estate.4 The richest quintile by income is the only income group that holds stock wealth in equal value to their home equity. The bottom four quintiles typically have home equity equal to at least twice the value of their stocks.5
In addition to improving the asset side of the household balance sheet, homeownership also may be valuable to the liabilities side of the balance sheet. The Federal Reserve’s financial obligations ratio calculates the percentage of household
1 Thomas P. Boehm & Alan Schlottmann, DEP’T OF HOUS. & URBAN DEV., Wealth Accumulation and Homeownership: Evidence for
Low-Income Households 11–14 (2004), available at
http://www.huduser.org/Publications/pdf/WealthAccumulationAndHomeownership.pdf.
2 Zhu Xiao Di, Housing Wealth and Household Net Wealth in the United States: A New Profile Based on the Recently Released 2001 SCF Data 10 (Harvard U., Joint Ctr. for Hous. Studies Working Paper No. W03-8, 2003).
3 See Conchita D’Ambrosio & Edward N. Wolff, Is Wealth Becoming More Polarized in the United States? 14–16 (Jerome Levy Economics Inst. of Bard College Working Paper No. 330, 2001), available at http://ssrn.com/abstract=276900. Wealth inequality appears to have increased over time, but wealth “polarization” is different from “inequality” in that polarization studies the clustering of homogeneous groups, such as homeowners, within a heterogeneous population. See id. at 2. Thus, it is a more useful tool for examining the effect on wealth of particular subsets, such as homeowners.
4 Di, supra note 2.
5 Id.
income dedicated to monthly payment obligations, including monthly rental payments on homes, apartments, and automobiles, real-estate tax obligations, and the debt-service burden, which includes monthly payments on mortgages, car payments, student loans, and credit cards.6 The household financial obligations ratio (“FOR”) is substantially higher for those households that rent compared to those that own their homes.7 Data indicates that homeowners also save more than do non-homeowners.8 Although some of this difference surely is attributable to the fact that homeowners generally have higher incomes than renters, renters also are more likely to revolve credit card debt and to hold student loan debt, both of which generally carry higher interest rates than mortgage debt.
In addition to these direct benefits, homeownership apparently has a number of indirect benefits. For instance, homeownership is correlated with a substantial increase in one’s propensity to vote, dramatic improvements in children’s life outcomes, and improvements in labor-market outcomes. Homeownership also creates incentives to improve property, generally increases life satisfaction, and is correlated with a reduction in crime rates. 9 Recent research, however, questions the long-believed causal link between homeownership and these other benefits, arguing instead that there is a selection mechanism at work, i.e., that people with certain attributes tend to self-select into homeownership, or that other factors (such as reduced mobility caused by homeownership) explain the relationship between homeownership and observed positive
6 See FED. RES. BOARD, HOUSEHOLD DEBT SERVICE AND FINANCIAL OBLIGATIONS RATIOS (June 10, 2008), available at http://www.federalreserve.gov/Releases/housedebt/.
7 The Federal Reserve defines these measures as follows: “The household debt service ratio (DSR) is an estimate of the ratio of debt payments to disposable personal income. Debt payments consist of the estimated required payments on outstanding mortgage and consumer debt. The financial obligations ratio (FOR) adds automobile lease payments, rental payments on tenant-occupied property, homeowners' insurance, and property tax payments to the debt service ratio.”
8 Ed Gramlich, Subprime Mortgages: America’s Latest Boom and Bust 75–77 (2007).
9 See id. at 58–60; Christopher E. Herbert & Eric S. Belsky, DEP’T OF HOUS. & URBAN DEV., The Homeownership Experience of Low-Income and Minority Families: A Review and Synthesis of the Literature (Feb. 2006); Robert D. Dietz & Donald R. Haurin, The Social and Private Micro-Level Consequences of Homeownership, 54 J. URB. ECON. 401 (2003).
outcomes. 10 Thus, while there is a correlation between homeownership and many personal and social benefits, that correlation may not be causal in nature. There are costs to homeownership as well, notably increased sprawl and a less mobile labor force. 11 Nonetheless, policymakers have long (and somewhat reasonably, based on prevailing data) believed that the benefits of widespread homeownership outweigh the costs, and, therefore, expanding homeownership rates historically has been a linchpin of American financial and social policy. 12
The Growth in Homeownership
Homeownership grew rapidly beginning in the mid-1990s and continued to rise until it
reached its peak in 2004, as seen in figure 1, below:
10 See discussion in Kristopher S. Gerardi & Paul S. Willen, Subprime Mortgages, Foreclosures, and Urban Neighborhoods, FED. RES. BANK OF BOSTON No. 08-6 (Dec. 22, 2008).
11 Fernando Ferreira, Joseph Gyourko & Joseph Tracy, Housing Busts and Household Mobility (Nat’l Bureau of Econ. Research Working Paper No. 13410, 2008), available at http://www.nber.org/papers/w14310; Robert D. Dietz & Donald R. Haurin, The Social and Private Micro-Level Consequences of Homeownership, 54 J. URB. ECON. 401, 405 (2003).
12 See Melissa B. Jacoby, Homeownership Risk Beyond a Subprime Crisis: The Role of Delinquency Management, 76 FORDHAM L. REV. 2261 (2008).
Figure 1
Ownership by minority groups grew during the period of expanding
homeownership:
Figure 2
And the young:
Figure 3
The Rise and Fall of the Mortgage Market
Beginning in late 2006 and continuing into 2007 and 2008, the United States residential real-estate market collapsed into widespread turmoil. Foreclosures rose steeply, resulting in chaos in the banking industry as well as complex securities backed by these mortgages collapsed in value. One Web site tracking the subprime bust has estimated that as of September 2009, 360 lenders have “imploded” since late 2006—i.e., gone bankrupt, halted major lending operations, or been sold at a “fire sale” price. 13
In fact, the American economy has suffered an unprecedented rise in foreclosures. Figure 4 illustrates foreclosure start rates by yearly average over the past several decades.
13 The Mortgage Lender Implode-O-Meter Homepage, http://ml-implode.com/ (last visited September 27, 2008).
Figure 4
The surge in foreclosures is often attributed to the growth of the subprime segment of the market during the 1990s and 2000s and the extension of mortgages to high-risk consumers who historically were locked out of the mortgage market. Congress, the Department of Housing and Urban Development, and Fannie Mae and Freddie Mac
all encouraged more lending to higher-risk borrowers. 14 Others have argued that this growth in high-risk lending was spawned by the rise of securitization of mortgages by
Wall Street, which created an “originate to distribute” model of reckless lending. 15 Whatever the inspiration for increased lending to higher-risk borrowers, to make these loans possible mortgage originators developed a variety of novel lending products, such as no or low downpayments, interest-only loans, reverse amortization, no- or low-documentation loans, and loans with high loan-to-value (LTV) ratios. In turn, many of
14 Russell Roberts, House Government Stoked the Mania, WALL ST. J., Oct. 3, 2008; Peter J. Wallison & Charles W. Calomiris, The Last Trillion-Dollar Commitment: The Destruction of Fannie Mae and Freddie Mac, AM. ENTER. INST. FOR PUB. POLICY RESEARCH, Sept. 30, 2008, available at http://www.aei.org/docLib/20080930_Binder1.pdf.
15 On the other hand, those involved at every step in the loan process from origination to securitization to default insurance have suffered massive losses from the collapse of the subprime market thus it doubtful that this “originate to distribute” model explains much of the rise and fall of the subprime market. See Todd J. Zywicki and Joseph A. Adamson, The Law and Economics of Subprime Lending, 80 U. COLO. L. REV. 1 (2009).
them were securitized and sold throughout the United States and the world leading to global economic problems. Additionally, there was a severe deterioration in underwriting standards during the subprime boom and growth in more risky loan products.
While foreclosures (like bankruptcy) often result from forces outside the household’s control (such as unemployment), many consumers act rationally and respond to incentives in deciding whether to permit default and foreclosure. Changing the incentives to default will have an effect on the propensity of borrowers to default and permit foreclosure, which results from three basic factors: adverse “trigger” events, mortgage payment shock, and negative home equity. Each of these three factors has dovetailed to contribute to the extraordinary foreclosure rates that developed.
Adverse Trigger Events.
Foreclosure can be caused by adverse life “trigger” events, such as job loss, divorce, illness, or some other factor that causes an unexpected dramatic drop in household income or increase in expenses. Although many of these factors are chronic and universal aspects of the human condition, others can cause foreclosure spikes in particular places at particular times. Macroeconomic trends play a substantial role in increased mortgage default and delinquency.
Delinquencies and foreclosures began to rise in Michigan, Ohio, and Indiana,16 before the rest of the country as a result of troubles in the American automotive industry and resultant layoffs and plant closures. 17 Major natural disasters may also trigger
16 Where Subprime Delinquencies are Getting Worse, WALL ST. J. ONLINE, Mar. 29, 2007, at Map 2, available at http://online.wsj.com/public/resources/documents/info-subprimemap07-sort2.html (click “Map 2” header) (data provided by First American Loan Performance).
17 Chris Mayer, Karen Pence & Shane M. Sherlund, The Rise in Mortgage Defaults, 23 JOURNAL OF ECONOMIC PERSPECTIVES 27-50 (2009),, at 45.
geographical surges in foreclosure, as resulted in the areas of Louisiana and Mississippi affected by hurricane Katrina in 2005 following the expiration of a temporary moratorium period.18 Problems in local labor markets also exert downward pressures on local home prices, making refinancing more difficult and reducing incentives to retain a home in the face of financial pressures. Thus, the adverse trigger events theory of foreclosures explains some element of regional and temporal variations in foreclosures over time.
Yet foreclosures rates in states such as California and Florida are much higher than in these economically hard-hit states. 19 Adverse trigger events plainly cannot explain the record levels of foreclosures of recent years. Indeed, during the time that foreclosures skyrocketed, the economy remained relatively robust, with low unemployment and modest but positive economic growth. Indeed, whereas the adverse trigger events theory posits that rising foreclosures result from recession and a slowing economy, during the recent foreclosure crisis that basic causal relationship has been reversed—the dramatic rise in foreclosures has caused the subsequent financial crisis and recession.
Moreover, foreclosure and delinquency do not necessarily indicate the presence of unaffordable loans, predatory loans, rising interest rates, or borrowers under duress, especially with respect to subprime loans. Borrowers face a number of options with their loans—timely repayment, prepayment, delinquency, or default followed by foreclosure. Although the latter two options typically are assumed to be evidence of financial distress, the reality is more complicated. There is some evidence that subprime borrowers use their mortgages as a type of line of credit and choose to miss an occasional payment and
18 Id.
19 See National Association of Realtors Field Guide to Foreclosures, updated February 2009, available at http://www.realtor.org/library/library/fg329.
remain delinquent in order to smooth temporary financial problems. 20 In fact, loans that are delinquent over a long period of time typically terminate in prepayment rather than eventual default.21 This counterintuitive finding suggests that these homeowners are likely using the opportunity to remain delinquent to take advantage of the “free rent” of the delinquency period, using the opportunity to miss payments in order to smooth their income and manage their finances and to simply take advantage of the opportunity to delay and develop a solution to the problem.22
Mortgage Payment Shock.
Foreclosure can also result from an unexpected increase in a household’s monthly payment obligations. In recent years, this resulted from the proliferation of adjustable-rate and “hybrid” ARM loans that had an initial period of a fixed “teaser” rate followed by adjustable rates for the duration of the loan. The easy-money policies followed by the Federal Reserve in the wake of the September 11, 2001 terrorist attacks opened up a substantial gap in market interest rates between short-term and long-term mortgages. Subsequent tightening of Fed monetary policy closed this gap, triggering resets on mortgage payments at higher rates. Borrowers who took ARMs initially were able to qualify for much larger principle amounts than was possible at long-term interest rates. Many of the states with the greatest percentage of ARMs (especially California and Florida) also saw the fastest run-ups in housing prices—and now the highest foreclosure rates.
20 Cutts & Van-Order, On the economics of subprime lending at 172. Even with penalties, the cost of credit through a delinquent mortgage is often lower than the alternatives a subprime borrower faces.
21 Michelle A. Danis & Anthony Pennington-Cross, A Dynamic Look at Subprime Loan Performance 13 (Fed. Res. Bank of St. Louis, Working Paper 2005-029A, May 2005), available at http://research.stlouisfed.org/wp/2005/2005-029.pdf.
22 Id.
Interest rates have generally fallen over the past twenty-five years following the
exceedingly high mortgage interest rates of the early 1980s, as shown in figure 5.
Figure 5
Falling interest rates probably reflect a general consensus in the western world about the value of low-inflation policies and steady economic growth in many countries that kept economic growth ahead of any inflation. Whatever the cause of this downward trend in interest rates over the past two decades, they were low during the period of the house price boom and short-term interest rates even dipped into what was likely a negative interest rate in light of actual inflation.
More important, however, was the pattern of interest rates during the crucial period of 2001–2007. Interest rates on 30-year fixed-rate mortgages remained relatively steady during that period. Interest rates on adjustable rate mortgages, by contrast, dipped to extraordinarily low interest rates, reflecting the Federal Reserve’s “easy money” policy during this period. These low ARM interest rates allowed many borrowers—both prime and subprime borrowers—to qualify for much larger mortgages than would otherwise be
the case. As can also be seen, however, beginning in 2004 the Federal Reserve began raising interest rates, causing a dramatic rise in the ARM rate, until by 2006 the interest rate on ARM and FRM mortgages had essentially converged.
The Federal Reserve’s monetary policy generated an incentive to consumers to utilize ARMs to finance their homes. These recent low-interest rates also seems to have had a measurable effect on consumers’ abilities to afford a home, despite the general rise in prices. Figure 6 presents the “Housing Affordability Index,” as reported by the Department of Housing and Urban Development, which measures the ratio of median family income to the income necessary to qualify for a mortgage to purchase the median-priced house at prevailing interest rates. Thus, an index value of over 100 indicates that the typical (median) family has more than sufficient income to purchase the median-priced home. As seen in the data, home affordability appears to keep pace with the previous two decades over the past few years despite a jump in housing prices. Favorable interest rates likely facilitated the larger mortgages that kept affordability at a stable level during the past several years.
Figure 6
Despite relatively stable affordability during the recent rise in the housing market, however, a large number of highly leveraged buyers became homeowners during this time. Figure 7 shows data from the Federal Reserve on the mortgage debt service ratio— the percentage of monthly income dedicated to mortgage debt service since 1980.
Figure 7
The portion of disposable income, that is post-tax income, going to pay off mortgage debt rises from 1980 to the early 1990s, after which it hovers around 6 percent until a rise beginning in 2000. While the recent increase may be described in part by a rising tax burden or the upward surge in the stock market, which left many homeowners feeling wealthier and led them to convert stocks into larger mortgages, there is another potential factor at play. Due to low-interest-rate loans and the increasing availability of adjustable-rate mortgages, more financially marginal homeowners were likely entering the market. Indeed, during the period that saw an increasing homeownership rate and
stable affordability, the proportion of income spent on mortgages sees its sharp rise. Presumably, lower-income homebuyers holding larger mortgages than they used to have greatly contributed to the increase seen in the figure above.
While there may be discussion on the extent to which riskier homeowners have contributed to the trend of increased income spent on mortgages, it is certain that more and more buyers took advantage of ARM mortgages to finance housing purchases. The large dip in ARM interest rates, relative to interest rates for FRM mortgages, led to a general growth in ARMs during the past several years, as shown in figure 8.
Figure 8
As can be seen, the ratio of ARMs to FRMs rose during the period of low ARM interest rates. But this high market share for ARMs is not unprecedented. ARMs are not uncommon in recent American history and, in fact, were much more common in the past than in recent years. In 1984, ARMs comprised 61 percent of the conventional mortgage market and in 1988 the figure was 58 percent. Moreover, ARMs (with average loan terms substantially shorter than the 30-year term in the United States) are standard fare in the
rest of the world and efforts to introduce the American 30-year fixed-rate mortgage have generally failed. This suggests that ARMs are not inherently dangerous products.
The popularity of ARMs appears to be driven by one overriding factor—the spread between fixed and adjustable rates, i.e., as the spread between fixed and adjustable rates widens, consumers shift to adjustable rates. As seen in figure 9, over time there is a clear relationship between the spread between interest rates on ARM and FRM mortgages and the percentage of mortgages that are ARMs.
Figure 9
At the height of the housing boom in 2004, the spread between ARMs and FRMs was about two percentage points and about 40 percent of the mortgages that were written were ARMs. As can be readily seen, however, the percentage of ARMs was even higher at times in the past, yet this did not lead to a financial calamity. This strongly suggests that ARMs are not inherently dangerous.
Fixed-rate mortgages provide homeowners with insurance against fluctuations in interest rates. And as figure 9 illustrates, this insurance usually is far from free:
Borrowers pay about 100 basis points on average to avoid bearing this risk (and sometimes more than 200 basis points). The risk of ARMs is that one’s mortgage interest rate will rise if interest rates rise. But the equally obvious benefit of an ARM is that one’s interest rate will fall if interest rates fall. Although adjustable-rate mortgages appear unreasonably risky when interest rates rise, it must be recognized that they are also equally beneficial when interest rates fall. For a fixed-rate borrower to benefit from falling interest rates, she had to incur the substantial cost and hassle of refinancing the mortgage as well as the uncertainty about whether interest rates would go still lower. Because ARMs offer lower interest rates, they may also be especially attractive to homeowners who plan to move within a few years and thus have little need to pay a premium to buy “insurance” to hedge against long-term fluctuations in interest rates.
Interest-rate resets connected to adjustable-rate mortgages helps to explain the rapid rise in foreclosure rates. Moreover, it helps to explain the spread of the foreclosure contagion beyond the subprime market into the prime market in many areas, as seen in figures 10 and 11.
Figure 10
Figure 11
As can readily be seen, the initial surge in foreclosures for both prime and subprime mortgages were a manifestation of ARMs, not of subprime lending. A dramatic rise in the subprime ARM foreclosure rate begins in 2006, and although the foreclosure rate on subprime FRMs rises, it actually remains lower than at periods in the past. A similar pattern can be seen observed in the prime mortgage market—the growth in ARMs greatly
outpaces the growth in FRMs. In part this distinction in default rates reflects differential sorting by lenders among subprime borrowers for fixed- and adjustable-rate mortgages as subprime ARM borrowers have substantially lower FICO credit scores and higher combined LTV ratios than subprime FRM borrowers.23 The difference, however, is not huge and it is difficult to imagine that the characteristics of the borrowers alone rather than the characteristics of the loans themselves explain the dramatically different performance of these loans.
In short, the “payment shock” theory may have some validity in the current climate although the mechanism of transmission is difficult to understand. The artificial lowering of interest rates from 2001–2004 pushed down short-term interest rates, allowing borrowers to qualify for larger mortgages than they otherwise could. But this was a phenomenon that was not limited to the subprime market. On the other hand, ARM-related payment shock does not provide a comprehensive explanation of all foreclosures. One estimate of subprime loans facing foreclosure in the early wave of foreclosures found that 36 percent were for hybrid loans (with an initial fixed period, followed by adjustable rates for the duration of the loan), fixed-rate loans account for 31 percent, and adjustable-rate loans for 26 percent.24 Of those loans in foreclosure, the overwhelming majority entered foreclosure before there was an upward reset of the interest rate.25 Most defaults on subprime loans occur within the first 12 months of the
23 Mayer, Pence & Sherlund, The Rise in Mortgage Defaults, supra note 17 at 32.
24 James R. Barth et al., Mortgage Market Turmoil: The Role of Interest-Rate Resets, in SUBPRIME MORTGAGE DATA SERIES (Milken Inst.) (2007); C.L. Foote, K. Gerardi, L. Goette & P.S. Willen, Subprime Facts: What (We Think) We Know about the Subprime Crisis and What we Don’t, FED. RES. BANK BOSTON PUBLICLY POLICY DISCUSSION PAPER 08-02 (2007); Mayer, Pence & Sherlund, The Rise in Mortgage Defaults, supra note 17.
25 Barth at 18, available at http://www.ghb.co.th/en/Journal/Vol2/07.pdf. Of those subprime loans in foreclosure, 57 percent of 2/28 hybrids and 83 percent of 3/27 hybrids “had not yet undergone any upward reset of the interest rate.”
loan, well before any interest adjustment.26 Furthermore, after examining the evidence, several economists from the Boston Federal Reserve flatly state, “Interest-rate resets are not the main problem in the subprime market.”27
Economists generally conclude that of more importance to foreclosures is falling house prices—the interest rate on a mortgage is largely irrelevant if the borrower can refinance or sell out of the mortgage. It is only when the borrower is unable to sell or refinance that the interest rate matters, thus adjustable rate or hybrid mortgages matter for foreclosures only in a falling real-estate market. Mortgages with positive equity tend to terminate in a prepayment of the mortgage (either as the result of a sale or refinance) whereas those with negative equity tend to terminate in foreclosure.28
The relationship between ARMs and foreclosures appears to have been a manifestation of the unique circumstances of the past several years rather than an inherent problem of ARMs. The percentage of ARMs in the market have been much higher at times in the past, yet they did not previously result in the surge of foreclosures that have resulted in the most recent environment. In fact, adjustable-rate mortgages are the norm in most of Europe and the rest of the world without the catastrophic events that have transpired in the United States in recent years.29 The primary difference, it appears,
26 Mayer, Pence & Sherlund, The Rise in Mortgage Defaults, supra note 17 at 41; Shane Sherlund, The Past, Present, and Future of Subprime Mortgages, Federal Reserve Board (Sept. 2008); Kristopher Gerardi, Adam Hale Shapiro & Paul S. Willen, Subprime Outcomes: Risky Mortgages, Homeownership Experiences, and Foreclosures, Federal Reserve Bank of Boston Working Paper No. 07-15. Mayer, Pence, and Sherlund find a dramatic rise in “early payment defaults” well before any interest rate adjustment takes place.
27 Christopher L. Foote, Kristopher Gerardi, Lorenz Goette, and Paul S. Willen, Subprime Facts: What (We Think) We Know about the Subprime Crisis and What We Don’t, FED. RES. BANK OF BOSTON PUBLIC POLICY DISCUSSION PAPERS 2 (May 30, 2008). Other studies have confirmed this conclusion about the limited role of interest-rate resets in driving increased foreclosures when compared to falling house prices and deterioriating underwriting standards. See Patrick Bajari, Chenghuan Sean Chu & Minjung Park, An Empirical Model of Subprime Mortgage Default from 2000 to 2007, NBER WORKING PAPER 14625 (Dec. 2008) (finding that interest rate resets play a positive, but relatively minor role, in defaults).
28 Anthony Pennington-Cross, The Duration of Foreclosures in the Subprime Mortgage Market: A Competing Risks Model with Mixing 4-5 (Fed. Reserve Bank of St. Louis, Working Paper No. 2006-027A, 2006).
29 Richard K. Green & Susan M. Wachter, The American Mortgage in Historical and International Context, 19 J. ECON. PERSP., Fall 2005, at 93, 107–08 (2005). Most other countries also have shorter mortgage maturity payments combined with a final balloon payment in contrast to the 30-year fixed-rate self-amortizing mortgage that is standard in the United States.
is that in recent cases, the interest rates on ARMs were pushed artificially and unsustainably low, thus the eventual interest-rate reset resulted in the interest rate on ARMs rising back to the level of FRMs, rather than FRMs falling to the level of ARMs (as was generally the case in the past). It appears that it is only when ARMs are combined with a monetary policy that pushed short-term interest rates to unsustainably low rates that ARMs became a problem.
Negative Home Equity
The decision to maintain homeownership or default and allow foreclosure can be modeled as a financial option. Where the option is “in the money” (i.e., the home is worth more than the amount owed) the homeowner can treat the house as a “call” option—if the homeowner is unable or unwilling to make her monthly payments (perhaps because she is moving) then she can either sell the home or refinance it and pay off the underlying mortgage. Thus, the option to allow foreclosure is of low value to the homeowner in a rising market because the homeowner can instead sell or refinance the house and pocket the equity. But where the house has negative equity (often referred to as “under water” or “upside down”), then the consumer has a put option—either she can continue to pay the mortgage and retain ownership or exercise the “option” to default and allow the lender to foreclose. If this option rises in value or becomes less expensive to exercise, homeowners will become more likely to exercise it.
Under the option theory of foreclosure, therefore, the decision to allow default is essentially a voluntary and rational response to the incentives created by the change in value of the asset—the borrower could continue to service the loan but chooses not to.
Default and foreclosure result because the borrower strategically chooses the option of foreclosure over the option of continued payment of the loan. Empirical studies traditionally have tended to support the option theory of foreclosure.30 For instance, even though interest rates generally rise uniformly across the country, the foreclosure rate is lower for residential real estate where price appreciation has been higher.31 This suggests that in deciding whether to default the primary consideration by homeowners is the amount of equity that they have accrued in their property (which might be lost in the event of a foreclosure) rather than “payment shock” resulting from an unexpected rise in interest rates. Similarly, those who have drawn against accumulated home equity through home-equity loans or junior liens exhibit a greater propensity to default than those who have retained their equity. 32
Falling real-estate prices helps to explain the rising foreclosure rate. There is a very close relationship between the timing of the nationwide drop in housing prices and the rise in the foreclosure rate. This striking relationship can be seen in figure 12 below and seems to lend support to this option theory:
Figure 12
30 See Kerry D. Vandell, How Ruthless Is Mortgage Default? A Review and Synthesis of the Evidence, 6 J. HOUSING RES. 245 (1995); James B. Kau & Donald C. Keenan, An Overview of the Option-Theoretic Pricing of Mortgages, 6 J. HOUSING RES. 217 (1995); Patric H. Hendershott & Robert Van Order, Pricing Mortgages: An Interpretation of the Models and Results, 1 J. FIN. SERVICES RES. 19 (1987).
31 Mark Doms, Frederick Furlong & John Krainer, House Prices and Subprime Mortgaged Delinquencies 1–2 (FRBSF ECON. LETTER NO. 2007-14, 2007); Brent W. Ambrose, Charles A. Capone, Jr. & Yongheng Deng, Optimal Put Exercise: An Empirical Examination of Conditions for Mortgage Foreclosure, 23 J. REAL EST. FIN. & ECON. 213, 218 (2001) (higher default rates where home price appreciation slower); Kristopher Gerardi, Adam Hale Shapiro & Paul S. Willen, Subprime Outcomes: Risky Mortgages, Homeownership Experiences, and Foreclosures 2–3 (Fed. Res. Bank of Boston, Working Paper No. 07-15, 2008), available at http://www.bos.frb.org/economic/wp/wp2007/wp0715.pdf (concluding that dramatic rise in Massachusetts foreclosures in 2006-07 resulted from decline in house prices beginning in summer 2005); Ellen Schloemer, Wei Li, Keith Ernst & Kathleen Keest, Losing Ground: Foreclosures in the Subprime Market and Their Cost to Homeowners, CRL RES. REPORTS, (Ctr. for Responsible Lending, Durham, N.C.), Dec. 2006, at 1, 13.
32 See Michael LaCour-Little, Equity Dilution: An Alternative Perspective on Mortgage Default, 32 REAL ESTATE ECON. 359, 369 (2004).
Source: OFHEO Home Price Index (Sales) and Mortgage Bankers Association
Another practice that increased the incentives for strategic default was the growth of lending products that reduced certain homeowners’ equity investments in their loans, such as low or no-downpayment loans, as well as certain lending products like interest-only mortgages that meant that consumers accumulated no equity through their monthly payments.33 Gerardi, et al., find that the most dramatic change in the subprime lending market over the course of the housing boom was the dramatic growth in the number of high loan-to-value (LTV) ratio loans in the latter stages of the boom. 34 While housing prices were rising, these loans performed exceedingly well, as borrowers could either sell or refinance if they were unable or unwilling to make payments. When housing prices turned down, however, high-LTV loans quickly went underwater, leaving homeowners with strong incentives to permit foreclosure.
33 This latter factor may be of minimal importance, however, as 30 year conventional fixed mortgages provide for the payment of a much greater ratio of interest to principal at the beginning of the loan repayment term, thus equity accumulation is minimal for many years.
34 Kristopher Gerardi, et al., Making Sense of the Subprime Crisis, BROOKINGS PAPERS ON ECONOMIC ACTIVITY (Douglas W. Elmendorf, N. Gregory Mankiw, and Lawrence Summers eds, Fall 2008) at 9-10.
The positive experience with unconventional products in the early stage of the boom, however, encouraged lenders to increasingly combine various unconventional terms, a practice known as risk layering. As housing prices started to decline, the combination of more than one unconventional term have proven particularly problematic and likely to trigger foreclosure, with the interaction between different risk-layering terms giving rise to a geometric increase in the propensity to default rather than being merely additive.
One technique that led to this result was the growing popularity of “piggyback loans.” With a piggyback loan, the borrower simultaneously takes out a first mortgage and a junior-lien (piggyback) loan. The piggyback loan finances the portion of the purchase price that is not being financed by the first mortgage.35 Piggyback loans often were taken out so that the first-lien mortgage can meet the conforming loan size limits.36 Virtually nonexistent in 2000, by 2006 about 22 percent of mortgage loans for owner-occupied houses also had piggyback second-lien mortgages attached. 37
As noted above, a primary factor driving foreclosure is the presence or absence of equity in the property. Thus, loans with little or no down payments (such as those with high LTV or mortgages combined with piggyback loans) offer an unusually powerful incentive to default if property values fall.38 Lower downpayments are correlated with
35 Id.
36 Id. at A85.
37 Robert B. Avery, Kenneth P. Brevoort & Glenn B. Canner, The 2006 HMDA Data, 93 FED. RESERVE BULLETIN A 73, at A85; see also EDWARD VINCENT MURPHY, CONGRESSIONAL RESEARCH SERVICE, ALTERNATIVE MORTGAGES: CAUSES AND POLICY IMPLICATIONS OF TROUBLED MORTGAGE RESETS IN THE SUBPRIME AND ALT-A MARKETS (2008), at 5. The apparent absence of piggyback loans before 2000, however, may overstate the distinction. Although the purchase-money lender did not traditionally provide a piggyback home equity loan, for many decades consumers who could not come up with a full 20% downpayment might borrow the needed amount from a consumer finance company (presumably on an unsecured basis). See PAUL MUOLO & MATHEW PADILLA, CHAIN OF BLAME: HOW WALL STREET CAUSED THE MORTGAGE AND CREDIT CRISIS 37 (2008).
38 In fact, LaCour-Little, et al., conclude that negative equity for homes in foreclosure are more often the result of post-purchase cash-out refinancing or home equity loans are more responsible for the presence of negative equity than housing price declines. See Michael LaCour-Little, Eric Rosenblatt & Vincent Yao, Do Borrowers Facing Foreclosure Have Negative Equity? 20 (July 11, 2008) (working paper, available at http://ssrn.com/abstract=1162398).
higher rates of default39 and lower LTV ratios are reflected in lower risk premiums in interest rates.40 One study found that conventional mortgages with loan-to-value ratios at origination of 91–95 percent were twice as likely to default as loans with LTVs of 81–90 percent and five times more likely to default than those with LTVs of 71–80 percent.41
A related factor in the general reduction in homeowner equity cushion was the growing use of cash-out refinancing in recent years, especially in the later stages of the housing boom. The United States is almost unique in the world in adopting a general practice of permitting an almost unlimited right of mortgage prepayment and thus the ability to refinance at almost any time. 42 Most commercial loans and subprime mortgages, by contrast, prohibit or penalize prepayment for certain periods of time at the outset of the mortgage.
From 2003 to 2006, the percentage of refinances that involved cash-out doubled from under 40 percent to over 80 percent,43 and among subprime refinanced loans in the 2006–2007 period around 90 percent involved some cash out.44 The result of this cash-out activity was similar to that of piggyback home-equity loans, namely to strip out borrower’s equity cushions, thereby making it more likely that a subsequent fall in the value of the home would bring the mortgage into negative equity and bring about circumstances for a default and foreclosure.
Anecdotal reports in the current market also report a growing number of “mortgage walkers” who are exercising their “put” option to voluntarily surrender their
39 See id.
40 See Gregory Elliehausen, Michael E. Staten & Jevgenijs Steinbuks, The Effect of Prepayment Penalties on the Pricing of Subprime Mortgages, 60 J. ECON. & BUS. 33, 34 (2008) (reviewing studies)
41 Robert B. Avery, Raphael W. Bostic, Paul S. Calem & Glenn B. Canner, Credit Risk, Credit Scoring, and the Performance of Home Mortgages, 82 FED. RES. BULL. 621, 624 (1996).
42 Green & Wachter, at 100-01.
43 Luci Ellis, The Housing Meltdown: Why did it happen in the United States? Bank for International Settlements Working Paper No. 259 (2008), at 22 and Fig. 9.
44 C J Mayer & Karen Pence, Subprime Mortgages: What, Where, and To Whom, NBER Working Paper no. 14083.
home to the lender, a practice known as “jingle mail” after the practice of the borrower mailing her keys to the lender and surrendering the house.45 As house prices fall, mortgage walking has begun to spread beyond the subprime market. Kenneth Lewis of Bank of America recently observed that while in the past, consumers would default only after falling behind on car payments, credit cards, and other debts, there has been a general change in social norms regarding mortgage default.46 Today, Bank of America reports a growing number of borrowers who are current on their credit cards but defaulting on their mortgages suggesting that “[a]t least a few cash-strapped borrowers now believe bailing out on a house in one of the easier ways to get their finances back under control.”47 This temptation is especially strong for those homeowners who put little or nothing down or borrowed against their home equity.
The incentives to “walk” are especially strong in those states with antideficiency laws that limit creditor’s remedies to foreclosure without the right to sue the borrower personally for the deficiency. In a study of the neighboring provinces of Alberta and British Columbia in Canada, Lawrence Jones found that “in a period of sizable house-price declines, the prohibition of deficiency judgments can increase the incidence of default by two or three times over a period of several years.”48 In fact, in Alberta (which had an antideficiency law) 74 percent of those who deliberately defaulted had negative equity; in British Columbia (which permitted deficiency suits) only one homeowner defaulted with negative book equity.49 Moreover, if it is the case (as it appears to be) that the propensity for default and foreclosure is a function in part of state laws regarding the
45 Nicole Gelinas, The Rise of the Mortgage “Walkers,” WALL ST. J., Feb. 8, 2008, at A17.
46 George Anders, Now, Even Borrowers With Good Credit Pose Risks, WALL ST. J., Dec. 19, 2007, at A2.
47 Id.
48 Lawrence D. Jones, Deficiency Judgments and the Exercise of the Default Option in Home Mortgage Loans, 36 J. L. & ECON. 115, 135 (1993).
49 Id. at 128–29. Jones states that the one defaulter in British Columbia reportedly left the country. Id. at 129.
collection of deficiency judgments and judicial foreclosure actions and that lenders have already priced that risk ex ante in the loan, this raises questions about the propriety as a matter of equity and efficiency of governmental “bail outs” for distressed borrowers and lenders.
Even where the laws do not mandate that mortgages are nonrecourse, lenders have exhibited willingness to voluntarily waive an action for deficiency.50 Although laws vary among states, over a dozen states have some type of antideficiency laws that limit creditors to seizure of the property in the event of default, with no right of recourse against the borrower personally. Many of the states with antideficiency laws, such as California and Arizona,51 are also among the states with the highest foreclosure rates. Other high-foreclosure states, such as Nevada and Colorado, have laws that limit the amount that lenders can recover from borrowers but do not bar deficiency judgments completely. Antideficiency laws also appear to affect homeowners’ incentives to maintain their property—homeowners in states that have antideficiency laws may be less willing to invest in maintenance and improving their homes. 52
What Happened?
The underlying cause of the housing boom and bust, and the subsequent rise in foreclosures, thus seems to be largely explained by two fundamental factors. First, artificially low short-term interest rates, relative to long-term interest rates that provided
50 There is also evidence that subprime lenders tend to foreclose much more slowly. See Dennis R. Capozza & Thomas A. Thomson, Subprime Transitions: Lingering or Malingering in Default?, 33 J. REAL ESTATE FIN. ECON. 241, 257 (2006).
51 See Michael T. Madison, Jeffry R. Dwyer & Steven W. Bender, 2 THE LAW OF REAL ESTATE FINANCING §12:69 (Dec. 2007), available in Westlaw REFINLAW § 12:69.
52 John Harding, Thomas J. Micelli & C.F. Sirmans, Deficiency Judgments and Borrower Maintenance:
Theory and Evidence, 9 J. HOUSING ECON. 267, 271 (2000); see also John Harding, Thomas J. Micelli & C.F. Sirmans, Do Owners
Take Better Care of Their Housing Than Renters?, 28 REAL ESTATE ECON. 663, 669–70 (2000).
incentives for consumers to switch from fixed-rate to adjustable-rate mortgages, allowed borrowers to qualify for larger mortgages than would otherwise be the case, and resulted in trouble for some borrowers who were unable to make their payments when short-term interest rates rose. This household financial distress was exacerbated as the economy dipped into recession, piling traditional causes of foreclosures (such as job loss), on top of this distress caused by interest-rate adjustments. Second, a rapid, severe, and sustained fall in house prices provided many consumers with an incentive to exercise their default option and to allow foreclosure to go forward on their homes. This was exacerbated by a variety of factors, including new mortgage lending practices that led to little or no equity for many homeowners and certain states’ laws that provide great protection for borrowers in the event of a foreclosure, such as antideficiency or non-recourse laws.
Basic economic theory, therefore, seems to explain most of the underlying dynamics of rising foreclosure rates and bankruptcy filing rates by explaining the basic decision-making of homeowners. On the other hand, this analysis does not address the more fundamental questions, which are: Why did the housing price bubble develop as it did, why did foreclosures rise so dramatically as house prices fell, and why did Wall Street and the banking industry so badly misjudge the financial problems?
Conventional wisdom about the mortgage crisis provides several hypotheses, such as the rise of securitization of mortgage debt or the role of government policies that provided incentives for overinvestment in housing and reckless credit expansion to poor risks. Although the causal mechanisms differ, these hypotheses share a common similarity that they both interpret the fundamental cause of the mortgage crisis as increased reckless lending to risky borrowers or loan terms that were excessively risky.
On closer examination, however, while these factors may have exacerbated the underlying mortgage crisis, it is doubtful that they can explain the fundamental nature of the housing boom and bust.
The Two Phases of the Mortgage Crisis
Before looking in some detail about the causes of the housing boom and bust, it is important to consider an often-overlooked element of the crisis; namely, that there are really two phases of the housing boom and mortgage crisis, one lasting from about 2001– 2004 and a second running from about 2005–2007. While loan performance during the later phase (2005–2007) has been disastrous, loan performance during the earlier phase was largely non-problematic, even those loans that contained particular terms that have subsequently drawn criticism, such as hybrid mortgages, low-documentation loans, and low-downpayment loans. Indeed, as will be developed more below, it is likely that the disastrous collapse of the housing and mortgage markets came about precisely because the strong performance of non-traditional loans in the first phase of the credit expansion encouraged more aggressive loans in the second phase of the boom.
Empirical data provide a picture of the differences between these two periods of the housing boom. A few key differences between the two phases of the housing boom emerge when examining the data. The most important factor—to put the matter quite simply—is that house prices were rising in many parts of the country in the early stages of the mortgage boom. Second, the structure of the loans in the later phase of the housing boom were substantially different from the first phase. In particular, in the first phase many subprime borrowers were at least as risky as borrowers in the second phase;
moreover, many of the loans issued in the first phase included many of the features that were later criticized, such as low-documentation, low-downpayment, or interest-only loans. The difference in the second phase, however, was that loans increasingly combined these various features, a practice known as “risk layering.” Of particular concern was the increasing use of no-downpayment loans, often combined with interest-only or negative amortization features.
House Prices and the Foreclosure Crisis
First, as suggested above, a primary distinction between the first period of the housing boom (2001–2004) and the second (2005–-2007) was that house prices were appreciating rapidly in many areas during the initial period but falling during the second period. This meant that homeowners during the first period were accumulating equity in their homes, enabling them to either sell or refinance if necessary. Moreover, this appreciation in house prices meant that a steeper drop in house prices was necessary before they were in a negative equity position and thereby tempted to default, an equity buffer that was reinforced by the higher likelihood of having made a downpayment and lower likelihood that they would have engaged in a cash-out refinance that would have further depleted their equity cushion.
Refinances in the earlier stages of the housing boom also were more likely to have been triggered by falling interest rates and thus cash-out refinances were less common then at later times.53 But refinancings continued even as interest rates began to rise beginning in 2005. But the nature of these refinancings changed from the earlier wave— refinancing in the 2005–2007 period were much more likely to be cash-out refinancing,
53 Ellis, supra note 43.
suggesting that these homeowners were refinancing for different purposes but also that they were depleting equity at a greater rate than earlier borrowers, leaving them with a much higher combined loan-to-value ratio than earlier refinancers.
But this raises a question—did the causal relationship in the early stage of the boom run from house prices to expansive lending practices or from expansive lending to higher house prices? This is actually quite a complex question and available data suggests that both elements are present. In general, however, it appears that in the earlier stages of the boom, house-price appreciation was caused primarily by underlying economic factors; in turn, this strong house-price appreciation led to an extraordinarily strong performance record for all mortgages made during this time, including novel subprime mortgages. But, in turn, this early record of success resulting from rising house prices in the earlier stages of the boom fueled more aggressive and risky lending in the later stages of the boom.
As suggested above, the early phase of the boom seems to have been fueled by a variety of macroeconomic factors that led to a run-up in home prices in many markets. Most importantly, extremely low short-term interest rates allowed many borrowers to “stretch” to pay more for homes than otherwise would have been the case otherwise. Thus, consumers could afford “more” home than they might otherwise. As noted above, the HUD housing affordability index was at record highs during this period—thus, even as housing prices were rising, they were more than offset by record-low interest rates.
In addition, a variety of other factors raised the return to home ownership and led to increased house prices. Most notably, in 1997 the tax code was amended to permit homeowners to pay no tax on any capital gains of up to $500,000 upon the sale of their
home. This led to a strong tax code preference for investments in housing relative to other forms of investment and saving leading to household overinvestment in real estate.54 By contrast, ordinary saving is “double-taxed” as income when first earned as well as when interest is paid. And financial investments do not have this preferential capital gains tax treatment. Moreover, the bursting of the dot-com bubble and the struggles of the stock market in the immediate aftermath may have persuaded many consumers that homeownership was a more reliable form of wealth accumulation than financial assets.
To understand the difference between the earlier and later phases of the housing boom and the different performance of the mortgages issued during those two vintages, it is necessary to understand the nature of housing markets in the United States. There are effectively three different types of housing markets and differences among these markets help to explain the different performance patterns of mortgages from these two different eras. 55 Fundamentally, these housing markets are differentiated by underlying supply and demand dynamics.
The first type of housing market is those markets with traditionally cyclical markets that experience high but essentially predictable volatility, such as New York, Washington, DC, and Boston. Because of zoning and other constraints on construction of new homes, these markets have a highly inelastic supply of housing supply. Thus, when housing demand rises or falls for exogenous reasons, prices fluctuate widely in these markets. As explained above, extremely low interest rates and other governmental policies dramatically increased demand for home ownership. Therefore, prices rose
54 Vernon Smith, The Clinton Housing Bubble, WALL ST. J., Dec. 18, 2007, at A20.
55 See Christopher Mayer & R. Glenn Hubbard, House Prices, Interest Rates and the Mortgage Market Meltdown (working paper, Columbia Business School).
dramatically in these markets, a price appreciation that fundamentally reflected supply and demand dynamics, although exaggerated by various other factors and the artificial nature of the demand boost. As a result, although prices have fallen in these markets, foreclosures have not risen as much, as the initial “bubble” was not as much a bubble as an exaggerated response to fundamental supply and demand dynamics and as homeowners expect for prices to rebound in the near future.
A second type of market is the “steady” markets that underwent a steady appreciation in home prices over the past decade, with prices driven largely by underlying supply and demand dynamics.56 Steady markets that have relatively modest regulations and restraints on expansion of housing supply to meet demand growth and thus have a relatively elastic housing supply. These markets, therefore, tend to respond to increases in demand by a relatively rapid increase in supply. Thus, these markets did not experience the same sort of house price bubble as many other markets—nor are they experiencing the subsequent house-price collapse and the resulting foreclosure crisis. These markets include cities such as Atlanta, Charlotte, Chicago, Denver, and Detroit.
But during the most recent housing boom a third type of market materialized— cities with modest restrictions on building new supply, and in fact experienced a dramatic growth in supply, but which nonetheless saw dramatic increases in home prices. These markets can be characterized as “late-boom” markets and include cities such as Las Vegas, Miami, Phoenix, and Tampa. 57 These markets began the housing boom resembling the second type of cities—demand growth manifested itself in rapid increase
56 Id.
57 Presumably this list would also include areas like the distant suburbs of Northern Virginia and California that saw rapid construction of new bedroom communities outside the traditional metropolitan areas and which experienced a very rapid boom and bust price cycle culminating in widespread foreclosures. In particular, many of these markets sprung up outside traditional cyclical markets, thereby adding a late-boom element to a traditional cyclical market (and perhaps exacerbating the price swings in both).
in supply, rather than a rapid increase in prices. But toward the end of the boom, these markets also saw a dramatic run-up in prices as well. Unlike the traditionally volatile markets, the price appreciation in these markets occurred toward the end of the boom, rather than the beginning, even though this was the period when interest rates were rising rather than falling. Moreover, this rapid price appreciation appears to lack plausible grounding in underlying economic logic—prices were rising, even as both supply and interest rates were rising as well.
The results have been catastrophic—the artificially high prices have collapsed, as prices have come to reflect the supply and demand dynamics of the massive expansion of new housing that was constructed during the boom. Prices have fallen toward their equilibrium levels, and given the huge expansion of housing supply in those markets in recent years, there is little expectation of a major price recovery in the near future. Moreover, slowing of the economy has also slowed population immigration into these markets (both legal and illegal). Thus, these markets have experienced dramatic drops in home prices with little expectation of price recovery in the near future. Foreclosures skyrocketed in these markets as home prices plunged.
The rapid house price appreciation in the “late-boom” cities closely matches the timing of the second stage of the mortgage crisis. In fact, some commentators have suggested that rather than the spread of subprime lending fueling the house-price boom in many markets, the house-price boom fueled a rise in subprime lending as buyers rushed in to gain a piece of the action.58 Such speculative motivation would be consistent with the high penetration of adjustable-rate mortgages in the subprime market, which would
58 Mayer & Pence, for instance, find that areas with high house price appreciation saw a rise in the following year in subprime mortgage originations. See Christopher J. Mayer & Karen M. Pence, Subprime Mortgages: What, Where, and to Whom? NBER Working Paper W14083 (2008) available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1149330.
generally be preferred by less risk-averse borrowers and speculators with a short-term time horizon.59 Finally, the possible presence of substantial speculation in these markets is consistent with overheated activity in the real estate market in these cities being in properties such as new condominiums and new suburban homes, that are standardized products (in terms of style, quality, and neighborhood quality) that are amenable to rapid flipping.
Although the story is oversimplified, a general picture begins to emerge. The first phase of the housing price boom was driven principally by traditionally volatile markets responding to the various incentives created by low interest rates and other policies that promoted homeownership and home investment. During this phase of the boom, much mortgage activity was to refinance for lower interest rates and many home equity loans were to fund home improvements (often because consumers wanted to move to larger homes but price appreciation precluded them from doing so, so they responded to the provided incentives by increasing the value of their own home).60 Price appreciation in these traditionally volatile markets roughly reflected some degree of underlying supply and demand dynamics. Increased demand in other markets was channeled into new home construction. Families in those markets responding to the same dynamics still had a tendency to increase their investments in residential real estate and the subprime market was born. Price appreciation relieved homeowners of the incentives to exercise their option for foreclosure. Those who were overextended or suffered economic distress could sell or refinance their home in a rising market, rather than exercising foreclosure.
59 Stan Liebowitz, Op-Ed, The Real Scandal, N.Y. POST, Feb. 5, 2008.
60 Alan Greenspan & James Kennedy, Sources and Uses of Equity Extracted from Homes (Div. of Research & Statistics & Monetary Affairs, Fed. Reserve Bd., Fin. & Econ. Discussion Series, Working Paper No. 2007-20, 2007).
The timing of the excesses of the second half of the mortgage boom also coincides with the artificial price bubble of the late-boom housing markets. Lenders erroneously extrapolated the performance of higher-risk mortgage from the first phase of the bubble to the dissimilar markets and borrowers of the second stage of the bubble. Lenders underestimated the impact of risk layering—and especially the dangers of low-downpayment, interest-only, and other mortgages that led to heightened borrower incentives to default if home prices fell. And fell they did—as was inevitable in light of the underlying supply and demand dynamics that underlay these markets.
Risk Layering
A second feature that distinguishes the late boom from the early boom is the increased presence of risk layering in the later phases of the boom. Risk layering is the practice of combining more than one risky term together in a given mortgage.
Many of the terms in loans that have drawn the most criticism, such as hybrid mortgages or low-documentation loans, do not appear to be excessively risky—when they appear in isolation. During the early part of the boom, there was little evidence to suggest that hybrid, low-documentation, low-downpayment, and other exotic mortgage terms were excessively risky, in large part due to the rising housing market, but also because certain of these terms, in isolation, may simply not be that risky.
Consider, for instance, much-maligned “low documentation” loans, sometimes referred to as “liar’s loans.” Low-documentation loans forego many of the formalities associated with a typical loan, such as an appraisal, detailed income and assets review, and a detailed loan application, in favor of a much simpler process based on a credit score
and simplified review process. Although low-documentation loans seem inherently risky, they may be completely appropriate for refinance loans based on an established track record of successfully making mortgage payments, a regular job, accumulated home equity, and a house that has had home-price appreciation. They seem less sensible for riskier purchase-money borrowers with riskier property and no equity. Even then, low-documentation loans may be sound if the borrower has sufficient equity in the property at the outset, such as an especially low loan-to-value ratio. In fact, during the first phase of the housing boom, low-documentation subprime loans performed just as well as full-documentation loans. 61
In fact, it appears that non-traditional mortgages in the early phase of the boom also tended to have offsetting features that lowered the risk, such as a lower loan-to-value ratio than normal. The fact that apparently risky terms were generally confined to appropriate contexts or were offset by alternative risk-reducing features of the loan, along with the strong appreciation in house prices during this period, may account for the surprisingly strong performance of loans with these terms in the early stage of the boom.
In the later stages of the boom, however, some of this restraint was abandoned. Rather than offsetting riskier terms with other adjustments, lenders increasingly engage in risk layering of loan terms, perhaps bolstered by the early sound performance of loans with these terms present in isolation. Because low-documentation and low-downpayment mortgages in isolation were found to have modest and manageable risk associated with them, it might have been predicted that combining the two elements would increase risk, but only modestly so. Instead, it appears that combining two such terms increased the
61 Kristopher Gerardi, et al., supra note 34.
risk of the resulting product dramatically—a risk that became explosive when combined with plunging home prices.
Loans with high loan-to-value ratios combined with low documentation proved to be especially prone to default.62 This was exacerbated by a substantial increase in loan-to-value ratios in the later stages of the boom.63 The median combined LTV for subprime purchase loans rose from 90 percent in 2003 to 100 percent in 2005, “implying that in the final years of the mortgage boom more than half of the borrowers with subprime mortgages put no money down when purchasing their homes.”64 Piggyback loans also became more common during this time, 65 although many of these loans were “silent second” that were not disclosed to the originator of the first mortgage.66 The presence of these silent seconds increased the risk of default and foreclosure to the senior lender, but without the senior lender’s knowledge (and so without adjusting the risk premium).67
Loans with minimal downpayments, cash-out refinancing, and aggressive use of home equity loans, became especially prone to default as the housing bubble has burst. When home prices fall, these loans quickly turn into negative equity, providing borrowers with a strong incentive to default. Interest-only and negative amortization loans create similar incentives by minimizing equity accumulation, but loans with these features appear to have been relatively rare in the subprime market although quite common in the prime mortgage market.68 Although, subprime refinance borrowers had on average lower
62 Id. at 5.
63 Id. at 1.
64 Mayer, Pence & Sherlund, The Rise in Mortgage Defaults, supra note 17, at 31.
65Id.; see also Gerardi et al. supra note 34 at 10 (noting dramatic rise in number of mortgages with second liens).
66 Between 1999 and 2006 the percentage of subprime loans with silent seconds rose from 1 percent to over 25 percent and for Alt-A loans the rates rose from 1 percent to nearly 40 percent of securitized Alt-A mortgages.
67 Empirical evidence has found that holding combined LTV constant, borrowers who reached that LTV through a home equity loan are more likely to default than an identically-leveraged borrower who reached that LTV through a purchase or refinance loan, see Gerardi et. al., supra note 34.
68 Mayer, Pence & Sherlund, The Rise in Mortgage Defaults, supra note 17. Mayer, et al., find that 40 percent of Alt-A mortgages
FICO scores than purchase-money borrowers (19–35 points lower), average credit scores were relatively constant over the duration of the housing boom.69 This suggests that the risk of negative equity is more important than the risk profile of the borrower in predicting whether the mortgage eventually would default.
Thus, the fundamental problem on this score may not have been excess “greed” or recklessness of sophisticated lenders making inherently crazy loans. Instead, if there was a flaw, it was likely that lenders extrapolated too aggressively from too small of a sample of loans, especially in the subprime market. Had the initial ventures into the subprime market turned out to have been a failure rather than a success, it is likely that the market would have been nipped in the bud and would never have expanded. The pattern that thus emerges is a somewhat surprising one—the seeds of the mortgage crisis were not grounded in inherently risky lending to unusually risky borrowers.70 Instead, it appears that lenders simply underestimated the likelihood of an extended and dramatic home price collapse like that which actually occurred. 71
This over-optimism by lenders was mirrored by over-optimism on the part of many buyers that home prices would increase without interruption. Many otherwise-homeowners essentially became de facto real estate market speculators through interest-only and low-downpayment mortgages that resulted in borrowers with minimal equity positions that seemed predicated on the assumption that housing prices would rise.72 Lenders underestimated that likelihood that so many borrowers would end up in such
had interest-only features, compared to 10 percent of subprime; 30 percent of Alt-A mortgages permitted negative amortization, subprime loans did not have these features.
69 Id., at 17.
70 As Gerardi et al., supra note 34, put it, “These results are consistent with the view that a factor other than underwriting changes was primarily responsible for the increase in mortgage defaults.”
71 Gerardi et al., supra note 34, at 36.
72 See Monika Piazzesi & Martin Schneider, Momentum Traders in the Housing Market: Survey Evidence and a Search Model, NBER Working Paper 14669 (Jan. 2009) (noting rise of “momentum traders” in the housing market in the later stages of the boom).
severe negative equity positions that would provide such strong incentives for so many borrowers to allow foreclosure or simply walk away from their mortgages. This dramatic increase in the default rate may have been either because homeowners as a group became unusually responsive to home price declines or because home prices declined much more than expected and homeowners responded as would have been predicted in such an unlikely and catastrophic scenario—or both.
Securitization
Many commentators have charged that the rapid spread of securitization of mortgage debt, especially subprime mortgages, explains the underlying mortgage crisis. The basic story is that over time, securitization of mortgage debt, especially subprime mortgages, rose dramatically. This is certainly true.73 From 2000–2005, for instance, the volume of subprime mortgages securitized by Wall Street rose almost tenfold, from about $56 billion annually to $508 billion and the percentage of subprime loans that were securitized rose from about 50 percent to over 80 percent during that same time frame, a time period that correlates with the expansion of the subprime market.
The link between securitization and risky mortgage underwriting, it is argued, is a
chain of agency cost relationships generated by securitization. 74 In particular,
securitization is said to have given rise to an “originate to distribute” model of mortgage lending, where the originating lender does not bear the risk that the loan will fail. Thus, mortgage brokers originate the loan, but resell it to the wholesale supplier of money,
73 Kathleen C. Engel & Patricia A. McCoy, Turning a Blind Eye: Wall Street Finance of Predatory Lending, 75 FORDHAM L. REV. 2039, 2045 (2007). Gary B. Gorton, The Subprime Panic 6 tbl.4 (Nat’l Bureau of Econ Re-search, Working Paper 14398 Oct. 2008), available at http://www.nber.org/papers/w14398.
74 Engel & McCoy, Christopher L. Peterson, Predatory Structured Finance, 28 CARDOZO L. REV. 2185, 2200–06 (2007)
which then in turn bundles the loans, subdivides them into tranches, and resells those bundles to investors. It is argued that this creates a series of agency relationships, all of whom have incentives to maximize loan volume and ignore heightened risk and deteriorating underwriting standards so long as they can pass on these loans (and their risk) to subsequent holders.
Plainly, there seems to be a correlation between the rise of securitization and the subprime lending boom and housing price bubble. But it is doubtful that the growth of securitization can provide a convincing causal explanation. First, securitization has been a well-established model of lending for years in other consumer credit markets (such as credit cards, auto loans, and prime mortgages). Scholars also have noted that other countries have seen a dramatic rise in home prices and a deterioration of underwriting standards, most notably England, even though securitization remains nonexistent. 75 Second, many of those who either sold or bought these securities were highly-sophisticated investors such as Bear Stearns, Merrill Lynch, or Citibank, who were likely well aware of the risks and protected against them. Furthermore, the investment banks that supposedly orchestrated this Ponzi scheme are now either bankrupt or have been merged into other financial institutions as a result of investing in securities backed by subprime loans, a reality that is difficult to square with the purported incentives of the originate-to-distribute model.
Other considerations contribute to skepticism about the role of securitization in fueling the mortgage crisis. For instance, as noted above, the subprime mortgage boom appears to have two distinct phases; securitization, however, grew steadily throughout both periods. Thus, the incentives created by securitization were constant during this
75 Ellis, supra note 43, at 6-7.
period, suggesting that some factor other than securitization intervened between the first and second periods to lead to the dramatically worse performance of the mortgages originated in the second period. Thus, although the role of securitization in creating agency costs is theoretically possible as a major cause of the subprime mess, it seems doubtful that the incentives created by securitization was an important contributor to the mortgage crisis—although, of course, simple errors and miscalculations are possible for reasons unrelated to the incentives created by securitization.76
Conclusion
By observing consumer behaviors and homeowners’ responses to various incentives during the mortgage crisis, observers can glean a picture of some of the primary causes of the meltdown. Specifically, low short-term interest rates encouraged consumers to switch from fixed-rate to adjustable-rate mortgages or hybrid mortgages and also motivated purchases of real estate as a speculative activity. Keeping in mind the structure of this borrowing and the little equity homeowners held in new properties, along with various the reasons for consumer foreclosure, one can observe that a rapid decline in housing prices presented a strong incentive for property owners to exercise their foreclosure options.
This analysis also points out that there was likely no sharp decline in individuals’ abilities to afford their homes, nor is an inability of families to keep up with their mortgage payments the culprit behind the current mess. Instead, the rise in foreclosures that has had adverse affects throughout the credit markets and he economy has likely
76 Gerardi, et al.,supra note 34, also doubt the importance of securitization in explaining the subprime boom and bust suggesting that there is no inherent link between them.
resulted from consumers’ rational responses to certain incentives that arose during the two phases of the crisis. Certainly, the full implications of the events unfolding have yet to be realized, however this general understanding of the mortgage meltdown can help pinpoint some of the fundamental causes and shed more light on the future of the crisis.
LAPPEENRANTA UNIVERSITY OF TECHNOLOGY Department of Industrial Engineering and Management
MASTER’S THESIS
MANAGING TECHNOLOGIES IN RESEARCH ORGANIZATION:
FRAMEWORK FOR RESEARCH SURPLUS PORTFOLIO
The subject of the thesis has been approved by the council of the Department of Industrial Engineering and Management in the meeting of 4 October 2006 in Lappeenranta University of Technology.
Supervisor: Professor Marko Torkkeli
Instructor: M. Sc (Econ.) Pekka Salmi
Kouvola, October 13, 2006
Sari Viskari
Torikatu 2 A 20
45100 Kouvola
Tel. +358 40 7317673
ACKNOWLEDGEMENTS
I would like to thank my supervisor Professor Marko Torkkeli and instructor Pekka Salmi from the Kouvola unit of Lappeenranta University of Technology. Many discussions with you and your constructive comments have been very helpful during the thesis work.
I would also like to thank Nokia, Nokia Research Center and TEKES (Finnish Funding Agency for Technology and Innovation) about Onions-project and enabling the thesis. I thank project members for collaboration, especially Project Manager Matti Karlsson and Jukka P. Saarinen, Head of Multimedia Laboratory in NRC.
Also, I thank my family and friends for the support. Special thanks to my friend, Jutta Jäntti, who have helped me to improve my English.
Finally, my greatest thanks go to my best friend, Pasi, who have supported me not only during the thesis work but also during the whole master’s studies. I would not be here without you.
Kouvola, October 13, 2006
Sari Viskari
ABSTRACT
Author: Sari Viskari
Title: Managing Technologies in Research Organization:
Framework for Research Surplus Portfolio
Department: Industrial Engineering and Management
Year: 2006 Place: Kouvola
Master’s Thesis. Lappeenranta University of Technology.
76 pages, 22 figures, 6 tables and 2 appendices.
Supervisor: Professor Marko Torkkeli
Keywords: Research Surplus Portfolio, portfolio management, non-core technologies,
open innovation
Hakusanat: tutkimusylijäämäportfolio, portfoliojohtaminen, avoin innovaatio
Open innovation approach and the effective use of innovations are becoming the essential parts of companies’ R&D processes. The purpose of the thesis is to create a framework for managing non-core technologies in more efficient way in the research organization.
In the thesis, the constructive concept of Research Surplus Portfolio (RSP) is constructed based on the literature review of intellectual capital management and portfolio management. In addition, tools and techniques for the evaluation of surplus technologies are identified.
The new portfolio for non-core technologies can be utilized as a searching engine, an idea bank, a communication tool or a market place for technologies. The important phases of the management process of RSP are documentation of the data of non-core technologies, the evaluation of them and maintaining and updating the system.
TIIVISTELMÄ
Tekijä: Sari Viskari
Työn nimi: Teknologioiden johtaminen tutkimusorganisaatiossa: Viitekehys tutkimusylijäämäportfoliolle
Osasto: Tuotantotalous
Vuosi: 2006 Paikka: Kouvola
Diplomityö. Lappeenrannan teknillinen yliopisto
76 sivua, 22 kuvaa, 6 taulukkoa ja 2 liitettä
Tarkastaja: professori Marko Torkkeli
Hakusanat: tutkimusylijäämäportfolio, portfoliojohtaminen, avoin innovaatio Keywords: Research Surplus Portfolio, portfolio management, non-core technologies, open innovation
Avoimesta innovaatiosta ja innovaatioiden tehokkaasta hyödyntämisestä on tulossa tärkeitä osia yritysten T&K-prosesseihin. Diplomityön tarkoituksena on luoda viitekehys
teknologioiden, jotka eivät kuulu yrityksen ydinliiketoimintaan, tehokkaampaan
hallinnointiin tutkimusorganisaatiossa.
Konstruktiivinen viitekehys on rakennettu pohjautuen aineettomien pääomien
johtamisen ja portfolion hallinnoinnin teorioihin. Lisäksi työssä määritellään työkaluja ja tekniikoita ylijäämäteknologioiden arviointiin.
Uutta ylijäämäteknologioiden portfoliota voidaan hyödyntää hakukoneena,
ideapankkina, kommunikaatiotyökaluna tai teknologioiden markkinapaikkana. Sen
johtaminen koostuu tietojen dokumentoinnista järjestelmään, teknologioiden arvioinnista ja portfolion päivityksestä ja ylläpidosta.
TABLE OF CONTENTS
1 INTRODUCTION 1
1.1 Overview 1
1.2 Objectives, Restrictions and Research Method 3
1.3 Structure 5
2 OPEN INNOVATION PARADIGM 7
2.1 Closed, Traditional Innovation Model 7
2.2 Changing Environment and Challenges for R&D 8
2.3 Open Innovation Model 10
3 INTELLECTUAL CAPITAL MANAGEMENT 13
3.1 Intellectual Capital and the Open Innovation 13
3.2 Concept of Intellectual Capital 14
3.3 Extracting Value from Intellectual Capital 16
3.3.1 Managing Intellectual Property 16
3.3.2 Managing Intellectual Assets 20
3.3.3 Managing Non-Core, Technology-Based Assets 22
3.3.4 Transformative Capacity 24
3.4 Managing Intellectual Capital 27
4 PORTFOLIO MANAGEMENT 29
4.1 Importance of the Portfolio Management 29
4.2 Portfolio Management Process 30
4.3 Methods for the Portfolio Management 32
4.3.1 Financial Methods 32
4.3.2 Strategy Related Methods 33
4.3.3 Bubble Diagrams and Portfolio Maps 34
4.3.4 Portfolio of Real Options 35
4.3.5 Scoring Models and Check Lists 37
4.3.6 Methods in Use 37
4.4 Technology Evaluation 38
4.4.1 Technology Assessment Process 38
4.4.2 Methods for Evaluating Technologies 39
4.4.3 Managing Real Options 41
5 FRAMEWORK FOR RESEARCH SURPLUS PORTFOLIO 43
5.1 Nokia Research Center 43
5.2 Open Innovation in NRC 44
5.3 Surplus and Its Storage Now in NRC 45
5.4 Goals and Requirement for RSP 46
5.5 RSP Concept 48
5.6 Utilization of RSP 50
5.6.1 RSP as a Search Engine 50
5.6.2 RSP as an Idea Bank 51
5.6.3 RSP as a Communication Tool 52
5.6.4 RSP as a Technology Market Place 52
5.7 Management of RSP 53
5.7.1 Packaging of the Surplus 53
5.7.2 Toolkit for the Evaluation of the Surplus 55
5.7.3 Managing RSP 60
5.8 RSP Database 61
5.9 Implementation 62
6 CONCLUSIONS 64
REFERENCES 67
APPENDICIES:
APPENDIX 1: List of key characteristics of surplus
APPENDIX 2: Example of the RSP system in practice
LIST OF FIGURES
Figure 1. Starting point for Research Surplus Portfolio 3
Figure 2. The elements of the constructive approach in the thesis ...5
Figure 3. The structure of the thesis ..6
Figure 4. The closed innovation model ..8
Figure 5. Erosion factors 9
Figure 6. The open innovation model . 11
Figure 7. Components of intellectual capital 15
Figure 8. The intellectual property management system .18
Figure 9. The methods for extracting value from company’s intellectual assets 21
Figure 10. The modes for managing non-core, technology-based assets 23
Figure 11. Sample measures 28
Figure 12. Strategy table model 31
Figure 13. Expected commercial value decision tree ..33
Figure 14. Strategic Bucket Method 34
Figure 15. Technology/Market matrix .35
Figure 16.The R&D portfolio based on the real options .36
Figure 17. Nokia Research Center innovation network 43
Figure 18. The position of RSP in NRC’s new technology development process ..48
Figure 19. The division of the project database 49
Figure 20. The concept of RSP 50
Figure 21. Attributes of research surplus .54
Figure 22. The connections between the databases of core technologies, patents and RSP.61
LIST OF TABLES
Table 1. The comparison of the principles of the open and the closed innovation 12
Table 2. Practical implications of transformative capacity 26
Table 3. Saint-Onge’s categories of intellectual capital .27
Table 4. Traditional DCF versus Real option perspective .41
Table 5. Methods for evaluating the future potential of research surplus .59
Table 6. The comparison of the project portfolio management and RSP management….65
ABBREVIATIONS
DCF Discounted cash flow
EBRC Emergent Business Research Coalition
ECV Expected commercial value
IA Intellectual assets
IAMS Intellectual assets management system
IC Intellectual capital
IP Intellectual property
ITU International Telecommunication Union
IPMS Intellectual property management system
LUT Lappeenranta University of Technology
MIT Massachusetts Institute of Technology
MMT Multimedia technologies
NIH Not invented here
NRC Nokia Research Center
NPV Net present value
NSH Not sold here
PPM Project portfolio management
R&D Research and development
ROI Return of investment
RSP Research Surplus Portfolio
TOL Technology out-licensing
1 INTRODUCTION
1.1 Overview
The thesis is a part of the Innovation Practices for New Business Creation in NRC –project named as Onions, which is a business project in the program called “Uudistuva liiketoiminta ja johtaminen” organized by Tekes. The objectives of the Onions are to find innovation practices for new business creation in Nokia Research Center (NRC) and enhance innovation climate there. Apart from NRC and Lappeenranta University of Technology (LUT) other parties in the project are Tampere University of Technology, Emergent Business Research Coalition (EBRC), the University of Tampere, Technology Centre Hermia and consulting companies Ledi Oy and Professia Oy. (Liito, 2006)
We live in a knowledge era where information floods from everywhere and knowledge is spread and becomes more detailed. In the business world it means the shorter life cycles for products and technologies and tighter competition. Innovation, “the process of transforming an invention into something that is commercially useful and valuable” (Miller & Morris, 1999, 2), and intellectual capital have become essential. The research and development (R&D) activities of companies have to follow that change and become more and more efficient. Miller & Morris (1999, 3) state in their book that R&D needs to find new approaches to innovation processes. The fourth-generation innovation models have the following characteristic (Berkhout et al., 2006, 393):
§ Innovation is embedded in partnerships
§ Early interplay between science and business is important
§ Knowledge of emerging technologies is complemented by knowledge of emerging markets (combination of hard and soft knowledge)
1
§ The need for skills and new concepts for managing networks with partners such as specialized suppliers and early users
§ Entrepreneurship is vital
A company can not survive alone anymore and even the innovation processes should go beyond the boundaries of the firm. Henry Chesbrough (2003a) launched a term “open innovation” to describe a new way to manage innovation and the R&D process. This new open innovation paradigm is discussed in the first part of the thesis.
NRC produces a lot of ideas, technologies and inventions that can not be used for some reason or another in the corporation’s core business. Most of those are not licensed, either, because of incompleteness or the fear of knowledge flowing to the wrong hands. In the spirit of the open innovation NRC is looking for opportunities to create new businesses from the research surplus and that is also the ultimate goal of the Onions-project. One of the very first problems in the new business creation is that NRC does not have a suitable storage for its “research waste”. This study concentrates on the problem and aims to create a framework for a portfolio where research surplus can be stored and classified.
Figure 1 illustrates the position of Research Surplus Portfolio (RSP) in the new business creation framework in the general environment. Research Surplus Portfolio is a new term and has not been used in literature before, but intellectual capital in general, technology portfolio and portfolio management have been written about, so theory context comes from that literature.
2
Figure 1. Starting point for Research Surplus Portfolio
1.2 Objectives, Restrictions and Research Method
The purpose of the thesis is to develop a framework for Research Surplus Portfolio. The main research problem is to define the whole new concept, RSP. How RSP could be utilized in NRC and how it could be managed? It is good to keep in mind that RSP is created for the new business creation from the research waste and it will not just be a graveyard of innovations, which are not currently useful. Another objective for the thesis is to find tools and methods for managing RSP. As mentioned before the concept of Research Surplus Portfolio is new, but the portfolio management literature offers many tools for managing technologies and some of them could be applied in the RSP management as well. The main research questions of the thesis are:
1. What elements does Research Surplus Portfolio consists of?
2. What tools and techniques could be applied to the RSP management?
Research Surplus Portfolio does not mean an idea portfolio where new ideas are stored, or a technology portfolio where currently used technologies are sorted. It is important to notice that the technologies placed in RSP are already verified to be “waste”. The thesis is restricted to examine only the surplus portfolio. It does not concern other technology portfolio management issues and practices or other parts of innovation chain in NRC.
The goal of the thesis is to create the concept description of Research Surplus Portfolio. It does not discuss about a single technology that might be placed in RSP. The Implementation of RSP with the physical entity and the computer system design is left out and only general recommendations are given.
The theory of the thesis is a literature review of the open innovation, the intellectual asset management and the portfolio management. To create the framework for RSP, the constructive approach is used. It has been used as a research methodology since 1990s. The constructive research aims to build an innovative, new solution for a real-world managerial problem with both practical and theoretical contribution. The approach emphasizes close co-operation between the researcher and the practitioners, linkages to previous theoretical knowledge, testing the practical applicability of the new construction and the reflection of the findings to prior literature. The elements of the constructive approach in the thesis are visualized in Figure 2. (Lukka, 2000, 114) To create the constructive framework, information about NRC is collected from interviews with Project manager, Matti Karlsson and Jukka Saarinen, head of Multimedia Laboratory, from NRC (Kalrsson, 2006a, 2006b; Saarinen 2006). In addition, Nokia’s websites are used to collect the general data about NRC.
4
Practical problem: What is RSP and the elements of it
Prior literature about portfolio management and managing intellectual capital Practical solution:
Framework for RSP
CONSTRUCTION Theoretical
contribution: RSP
concept
Figure 2. The elements of the constructive approach in the thesis (adapted from Kasanen et al., 1993, 246)
1.3 Structure
The thesis starts with the literature review about the open innovation paradigm. It is justified to get acquainted with this new innovation phenomenon, because the inspiration of the Onions-project and the thesis comes from the open innovation. The second theory chapter deals with intellectual capital and its management. The technology portfolio management handled in the chapter four is a part of the intellectual asset management and especially important with the open innovation paradigm. In the chapter five the Research Surplus Portfolio framework is introduced and the best suitable portfolio management methods from the previous theory chapters are discussed. Finally, research conclusions are made. The structure of the thesis is visualized with an input-output scheme in Figure 3.
5
Input Output
Onions-project overview and background for the thesis.
Introduction of the closed and open innovation paradigms. Drivers for chancing innovation process.
Literature review to intellectual capital terminology and management.
Portfolio management as a part of the intellectual capital management. Methods from literature. Managing stored technologies.
Tools and methods for RSP from the previous chapter. Information from NRC
RSP framework and other results of the thesis
Chapter 1:
Introduction
Clearing research
perspective, goals,
methods and structure.
Chapter 2: Open
Innovation Paradigm
Literature review from
open innovation
literature.
Chapter 3: Intellectual Capital Management Managing intellectual capital and extracting
value with it.
Chapter 4: Technology Portfolio Management Technology portfolio management, tools for technology evaluation.
Chapter 5: Framework
for Research Surplus
Portfolio
Design of the RSP
concept
Chapter 6. Conclusions
Assessment of the thesis
and usability of RSP
Objectives, restrictions and structure for the thesis.
The understanding of new needs and challenges of innovation management.
Concept of intellectual capital. The importance of managing intellectual assets in the open innovation world.
The meaning of the portfolio management. Tools and methods used to managing technology portfolio.
RSP framework for NRC. Recommendations for the implementation of the portfolio
The result of the thesis.
Figure 3. The structure of the thesis
2 OPEN INNOVATION PARADIGM
2.1 Closed, Traditional Innovation Model
The innovation models have been changing during the past decades. The first simple “Technology push” and “need pull” models were used in the 1960s. After that “Coupling models” emerged in the 1980s followed by “Integrated systems” in the 1990s. Already then the business world understood the importance of flexibility and intercompany networking in the innovation process and the “Strategic Integration and Networking” model was discussed. (Rothwell, 1992, 221) But still companies’ R&D processes were very closed from outside the company. Tidd et al. (2001, 254-255) use the expression “the development funnel” to describe the transformation of an idea to a product or a service. Innovations move through different stages from the idea creation to the launch phase. Later the funnel approach was connected to Cooper’s State-Gate System (Cooper, 1990, 46).
Even if teamwork and cross-functional co-operation in the R&D process was found and widely used, Chesbrough (2003a, 21) calls the traditional innovation model as a closed innovation paradigm, because the whole innovation process from the basic research to the product implementation was classified information and it was protected from the business world outside a firm’s boundaries. The closed innovation approach worked well in the environment of the twentieth century and it led many companies to success. But changes in the knowledge landscape that are discussed in the next chapter forces the industrial R&D to develop new models for the innovation process. But before that part, the closer look to the closed innovation model is made.
Figure 4 pictures the closed innovation model. Research projects move through the development funnel. Some of them are terminated and some end up to the market as new products or services. R&D projects can only enter in and exit one way. (Chesbrough, 2006,
7
4) Companies believe that they have to do everything internally and “Not Invented Here” (NIH) syndrome dominates the industrial R&D thinking. If a company had not developed the technology itself, how it could be sure that the technology is qualitative, operative and useful for it. (Chesbrough, 2003a, 29-30) In the other side of the pipeline, people think that if the developed technology is not sold by us, why we should let anyone else sell it, either. This phenomenon is known as “Not Sold Here” (NSH) virus. (Chesbrough, 2003b, 4)
Research investigation Development New products/services
Figure 4. The closed innovation model (Chesbrough, 2006, 4)
2.2 Changing Environment and Challenges for R&D
Centralized, internally focused approach to innovation fitted well in the industrial R&D management in the early twentieth century and it still fits in some industries, but for many industries it has become outdated. In this chapter reasons for that are introduced.
It is clear that management has become more complex, because market expansion, access to information and opportunity to choose from many alternatives has given the power to
8
customers. Today, the management of knowledge and intangible assets is the essential element of success. Challenges for our generation R&D are for instance the multiple sources of knowledge, combining explicit and tacit knowledge, need for a new organizational model and an innovation process, new approaches to finance, decision making and accounting, and tools and processes for integrating all these elements. (Miller & Morris, 1999, 24)
Chesbrough (2003a, 34-41) names four erosion factors that have caused problems to the closed innovation model. The first factor is the increasing availability and mobility of skilled workers. The number of high educated and trained people has grown significantly after the Second World War, and increased labor market gives well-trained workers an opportunity to shift from one company to another. If a talented employee does not change the employer, she or he might start a company of her or his own with the help of a venture capitalist. The raise of the venture capital market is the second erosion factor. These two factors mentioned lead to the third, external options for ideas sitting on the shelf. The customers and the competitors will not wait for establishing of those ideas. If a company does not launch the technology, someone else will. The last erosion factor identified is the increasing capability of external suppliers. A successful company can trust its suppliers instead of doing everything on its own. Figure 5 collects the erosion factors.
Availability and mobility
of skilled people
The venture capital
market
Figure 5. Erosion factors
The use of the closed innovation model makes innovations more incremental. The innovation process concentrates on the current businesses and does not create new ones. The return of a R&D investment has been dissatisfaction in many corporations. It is also noticed that a big part of technologies developed by the company’s research labs lie actually unused in some kind of shelf. Only a small percentage is in use in the current business. The next chapter offers an answer for these problems – the open innovation. (Allio, 2005, 19)
2.3 Open Innovation Model
The open innovation paradigm (Figure 6) suggests that ideas for innovations can also emerge or go to market from outside the company as well as inside. The new model assumes that knowledge is spread widely and even the successful innovators with big R&D resources have to look for the external sources of innovation. (Chesbrough, 2006, 2-3) The open innovation leverages the role of R&D. Researchers’ job is now, not only to create knowledge, but also to capture it from outside the company. Once a new innovation has taken place a company can use several business models to bring it to the market. If a technology is not suitable for the current business model, it can be licensed or donated to other companies or a new spin-off can be created. (Chesbrough, 2003a, 52, 187-188) But even if openness in the innovation process is highly encouraged there will always be need for some closeness, too (Christensen et al., 2005, 1535).
10
Technology insourcing
Figure 6. The open innovation model (Chesbrough, 2006, 4)
Although the open innovation is a new term, the sub-areas of it have been written before Chesbrough. A decade ago von Hippel (1994) suggested that companies should use external sources, customers, suppliers, universities and other companies, in their R&D activities. At the same time, Cohen & Levinthal (1990, 149) proved with they empirical studies that firms have to learn from the environment. For doing that, R&D resources need to be allocated to the absorptive capacity, too. The importance of alliances and networks has been studied also in the 90s by for instance Gulati (1998).
Still, the open innovation model offers new perspectives to the innovation management. The open innovation paradigm keeps external knowledge as important as internal knowledge. The basic assumption of the model is that the knowledge landscape has changed. Useful knowledge can come from multiple external sources, from universities and government laboratories to start-up companies, and from individual inventors to graduate students. The business model has a central position in the open innovation. Besides a clear
11
current business model a company can use a variety of other business models to commercialize new innovations. The current businesses compete for new technologies with the external channels to the market. The approach to the intellectual property management has been defensive in the traditional innovation models. Now the open innovation gives a proactive role to the intellectual property (IP). There are plenty of options how to benefit from IP. Companies can sell, license, donate, release or buy it. These alternatives are discussed in the chapter 3.3.3. New intermediate markets have been created to offer information about and access to external IP. With the new open innovation model, new measures of the performance of R&D have been developed. Measures like percentage of innovations originate outside our company and investments in outside firms will expand the assessing of R&D activities. When assessing the potential of a new innovation, measurement errors (false positives and false negatives) are paid attention in the open innovation model. Those are discussed more deeply in chapter 3.1. Especially the measurement of the false negatives has not been studied before. Table 1 summarizes the principles of the open innovation and compares them with the principles of the closed innovation. (Chesbrough, 2006, 11-16)
Table 1. The comparison of the principles of the open and the closed innovation (Chesbrough, 2003c, 38)
Open Innovation Closed Innovation
Not all the smart people work for us. We need to work with smart people inside and outside our company.
External R&D can create significant value; internal R&D is needed to claim some portion of that value. We do not have to originate the research to profit from it.
Building a better business model is better than getting market first.
If we make the best use of internal and external ideas, we will win.
We should profit from other’s IP whenever it advances our own business model.
The smart people in our field work for us.
To profit from R&D, we must discover it, develop it and ship it ourselves.
If we discover it ourselves, we will get it to market first.
The company that gets an innovation to market first will win
If we create the most and the best ideas in the industry, we will win.
We should control our IP, so that our competitors do not profit from our ideas.
12
3 INTELLECTUAL CAPITAL MANAGEMENT
3.1 Intellectual Capital and the Open Innovation
How companies manage their intellectual capital depends on how open they are. It is widely admitted that most patents are worth little, technology itself does not bring value to companies and commercialization requires the suitable business model to success. The open innovation suggests that corporations should more actively sell and buy their intellectual property. If a company’s own business models are not proper for a new technology, it can be sold, licensed or even donated to someone else. Of course, there are still some cases when it is better to protect the technology, instead of outsourcing it. (Chesbrough, 2003c, 39-40)
In the era of the closed innovation, patents were used mostly as barriers to the entry of the industry. Now companies start to realize the revenue-generating opportunity and other alternatives to use IP. Chesbrough (2004, 24-25) refers to the false negatives, which are projects that companies abandon, because they seem to be unpromising and unsuitable to the firm’s business model. To manage these measurement errors in conditions of high technology and market uncertainty, he proposes that companies adopt a new way to manage innovation – play poker instead of chess.
The open innovation paradigm emphasizes the importance of the intellectual assets management. Already a big part of the assets in a knowledge firm are intellectual, and entire industries will emerge and grow up based on the exploitation and licensing of the intellectual property of other companies and institutions. (Hogan, 2005, 30-31). In this chapter general issues about managing intellectual capital are discussed.
13
3.2 Concept of Intellectual Capital
A term “intellectual capital” (IC) was first used in the middle of the twentieth century (Stewart, 2001, 192), but the interest towards its management start to grow dramatically in the last decade, and in the beginning of the twenty-first century intellectual capital has become an essential part of the business, especially to the technology-intensive companies. (Goldheim et al., 2005, 43)
There are many definitions of intellectual capital. Edvinsson & Sullivan (1996, 363) define it as “knowledge that can be converted to value”. Intellectual capital is an umbrella term that is divided into smaller components (Figure 7). Human capital includes employees’ skills, know-how, the memory of important things to the company and collective experience. The value of human capital is connected to the persons and can not be written down. The other component of IC is intellectual assets (IA). It could be classified into three groups: commercializable assets, customer-related assets and structure-related assets. Intellectual assets are the source of innovations that will be commercialized. Usually intellectual assets are used internally, but if an asset is protected, it becomes intellectual property (IP). Typically, IP refers to patents, but the other protection forms are for instance copyrights, trade secrets, trade marks and semiconductor marks.
Companies also have structural capital, which supports the translation of human capital to intellectual assets. It stands for the whole infrastructure of the firm. Structure-related and customer-related assets can be seen as the intangible element of structural capital. The final term introduced here is complementary business assets. Those are a part of structural capital and are needed to deliver the products and the services to the customers. Without them intellectual assets are worth very little. Complementary assets can also be tangible or intangible. In the thesis, only intangible are noted. (Edvinsson & Sullivan, 1996, 358-361) Stewart (1994, 4) has named earlier another form of capital, customer capital, which means intellectual capital from the customer relationships. It could be included in complementary
14
business assets. There are also several other divisions, but those are not discussed in the thesis.
INTELLECTUAL CAPITAL
Human Capital
Experience
Skills
General Know-How
Creativity Intellectual Assets
Commercializable Assets
Products
Services
Processes, technologies
Intellectual Property
Customer- and Structure-related Assets
(Structural Assets)
Culture
Vision, Mission, Values
History
Agreements
Plans
Procedures, processes
Complementary Business Assets
Relationships networks
Figure 7. Components of intellectual capital (adapted from Edvinsson & Sullivan, 1996)
The role of intellectual capital depends on the company. It can be defensive or offensive. IC could be used as a protection of the product or the service or as an avoidance of litigation. More offensive roles are: revenue generation, standard creation, access to other’s technology, basis for new alliance and creation of barriers to the entry of new competitors. (Harrison & Sullivan, 2000, 142)
15
Since there are several roles for IC, there are also several ways to manage it. It makes the management of intellectual capital a complex and difficult task. The IC management consists of two basic functions: value creation and value extraction. Value creation deals with the new knowledge creation through learning and acquisitions. It concerns mostly human capital and it is left out of the thesis. Value extraction, on the other hand, focuses more on a company’s intellectual assets and aims to extract more value from existing intellectual capital. (Sullivan, 1998, 10)
3.3 Extracting Value from Intellectual Capital
3.3.1 Managing Intellectual Property
A well-constructed intellectual property management system helps firms to extract value from IP. It also provides a good basis for creating an intellectual asset and intellectual capital managing systems. Intellectual property represents the current pieces of IC that are creating value to a company at the moment and most companies have already portfolios for intellectual property. (Sullivan, 2000, 127, 130)
There are two different ways to manage IP. The portfolio of intellectual property (the IP portfolio) can be used as a protection, but another portfolio-as-corporate-business-asset-view has received more and more attention. With this view, IP has very offensive roles (view chapter 3.2). (Sullivan, 2000, 131-135)
The value extraction of the IP portfolio could be carried out by reducing the portfolio expenses or by increasing the portfolio income. A big part of the expenses of the IP portfolio comes from the maintaining fees of the patents, and taxes. It is estimated that approximately 70-90 percent of a company’s patents are useless to it and by eliminating, licensing or donating them the cost reduction could be made (Tao et al., 2005, 54). The IP portfolio income could be increased by improving royalty incomes from out-licensing.
16
These actions are meant for short-term value extraction. If a corporation wants to extract value during the longer period, it has to focus on increasing the quality of the portfolio and the use of it in the business negotiations and, expanding licensing, joint venture and strategic alliance activities. (Sullivan, 2000, 131-135)
The IP management system (IPMS) is illustrated in Figure 8. It is a series of action that links the innovation process, the patent portfolio and the business strategy together. It can be divided into five different responsibility areas: generation of candidate intellectual property, portfolio management, IP valuation, competitive assessment and strategic decision making. (Sullivan, 2000, 144)
17
Innovation Process
Generating New
Innovations
Strategic
Position
Figure 8. The intellectual property management system (Sullivan, 1998, 113)
The generation of candidate intellectual property includes three different tasks. The first is the overseeing of the innovation process. Usually, firms have the specific descriptions of their innovation process, where stages: research, development and product creation, are identified. It is needed for continue/discontinue – decisions, investment decisions and resource allocation. The second task is about generating new patents. This is a crucial part and it determines the future of the firm. That is why the technology and business analyses
18
of patentable innovations must be carried out carefully. Next, the patents are categorized and coarse valuation is made. (Sullivan, 2000, 144-147)
The IP portfolio is screened routinely. Patents that no longer bring value to a company can be moved for cost reduction. Portfolio managers make budget and maintenance fee decisions. The other element of the portfolio management is patent enforcement decisions that have to be done when infringement of a company’s patent is noticed. (Sullivan, 2000, 147-148)
Valuing intellectual property is difficult, but quite often there is a need to valuate a firm’s own technologies, patented or not. Chesbrough (2003c, 55) highlights the meaning of the business model. Even if a technology has a potential commercial opportunity, the value of it depends on the business model. Goldheim et al. (2005, 44-45) list three types of IP valuation methods. Market reference is a one measure. The value of a patent or a technology is a price someone is willing to pay for it, and it can be defined by comparing firm’s own asset to a similar one in the market. Second, buyer can value a patent (or another piece of IP) at the cost of producing the patent itself. The third method, net present value (NPV), calculates the expected cash flows from the future and discounts them to the present. All of these methods have pitfalls and are not very suitable for valuation IP. Some tools for technology evaluation are introduced in the chapter 4.
Competitive assessment function scans the environment and the competition. It includes activities like gathering, conjoining and communicating the information from the competitors. The assessment function helps the strategic decision-making process (the fifth area of responsibility). The decision is made, whether commercialize intellectual properties or pace them into a store to wait better opportunities, perhaps until another developing technology makes it more profitable to commercialize them. (Sullivan, 2000, 149-150) The open innovation gives also other opportunities for non-core technologies. Those are considered later in the thesis.
19
3.3.2 Managing Intellectual Assets
Intellectual assets are directed more to the future than intellectual property. When discussing about the value extracting, strategies rather than tactics needs to be considered. An intellectual asset management system (IAMS) is very similar to the IPMS. Those two have the same elements and there are only two notable differences. The first difference is the portfolio. In the IPMS, it is a collection of each kind of IP and logically, in the IAMS, it is a collection of the portfolios of different kind of intellectual assets. Also competitive assessment function is wider and more complex in the IAMS, because the perspective is now broader and a company has to seek much more information. If in the IPMS, competitive assessment focuses only on competitors’ patent portfolios, in the IAMS, assessment pays attention to things such as competitors’ licenses, business practices, and internal systems and methods. (Sullivan, 2000, 134, 163-164)
There are many methods how the value of intellectual assets can be extracted. But first, it is good to remember that the value of every individual asset is different and most of the assets are of very little value. (Tao et al., 2005, 53) In Figure 9, there are methods for extracting value from IA. Those are separated out of two sections: external and internal methods. There are also value creation dimension and cost dimension in the picture.
20
INTERNAL VIEW EXTERNAL VIEW
Figure 9. The methods for extracting value from company’s intellectual assets (adapted from Tao et al., 2005, 55)
The most high-valued intellectual assets can be used internally into the new products or services, or they can be applied to start-up, spin-offs or partnership companies. The middle range IA can be licensed or cross-licensed outside the company. Internally, value extraction occurs with “freedom to practice”, right to make and sell. IA could also be very profitable in negotiations about arrangements or partnerships by providing or denying access to technologies (or other IA). Naturally, protected intellectual assets (IP) give protection to a firm’s products. Low value IA, which usually just makes portfolios bigger and more costly, could be donated (if licensing is not possible). But even criticized, low value IA can have some benefits, too. Many companies publish their total number of active patents for boosting their reputation as an active innovator. Large IA portfolios and encouragement to patent could also motivate researchers. (Tao et al., 2005, 54-56)
21
3.3.3 Managing Non-Core, Technology-Based Assets
Non-core technologies are those that a firm can not use in its core businesses. Not only technologies protected by patents are able to be non-core technologies but also ideas from idea databases, and more mature projects and business units, which could become non-core due to the company’s strategic changes. A firm should regularly review the whole technology portfolio because there might be technologies with great potential outside the parent company. (Parhankangas et al., 2003, 6-7) Some of the modes managing non-core technologies (intellectual assets) are already introduced, but some alternatives to manage and gain value with non-core assets are discussed in more detailed here.
The types of managing non-core technologies are classified into three groups (Figure 10) based on the involvement of external parties. With the external methods technology-based assets are transferred outside the parent company. The hybrid modes illustrate situations where a technology is commercialized with an external partner. With the internal modes, technologies are kept in-house and developed further, put on the shelf or terminated. (Parhankangas et al., 2003, 9)
22
Figure 10. The modes for managing non-core, technology-based assets (Parhankangas, 2003, 9)
Selling a non-core asset is probably the quickest way to gain money from the non-core technology, but after selling it the parent does not have any rights to it. Sell-off has some big challenges with pricing, motivating employees, the timing of the sale and finding a buyer. Spinning off is another alternative to give up a technology-based asset. Now the technology is moved to a new firm organized around it. The parent and the new company can be competitors, partners or customers to each other or the spin-off can be independent. If the parent owns a portion from the spin-off the case is hybrid. Typically, this approach is applied when the technology is disruptive or the hurdle rate is below the company’s threshold for profit contribution (Goldheim et al., 2005, 45). If a company wants to get tax benefits, make contacts to universities or other research units, or it does not find a buyer, an option is to donate the asset. (Parhankangas et al., 2003, 9-12)
23
Use of the hybrid model reduces cost and risk to launch a new technology, but profits have to be shared with a partner, too. There are many arrangements for collaboration, but co-operation could be divided into two main categories: strategic alliances with cross-ownership (such as joint ventures) and alliances with just exchange of knowledge, resources and services. Besides other companies, firms collaborate with universities and other non-profit organizations. Licensing is a good and already largely used method for extracting value from non-core assets. Similarly to the collaboration, there are many types of license arrangements. Licensing suits better for managing mature technologies than emergent ones. (Parhankangas et al., 2003, 12-14)
Internal development allows a company to have full control of a technology and future profits. Although technology is non-core, sometimes it is developed internally, for instance completed for licensing or selling. Many corporations in these days have their own corporate venturing units. Typically, radically new and high risky activities are carried through the venturing unit. The venturing unit is a temporary arrangement, because when a venture matures it is moved to a business unit or outside the company. (Parhankangas et al., 2003, 14-15)
Some non-core technologies end up on a shelf. This was a one of the pitfalls of the closed innovation and a question, why non-core assets should be kept in a company, arises. Even if technology is classified as non-core, the future is unpredictable. It may be wise to keep some of the unpromising technologies for future development and new business opportunity identification. In the next chapter that concerns transformative capacity, this subject is discussed more deeply. (Parhankangas et al., 2003, 14)
3.3.4 Transformative Capacity
In contrast to absorptive capacity, ability to look opportunities outside a company (see chapter 2.3), Garud & Nayyar (1994) argue that transformative capacity, ability to exploit the storehouse of the company’s own technologies, is even more important. To create new
24
business opportunities, existing resources could be combined, and knowledge transfer across time could be the basis for a new business. That is why knowledge or technologies needs to be codified and stored.
The thesis focuses strongly on this area. Research Surplus Portfolio will be the shelf described above, where NRC can place their non-core technologies that are no longer in the research focus. In addition, the Onions-project aims to create new opportunities from the surplus technologies, in other words, increase NRC’s transformative capacity.
Garud & Nayyar (1994, 381) give some practical advices for the management of the transformative capacity. Because those are closely related to the management of the surplus technologies and the portfolio, they are discussed in this thesis, too, and gathered in table 2.
25
Table 2. Practical implications of transformative capacity (Garud & Nayyar,1994, 381)
Choice
Gather information.
Choose difficult-to-create to maintain.
Adopt rich media when making choice decisions concerning tacit knowledge.
Coordinate efforts across business and research laboratories to identify technologies for shelving.
Develop criteria for evaluating technological options.
Brainstorm on which technological paths to follow and which ones to abandon.
Consider the impact on other businesses and technologies when making technology maintenance decisions.
Maintenance
Catalog shelved technologies.
Periodically review the catalog of shelved technologies.
Develop avenues for researchers to share information.
Permit “underground” research and development activity.
Conduct internal scanning for shelved technologies.
Provide incentives for maintaining currently unwanted technologies.
Retain key personnel who posses tacit knowledge.
Maintain a minimum threshold of knowledge.
In fast-moving environments, retain more knowledge.
Retain entire teams when knowledge is systemic.
Reactivation and synthesis
Encourage scientists and engineers to move around among product groups and research laboratories.
Coordinate the work of business and research laboratories through sharing information.
Organize symposia and expositions to share information.
Install lateral information processing mechanisms to encourage co-operation among researchers and business.
Internally publicize topics being researched.
Formalize the task of recognizing demand and supply triggers.
Minimize any negatives associated with the not-invented-here-and-now syndromes.
Reward reactivation.
Allow enough time for successful reactivation and synthesis.
Assess reliability and validity of retrieved knowledge.
Encourage the development of interface standards to allow synthesis later.
To store non-core technologies and use them later (the transformative capacity) a company needs to manage three tasks: The choice of technologies, the maintenance of technologies and, the reactivation of technologies and synthesis. First, a company must choose, which technologies it will maintain. The catalog of shelved technologies should be scanned periodically. The third task, reactivation, includes for example the business opportunity recognition and coupling the reactivated technologies with the current ones. (Garud & Nayyar, 1994, 378-383)
26
3.4 Managing Intellectual Capital
The management of intellectual capital is easiest to begin from IP and IA management. After that it can be moved to managing the whole intellectual capital by adding human capital management. (Sullivan, 1998, 261) Saint-Onge (1996, 10) has examined intellectual capital from the human capital perspective. He argues that the most of a company’s IC is tacit and should be managed with the knowledge-based models. He groups IC into three categories (table 3).
Table 3. Saint-Onge’s categories of intellectual capital (adapted from Saint-Onge, 1996, 10)
IC category Description
Human Capital Customer Capital
Structural Capital The capabilities of employees to meet customers need
External relationships with customers. It includes the depth, width, attachment and profitability of customers.
Organization’s capability to provide solutions to customers
Measuring intellectual capital means actually predicting the future of the company. Managing (and measuring) IC has to be tightened to the business strategy and the objectives. Measures must also be something that can be managed against. There are some measure examples in Figure 11. Measures can be either qualitative or quantitative. Qualitative measures are divided further into value-based and vector-based measures and quantitative into financial and non-financial measures. Several measurement schemes have been developed, such as the Skandia Navigator, the Balanced Scorecard, the Sveiby Model and the OECD Measures, but in the thesis, those are not introduced further. (Sullivan, 1998, 267-271)
27
Value-Based Vector-Based Non-financial Financial
- Value Category - Rate of Addition - Techniques - $ Invested
- Alignment with - Rate of Deletion available - $ Received
Visio & Strategy - Backlog - Age - Forecast Income
- Satisfaction - Market share forecast - Remaining life - Forecasted costs
- Quality of IAs - Coverage - Subject/technology
- Comprehensiveness
- Stock Price
Figure 11. Sample measures (Sullivan, 1998, 269)
There are quite a few factors that affect the way companies manage their IC, and different ways to manage it. Some take value creation perspective and focus on human resources and knowledge creation or ability to convert knowledge into intellectual assets. Other companies, with large amount of unused patents, pay attention to patent portfolio. Knowledge companies, whose strategy requires focusing on intellectual capital, manage both value creation and value extraction. (Edvinsson & Sullivan, 1996, 362-363)
28
4 PORTFOLIO MANAGEMENT
4.1 Importance of the Portfolio Management
The technology portfolio management is a part of the IC management. The portfolio thinking comes from the finance. The restrictions of the term “portfolio management” are different, but in the thesis the portfolio management refers to the technology, project or R&D portfolio. Cooper et al. (1999, 335) defines the portfolio management as “a dynamic decision process, where by a business’s list of active new products and (R&D) projects are constantly updated and revised. In this process, new projects are evaluated, selected and prioritized; and resources are allocated and reallocated to the active projects.” The other definition tells that the portfolio management is the science of meeting needs and expectations of the organization’s investment strategy with a set of knowledge, skills, tools and techniques (Dye & Pennypacker, 1999, xii). In the knowledge intensive environment and with growing interest in the intellectual capital management, the importance of the technology portfolio management is growing, too.
The portfolio management has three main goals: maximizing the value of the portfolio, balancing the portfolio and linking the projects to a company’s strategy (Cooper et al. 2000, 27-28). Other objectives related to value maximization are to maintain the competitive position and effective resource allocation. Financial reasons for the portfolio management are according to Cooper (2001, 364-366) the most important ones to companies. Besides these goals, improved communication and better project selection were mentioned in Cooper’s et al. survey. The following part will introduce a few frameworks for the portfolio management and some portfolio management methods for reaching the objectives.
29
4.2 Portfolio Management Process
A portfolio management process could be seen as a part of the company’s intellectual capital management process that was already considered in the last section (see for instance Figure 8. IP management system). However, the portfolio management process is a meager approach to the intellectual capital management. It handles (R&D) projects and leaves other IC out.
In the portfolio management process, the main goals of the portfolio management: value maximization, balance of the portfolio and strategic fit, should be kept in mind. Systematic framework for the portfolio management decreases the impact of personal opinion in decision making, guarantees that projects are evaluated equally, and helps managing situations where personnel changes. (Poskela et al., 2001, 85) The portfolio management process includes two phases. First of all, projects must be selected to the portfolio, and effective evaluation is needed, but the portfolio has to be reviewed continuously, too. With these tasks a company can be sure that projects in the portfolio aim for its strategic objectives. (Dooley et al., 2005, 469)
In the literature, several portfolio selection frameworks have been introduced. Cooper et al. (1997a, 44, 46 ) introduce two models: strategic bucket model that is discussed in the next section (methods for portfolio management) and StratPlan strategic check model that is pretty similar to the bucket model, but moves bottom up while the bucket model has top-down approach. Strategic table is another way to select R&D projects to the portfolio. All projects are opportunities to a company and resources are allocated among these opportunities. It can be divided into a five-step process (Figure 12) (Spradlin & Kutoloski, 1999, 26-27)
30
Framing the
problem Construct the
alternative table Create a
strategic table Evaluate
individual
opportunities Evaluate the
portfolio
Figure 12. Strategy table model (Aalto, 2001, 31)
The third framework for the portfolio selection discussed is Archer & Ghasemzadeh’s (1999, 211) model. The actual selection process is divided into five stages: Pre-screening, individual project analysis, screening, optimal portfolio selection and portfolio adjustment. Framework also takes into account the pre-process strategy development and methodology selection as well as the post-process stages: project development, phase evaluation and the successful completion of a project. Archer & Ghasemzadeh (1999, 213-214) also discuss about a possibility to integrate this selection process to the computer based decision support systems (DSS) and the group decision support systems (GDSS).
Regular portfolio reviewing meetings are essential (considered already in chapter 3.3.1) There decisions whether to continue a project or not are made. It is important to have courage to make a canceling decision as early as possible if needed. (Poskela et al. 2001, 86) In addition, with this reviewing process one of the big challenges of the portfolio management, control and communication between project teams, is met (Dooley et al., 2005, 470-471).
While the typical frameworks suggest that the portfolio should implement a company’s strategy and projects should be selected based on strategic fit, Martinsuo (2001, 73-74) considers the portfolio management process from a different point of view. A portfolio is managed like in the fashion world: a portfolio is a curriculum vitae or a sales documentation that is offered to customers. The customers’ impact on the contents of a portfolio and the portfolio management process actually modify the strategic direction. The RSP management is more about offering technologies to the customers (notice that the
31
customers can be inter-organizational, too) and modifying strategy than normal portfolio management.
4.3 Methods for the Portfolio Management
4.3.1 Financial Methods
The most popular portfolio management methods are financial, such as NPV, return on investment (ROI) and payback period (Cooper et al., 2001, 366). Those are widely well-known, but Figure 13 visualizes the Expected Commercial Value (ECV), a variant of NPV, which is a little less familiar method. It is based on the decision tree and the option pricing theory, and it takes into account constraining resources (Cooper et al., 2000, 27). It could be used when prioritizing projects, but the pitfall is that it does not consider portfolio balance. (Aalto, 2001, 36)
32
Commercial
Success
Technical
Failure
ECV NPV P SI C P D
CS TS
ECV = Expected commercial value of the project
SI = Strategic importance of the project
PTS = Probability of technical success
PCS = Probability of commercial success
D = Development Costs
C = Commercialization (launch & capital) costs
NVP = Net present value of project’s future earnings
Figure 13. Expected commercial value decision tree (Aalto, 2001, 36)
There are some problems associated with the traditional financial methods. First, the methods require accurate financial data for calculating the results right. Usually, input data comes from rough market and cost analyses that are easy to manipulate. Second, the decision whether to carry on a project or not must be done in the early state of the project, and financial data is therefore impossible to get. (Cooper et al., 2001, 378)
4.3.2 Strategy Related Methods
The use of the business strategy for allocating money to projects is the second used method for portfolio management. It starts with business’s goals, vision and strategy. Projects are divided into strategic buckets and every bucket has a certain budget. Then projects are
33
ranked within the buckets and projects under the spending limit are realized. This method is called Strategic Bucket approach and is illustrated in Figure 14. (Cooper et al., 2001, 366¬368)
New Products:
Product Line A
Target Spend: $8.7M New Products:
Product Line B
Target Spend: $18.7M Maintenance of Business
Product Lines A & B
Target Spend: $10.8M Cost Reduction
All Products
Target Spend: $7.8M
Project A 4.1 Project B 2.2 Project E 1.2 Project I 1.9
Project C 2.1 Project D 4.5 Project G 0.8 Project M 2.4
Project F 1.7 Project K 2.3 Project H 0.7 Project N 0.7
Project L 0.5 Project T 3.7 Project J 1.5 Project P 1.4
Project X 1.7 Gap = 5.8 Project Q 4.8 Project S 1.6
Project Y 2.9 Project R 1.5 Project U 1.0
Project Z 4.5 Project V 2.5 Project AA 1.2
Project BB 2.6 Project W 2.1
Figure 14. Strategic Bucket Method (Cooper et al., 2001, 368)
R&D spending can be split into the buckets several ways, for instance type of market, type of development (maintenance, exploratory or frontier research), product line, project magnitude or technology area. For prioritizing projects within a bucket, formal methods or just strategic approaches are used. The strategy is in a great concern when the go/kill decisions are made, and the strategy approach is therefore encouraged to use. (Cooper et al., 2001, 371)
4.3.3 Bubble Diagrams and Portfolio Maps
Bubble diagrams and portfolio maps are used mostly as a supporting method, because with them the balance of the portfolio is visualized. An idea is that projects are drawn into an X-Y coordinates by using bubbles. The size of a bubble will be the third dimension of the matrix. In addition, colors and different color brightness could be used to enrich the analysis (Aalto, 2001, 57). Maybe the best known example of the diagram analysis is Boston Consulting Group’s matrix – stars, cows, dogs and wildcats (Henderson, 1970).
34
The parameters or the criteria on the axes can vary. Companies could use the parameters such as fit with business and strategy, strategic importance, durability of competitive advantage, reward (based on financial expectations), probabilities of success, cost of completion, time to completion, business maturity, market potential, market size, technical familiarity and market attractiveness. (Aalto, 2001, 39-40) Figure 15 introduces a model used in the industry. The model links technology novelty and market novelty. (Stevens, 1997, 44)
TECHNOLOGY
New to the
World
New to the
company
Known to the
company
Known to the New to the New to the MARKET
company company World
Figure 15. Technology/Market matrix (Stevens, 1997, 44)
4.3.4 Portfolio of Real Options
The R&D portfolio can be visualized also as a portfolio of options. MacMillan & McGrath (2002, 50-56) introduce a bubble diagram, which employs three types of real options in Figure 16. Projects are placed in a technical/market uncertainty matrix. Positioning options are opportunities to compete with uncertain technology arena. Those are appropriate in situations where there are several technologies that could satisfy the market need and it is not clear, which one will be the winner, or the trajectory of development of the technology is unclear. Scouting options are investments to learn from the market. They are used when the technology is not a problem, but the company is not sure, which combination of
35
attributes is the most attractive. Stepping-stone options are opportunities that include both high uncertainty of the technology and the market. Some projects do not meet so much uncertainty and options are not necessary. In the matrix, those projects are divided into enhancement launches, which improve existing products and services, and platform launches, which require bigger investments and are more uncertain.
Positioning Options
15 % Stepping
Stone
Options 5%
Platform Launches 40 % Scouting
Options
15 %
Enhancement
Launches
20 %
LOW MEDIUM HIGH
Market Uncertainty
Figure 16.The R&D portfolio based on the real options (MacMillan & McGrath, 2002, 55)
The portfolio has to fit with the company’s strategy and it has to suit to the business environment, too. That is why every company has to make its own decisions, on which parts of the portfolio are the most essential ones, and weight resources based on that decision. In Figure 16, percentages represent the amount of allocated resources. However, those percentages are just a general guide for the allocation and can not be applied as such. After the mix of projects is determined, the projects are evaluated and prioritized only in their own category. For example the positioning projects compete for resources only with the other positioning projects.
36
4.3.5 Scoring Models and Check Lists
Scoring models are considered as an effective portfolio management tool. Projects are scored based on selected questions or criteria. The implementation of the scoring method varies. The criteria can be either simple or weighted, a project can be scaled for instance with low-medium-high, 0-5 or 0-10 scales. The model can be used to prioritizing projects against each other, or scores can be compared to some cut-off criteria (to make kill/go decisions). The selection criteria include for instance strategic fit, financial reward, risk, probability of success and own technological and business capabilities (Cooper et al., 2001, 368, 371)
A check list approach is pretty similar to the scoring model. Projects are evaluated with yes or no questions. The criterion to carry on the project could be a certain number of yes answers or every answer has to be yes. The check lists are not as popular as the scoring models and they are usually used in go/kill decisions, unlike the scoring models that are most popular as a ranking method. (Cooper et al., 2001, 368, 372)
4.3.6 Methods in Use
The best companies managing their technology portfolio in practice use explicit and formal methods. In those companies, rules and procedures for the portfolio management are well-defined and the tools are applied to all projects. None of the methods mentioned provides a universal answer. Typically, a company uses two or three methods. It is recommended to use a combination of the financial methods, the strategic approaches, the scoring tools and the bubble diagrams. (Cooper et al., 1999, 350)
With the portfolio methods, strategic alignment and the selection of high value projects can be ensured, but there are some problems in the portfolio management that the methods fail to solve. In many cases, projects interact with each others and compete for scarce resources (Archer & Ghasemzadeh, 1999, 210). The methods mentioned above are weak to select the
37
right number of R&D projects, encourage the timely completion of projects and balancing the portfolio. Because of the pitfalls main challenges for the portfolio management are: resource balancing, prioritizing projects against one another, making go/kill decisions in the absence of solid information and too many minor projects in the portfolio. The portfolio management should be integrated with the State-Gate processes to improve the quality of the information of projects, and with the resource capacity analysis, to balance between the demand and availability of resources. These methods connected to the right tools make the portfolio management more efficient. (Cooper et al., 2000, 19, 24-27)
Even if the efficient, well-defined models are used right, the decision-making is still done by people. The technology portfolios are complex and there are many liaisons between the projects, so a situation is impossible to visualize exactly with any of the models. Therefore, individuals in an organization must be tightly integrated into the decision-making process between projects. (Aalto, 2001, 49)
4.4 Technology Evaluation
4.4.1 Technology Assessment Process
The previous chapter gave examples how to manage the entire portfolio, but this chapter deals with the evaluation of an individual technology. Of course some of the portfolio management tools, such as the financial methods and the scoring models, can also be applied to assess a single technology in the portfolio, but more examples of methods are discussed here. A big part of the value of technologies, especially new ones, is related to real options. The approach is discussed in chapter 4.4.3. Technology assessment is a wider process than just an evaluating task. The process is shortly gone through before the methods.
38
To choose the right technologies for the further development is essential for a company. Technology assessment plays a vital role when R&D spending is increasing, competitive advantages are narrowing and life cycles are getting shorter. Doering & Parayre (2000, 77-96) illustrate four steps of a dynamic technology assessment process:
§ Scoping: A firm has to decide boundaries for technology assessment. Limits are established based on the firm’s capabilities, strategic intent, potential new markets and technologies.
§ Searching: A company can look for new technologies and opportunities from inside the firm, from the public licensor of technology and from the literature. This step includes sensing strong and weak signals from the environment and developing a “group mind” by capturing knowledge and information and gathering it.
§ Evaluating: Managers use different methods for evaluating and ranking promising technologies and possible development projects. Some tools for that task are introduced in the thesis, too. A firm’s strategic position, the environment and the different types of risks must be considered.
§ Committing: When decision to pursue a new technology is made, a firm has to decide how to do it, and it makes a strategic commitment.
4.4.2 Methods for Evaluating Technologies
Opportunity identification methods try to define different market or technology arenas, which a company may be interested in. Tools for assessing the uncertain future are roadmapping, technology trend analysis and forecasting, competitive intelligence analysis, customer trend analysis, market research and scenario planning. The methods could be used in the opportunity analysis, too. When in the idea identification, the tools were used to identify the opportunity, in the opportunity analysis, the same tools provide more detailed
39
information about the selected technology and help to allocate resources. (Koen et al., 2002, 15-18)
Roadmapping has been used since the beginning of the 1990s. It can be compared with the linkage-based structure of World Wide Web. Every element of the map is linked to more than one individual or data source. A well constructed map tells, for instance, where the information comes from, its timelines and responsibilities. (Foreier, 2002, 52) The value of the roadmap is actually the mapping process, a forum for sharing wisdom about projects’ resources and teams’ capabilities. (Koen et al., 2002, 16)
Scenario planning goes a step further from the mapping. It gathers information available to number of possible stages, and images possible futures of a company. It shows how different elements might interact under certain conditions. Scenarios should be made relevant, internally consistent, and they should describe generally different futures, not be the variations of a one. The scenario planning tries to illustrate changes that decision makers otherwise would ignore, organize the data of emerged possibilities into easier form and challenge the prevailing mind-set. The power of the scenario planning is that it deals with uncertainty and complexity the way the other planning and strategy tools do not. (Schoemaker, 1995, 25-26, 30; Shoemaker & Mavaddat, 2000, 211-214)
Competitive intelligence analysis, in other words, the gathering of information about the competitors, is largely used in the business world. The analysis refers to the practice of collecting, analyzing and communicating data about competitive environment trends. It should not be just the information gathering but also finding workable data. (Koen et al., 2002, 16)
The decision trees can be also used to estimate the value of a R&D project. The methodology provides an opportunity to eye projects’ value when it is possible to terminate a project at the each point of the development process. In the decision tree approach, weights are assigned to different scenarios and then weighted NPV is calculated. But only a small number of diverse scenarios can be made. If a manager wants to assign probabilities
40
for many variables at the same time, Monte Carlo analysis can be applied. (Boer, 1999, 291-297)
4.4.3 Managing Real Options
The traditional financial tools, like Discounted Cash Flow (DCF) methods, ignore some opportunities, such as the option to terminate, the options of making follow-on investment and the acceleration option, when estimating R&D projects. A new approach to assess technologies, which has gained a lot of detention in the literature, is real options. The traditional DCF model is compared to the real option approach in table 4. Although, real options were discussed earlier in the context of the portfolio management methods, those are used as a tool for valuating a single technology, too. Real options are analogue to financial options. Like financial options, if a company makes a strategic investment, is has the right, but not the duty to exploit opportunities among the investment in the future. (Boer, 1999, 290, 300; Boer, 2000, 26)
Table 4. Traditional DCF versus Real option perspective
Traditional DCF Perspective Real Option Perspective
Views uncertainty as a risk that reduces investment value
Assigns limited value to future information
Recognizes only tangible revenues and costs Views uncertainty as an opportunity that increases value
Values future information highly
Recognizes value of flexibility and other intangibles
Assumes clearly defined decision path Recognizes path determined by future information and managerial discretion
Real options are difficult to evaluate. They arise in the most technology investments, but they take many forms, and could be hard to recognize and implement. Some of them emerge naturally, but they could also be created. The valuation of real options might be difficult as well. A real option management process includes four phases: adopting an
41
options perspective, creating and structuring options, valuing options and implementing the real option approach. (Hamilton, 2000, 274-277)
Before real options can be valuated, they have to be identified. Some of them could be created by the systematic decision process. Real options create value by generating financial return from future commercialization, by strategic positioning, providing new opportunities and by creating new knowledge. To value these benefits is not an easy task. Financial models are the most popular method to valuate real options. The decision analyses (decision tree was discussed earlier) can also be used in the valuation. The third tool is threshold assessment, which is not trying to solve the absolute value of the options, but concentrates on a question whether the value of the option is enough to justify the investment, instead. It is crucial to keep in mind that the value of an option is not static and it depends on how and when the option is applied. (Hamilton, 2000, 277-286)
42
5 FRAMEWORK FOR RESEARCH SURPLUS PORTFOLIO
5.1 Nokia Research Center
Nokia invest strongly to research and development activities. R&D expenses were approximately 11 % of Nokia’s net sales in 2005. R&D resources are divided between Nokia’s principal business groups, Technology Platforms and Nokia Research Center. (Nokia, 2006a)
Nokia Research Center was founded in 1986. The mission of NRC is to renew Nokia through the strategic and long-term research. NRC supports Nokia’s strategy by developing technologies and concepts for existing Nokia businesses, but it also challenges the strategy by exploring and researching potential technologies for the future. Figure 17 visualizes the organization structure of NRC at the moment. (Nokia, 2006b)
Figure 17. Nokia Research Center innovation network (Nokia, 2006b)
43
NRC operates in six countries. Research centers are located in Tampere, Helsinki, Bochum, Budapest, Cambridge (USA), Mountain View (Palo Alto), Beijing and Tokyo. In 2005, there were 1097 employees in NRC, which is 5 % of Nokia’s R&D personnel. Half of Nokia’s essential patents originate from NRC. In 2005, it generated 311 patents. (Nokia, 2006b)
5.2 Open Innovation in NRC
Open innovation is becoming more and more important to Nokia, and also NRC implements many aspects of it. The most important element of the open innovation to NRC is external research collaboration that searches for new opportunities for Nokia. (Karlsson, 2006a) Nokia is a part of many consensus creating consortia, such as International Telecommunication Union (ITU). To NRC, bilateral cooperation with universities is also important. For example Nokia Research Center Cambridge joins Nokia and Massachusetts Institute of Technology (MIT). NRC has a powerful, global research network with universities, research institutes, international organizations, large corporations and venture capitalist companies, and R&D project cooperation in Europe, North America, China, Japan and India. (Nokia, 2006b)
In addition, Nokia supports the open source development. NRC participates in several open source projects as a host, contributor and/or sponsor. (Nokia, 2006c) But even if the external opportunity seeking and the research collaboration have functioned well for a long time, the other direction of the open innovation, new business models outside the parent company and core business, has not been applied much. The Onions-project concentrates on that area of the open innovation. In the future, the creation of new businesses from NRC’s technologies and funding activities by selling or out-licensing old technologies are interesting targets for the development. (Karlsson, 2006a)
44
5.3 Surplus and Its Storage Now in NRC
All activities and laboratories in NRC produce research surplus. It can also be in every possible form, such as A4-paper, code, research report, patent or demo. Naturally, it means that surplus items are in the different stages of development, too.
At this moment, there is no a research waste treatment system in NRC. All projects are stored in a single database. It means that NRC’s core and non-core technologies are in the same place. All documents, codes and other material from the research projects are in that system. The level of the documentation style and quality is good, but only basic information from the projects is available. (Saarinen, 2006) NRC does not have any kind of explicit principles for managing their research surplus and there is no portfolio for surplus technologies, either. Different laboratories do not know each other’s storage methods for surplus, if there even are such methods. (Karlsson, 2006a)
In Multimedia technologies (MMT) laboratory the descriptions of surplus material are written on a paper and collected to a folder, but this paper version of “surplus portfolio” is not connected to the other data systems and portfolios. Only the surplus patents are in the patent database. MMT-laboratory has used the following list of questions in storing their research surplus:
1. For how long the research/development has been done?
2. Current status/ stage of the technology?
3. How many people are currently employed? Names?
4. The way of technology transfer? Receiver?
5. Connection to Strategic Focus Areas?
6. Plans for the future?
7. Financiers?
8. Is this project in the core field of NRC?
9. Other information?
45
But the paper portfolio with these questions is not exactly Research Surplus Portfolio, though. It is done on an ac-hoc basis, and the purpose has not been to list surplus technologies for later utilization. So there is no reason why just these elements from the surplus should be collected. (Karlsson, 2006b)
Cancelled projects and the results of those projects have really not interested anyone, because the focus is towards the future; looking back at the old research projects has been considered being a waste of time. The surplus research has been only a part of the learning process. NRC has been strictly closed from outsiders, so there have not even been reasons for the waste research documentation. But now, when the open innovation is becoming common in many industries, like the telecommunication business, and openness will be more and more important, NRC will also follow the evolution. (Karlsson, 2006a)
5.4 Goals and Requirement for RSP
Research Surplus Portfolio is not like company’s ordinary technology portfolio and should therefore be managed differently than other intellectual assets. The biggest difference is that technologies in the RSP are already classified as non-core. In other words those are not used in the current businesses.
The ultimate goal of the creation of RSP is the new business creation from the research surplus. Another goal that should be considered is to generate profits and benefits from surplus technologies by selling, licensing or donating them. The RSP system should be build to serve these goals. The purpose is to get information from researchers’ heads into a computer database. The information should be easy to find, understand and analyze.
For making surplus technologies useful, the external technology environment should be scanned. It is more likely to find opportunities for waste technologies from outside the company or from the new business areas. However, the technology transfer to maintain the
46
current businesses is also an opportunity. In chapter 3.3.4 Garud & Nayyar (1994) gave guidelines especially for that. Because input to the portfolio is something that has been left over, the balance of the portfolio and its strategic fit are not important. Evaluation should be concentrated on the future opportunities.
In the portfolio literature, the management of a technology/project portfolio should be efficient and effective. It is easy to build a massive system with various complex tools, but that does not help a new product development process. The concept of RSP should be even lighter than a normal technology portfolio. The research surplus attributes must be easily stored, searched and updated. Because technologies are already classified as wastes, some phases of the portfolio management process can be dropped out. Typically, the portfolio managing process includes seven phases (see chapter 4.2): selection (storage), evaluation, prioritizing, resource allocation, balancing the portfolio and strategic fit consideration, decision making (go/kill decisions) and maintaining. The RSP managing process must have only a data storage phase, some kind of evaluation system and a maintaining process.
It is important that the new portfolio does not require much extra work. Selected tools should be light and easy to use and understand, but still efficient. A management team that is responsible for RSP is needed. To make sure that the portfolio is properly managed and maintained, the members of the team should be involved in the task for a long period.
Because the material that needs to be stored in RSP is in various forms (reports, codes, demos...), it is difficult to find a method for saving the useful information. The purpose of the whole portfolio has to be kept in mind. RSP aims to serve the creation of new businesses. That is why a one of the main questions in the RSP concept creation is how surplus should be collected and stored so that it would be useful. Other arising questions are for instance: What kind of storage RSP would be? How is it utilized? Which elements of the surplus technologies and cancelled projects should be stored in RSP? How the surplus is evaluated? As well as to all business activities, the cost/benefit consideration is valid to RSP and its development, too.
47
5.5 RSP Concept
Figure 18 illustrates the position of the new portfolio in Nokia’s innovation system. As mentioned before, all activities in NRC produce the surplus from their research. On the other hand, every activity in the new technology development process has access to and can benefit from RSP. In this part of the thesis a constructive concept for RSP is created and a framework for its utilization, management and implementation is considered.
Figure 18. The position of RSP in NRC’s new technology development process
Research Surplus Portfolio is an inventory for the non-core technologies from the projects that have been cancelled and the technologies that have not been transferred to Nokia’s businesses. To create RSP, a single database system for projects is split in two portfolios – to core technologies and non-core technologies. The latter will be new RSP. At this point, it is reasonable to view, what kind of surplus could there be in RSP. The surplus items placed in RSP could be technologies outside the current business, project results without value, ideas, parked technologies that are shelved for later use, possible out-licensing and selling cases, spin-off and venture opportunities and open source technologies (Saarinen, 2006). So RSP could be seen as a solution that separates non-core technologies mentioned in Figure 19 from core technologies. Technologies could be divided into different classes,
48
too. The classification is essential for the efficiency of the portfolio. Managers and researchers are able to find what they are looking for more easily. Figure 19 illustrates the division of the project portfolio
THE PROJECT DATABASE
CORE
TECHNOLOGIES NON-CORE TECHNOLOGIES
(RSP)
§ Technologies outside the current business
§ Projects results without value
§ Ideas
§ ”Parked” technologies
§ Out-licensing cases
§ Selling to the third party-cases
§ Spin-off opportunities
§ Venture opportunities
§ Open source technologies [Type]
Figure 19. The division of the project database (adapted from Saarinen, 2006)
Research Surplus Portfolio consists of two parts: the management of data in RSP and its utilization (Figure 20). Both are continuous processes that do not depend on each other, which means that data can be stored and searched anytime. At the storage stage, the research surplus is documented to a database and its future potential is evaluated. The third managerial task is the maintaining of the portfolio. The utilization possibilities of RSP are several. It could function as a search engine, a communication tool, an idea bank or a marketplace for surplus technologies. The following chapters will introduce the phases of the both sides of RSP. First, the four utilization possibilities mentioned above are discussed
49
and after that the management of RSP is considered. Last, some database and implementation issues are highlighted.
Figure 20. The concept of RSP
5.6 Utilization of RSP
5.6.1 RSP as a Search Engine
The searching facility is a very important aspect of RSP from the new business creation point of view and it has to be designed carefully. No matter what the final concept of RSP is, there must be an effective search tool included in RSP. On the other hand, the whole concept of RSP could be based just on searching technologies from other databases. This kind of Internet search engine would search data from all NRC’s data systems and
50
portfolios. RSP used in this kind of concept would be beneficial in the basic research work, too, when researchers could be able to search more efficiently the earlier research projects.
This approach demands no extra work from employees at the storage phase and not much managerial effort, either, but how beneficial the system would be, if the new business creation is considered. Is just a search tool able to identify the promising surplus technologies, if those are not documented properly? One big problem in utilization of the research surplus is that surplus technologies are usually very difficult to understand without special technical knowledge, and if there is not any summary of a particular surplus technology, it is hard for an outsider to recognize the opportunities that a technology may include.
5.6.2 RSP as an Idea Bank
When technological possibilities and market needs match, an idea occurs. New ideas are essential for the new products development and the success of the whole business, especially in the high-technology industries. A company has to have a system for the idea generation. Ideas can appear from several sources. One of the main sources is the company itself and its personnel. (Cooper, 1997b, 121-123,128)
RSP is the in-house source of new ideas. Researchers may find synergies between their own projects and the surplus, figure new possibilities to use surplus or invent an original idea based on the surplus. Like Cooper (1997b, 132) suggest, new ideas can be delivered by e-mail to researchers. Also, in the case of RSP, the information about some promising surplus technologies could be displayed on e-mail. It does not mean that information should be sent to every researcher in NRC. When a piece of the research surplus is documented to RSP, the documenter can pick a few persons to send the publication of the surplus material to. Mail could be sent for example to certain “RSP contact persons”, to one researcher from every laboratory or to out-licensing personnel.
51
5.6.3 RSP as a Communication Tool
Chien (2002, 367) emphasizes that portfolio selection approaches will serve communication between project teams and encourage discussion. Also, Bordley (1998, 407) brings up the project selection models as a method to ask questions from the entire organization. RSP can be an avenue for researchers to share information. It can be used for collecting information, sharing options, asking questions and contribute inter-organizational discussion.
In NRC, separated laboratories do not know details about each other’s projects and this is not just NRC’s problem. In companies, it is very common that only a few persons on the top can piece together the big picture. From RSP, different research groups, project teams and laboratories in NRC are able to observe, what kind of research has been made in house. Somebody’s waste might be useful in some different contexts. Communication with technology out-licensing (TOL) will be more effective with RSP, too. With more open attitude to alternative paths to markets, such as using of out-licensing, continuous discussion between laboratories and TOL experts will become more important in the future.
5.6.4 RSP as a Technology Market Place
Chapter 3.3.3 gave alternatives to how non-core technologies and assets could be employed. For the external and the hybrid modes, a technology marketplace is needed. One potential use for RSP is to use it as a marketplace for technologies. It requires that at least some parts of the database are open outside NRC. Other companies are able to search technical solutions useful for them, which will advance out-licensing and sell-off activities, when possible buyers could contact to NRC instead of NRC searching for them. Many large corporations, such as IBM, Philips and DuPont that have gained remarkable revenue through licensing have their own web pages for marketing their technologies.
52
On the Internet, there are already service providers, such as Yet2.com (Yet2.com, 2006), whose business is based on providing others’ technologies for selling and licensing and helping companies to contact with each other. Those vendors are one possibility to market the surplus technologies, if the database of the surplus technologies is chosen not to be open only for internal personnel.
5.7 Management of RSP
5.7.1 Packaging of the Surplus
When the development of some new technology ends and a project is cancelled for example due to changes in the environment or the strategy, or developed technology is not used in Nokia’s businesses, research results and other material are non-core material to NRC. At that point, a project (or a technology) will be packaged to RSP. In other words, it is stored into a database. A project manager fills in a questionnaire, where all information that is wanted to store will be asked. The questionnaire could be for example like a project proposal template, but of course if the purpose is to store technologies, it is not reasonable to use the project proposal templates. In some cases, the research surplus may be at a stage where proposal has not even been made, but if there is a proposal it could be employed in the data storage phase. Usually, the project proposals contain for instance evaluation of costs, value and risks.
You cannot manage what you do not know. Information that is gathered from surplus has to be enough descriptive to allow the surplus to be analyzed and new opportunities to be identified. Following mind map (Figure 21) gives some suggestions about attributes that could be stored, but the final decision about data, collection methods and place will be made by a RSP implementation team. The key is to choose the attributes that will yield information useful to researchers. In appendix 1, there is a more detailed list of the key characteristics of projects, which could be stored to the database.
53
Figure 21. Attributes of research surplus
Collectable attributes are divided into the five categories: general data, project information, technical data, market opportunities and results from analyses. Because the amount of the research surplus from the whole NRC will be huge, the identification of a single project or a technology is essential. General data is for that purpose. Project information gives basic background information, too. Technical information is needed so that researchers can observe the features of the surplus technologies. Some technology evaluating methods can be integrated into the RSP system. Results from those evaluations are one part of the saved information. Information about the potential market is very important. A market research can also be an evaluation tool, but the description of the business idea and the planned market for a surplus technology helps to determine the future use of a technology. If
54
information about the future opportunities is not available at the moment of documentation, the information could be added later and it does not necessarily have to be the project manager who identifies the market opportunities. There could also be a possibility to suggest ideas about the future use of the non-core technology.
Proper management of RSP and especially data storage phase is critical for the later opportunities of the research surplus. When it is done delicately, there are better changes that a surplus technology is founded from RSP and it can be utilized in the new business creation in NRC or licensed out of the company. To make sure that data is documented carefully, there could be some kind of bonuses or rewards for a team or a laboratory, whose surplus is the most utilized in the new business creation or whose surplus has created most value to the company.
5.7.2 Toolkit for the Evaluation of the Surplus
Tools for the research surplus management and evaluation are hard to find. This is a common problem for the portfolio management and the whole innovation process. Cooper (1997b, 169) argues that the evaluation methods must be user-friendly, but still realistic. Many tools are too complex to use or they contain too many simplifying consumptions. The relations between technologies, changing conditions, measurement of critical success factors and uncertainty increase the difficulty of evaluation.
Evaluation of technologies in RSP is done from different basis than in the project portfolio management (PPM) process. Project portfolios contain a firm’s current projects and technology portfolios current technologies. The normal PPM process aims to allocate scarce resources for the most beneficial projects and balance the portfolio of projects to fit the firm’s strategy. RSP does not have a role like that. Evaluation tools and techniques applied to RSP have to help to understand portfolio content, find new market opportunities and identify technologies that can be licensed out.
55
Chapters 4.2 and 4.3 introduced a collection of tools for managing portfolios and assessing technologies. In this chapter, tools, which are considered to be the most appropriate for the evaluation of the surplus, are discussed. The methods are evaluated based on how well they fit in the RSP concept, so the general benefits and pitfalls of the methods are not considered in the thesis. Table 5 summarizes the discussion.
Probably the most suitable method for the RSP concept is the scoring model. It is a simple, easy and effective tool. It allows a subjective analysis, which is useful feature for RSP, but the danger is that the analysis becomes too subjective. Another pitfall is that it is very hard to identify criteria that measure right issues, and weight those criteria right. Therefore, the criteria and their weights must be selected carefully. With scoring model every kind of surplus can be evaluated. It is an essential feature for the evaluation tool, because as mentioned before the surplus can be in every form from the patent to the code. Cooper et al. (2001, 369) categorize criteria used in scoring models into five sections: reward, business strategy, strategy leverage, probability of commercial success and probability of technology success. The categories are almost the same than the attribute suggestions in the previous chapter (Figure 21). Below, there is a list of criteria suggestion in every section that could be applied for the surplus:
Reward: NPV, payback time and other financial measures
Business strategy fit: how far a technology is from the strategy, the financial and
strategic impact of technology
Strategic leverage: proprietary position, growth opportunities, durability, synergies with
other technologies/programs
Probability of commercial success: market opportunities, competitive advantage,
market maturity, commercial/later use assumption
Probability of technology success: stage of development, technology complexity,
technical skill base
56
It is very likely that the research surplus contains real options that have not been wanted to be used or have not been founded yet. In the latter case, identification of those options should be done. The future options identification is basically the purpose of the whole RSP. The problem is that options are hard to define and value. Even if the real options approach is reasonable and useful in theory, there are not many workable applications in practice.
A major reason why new product development fails is the lack of market knowledge: inadequate market research and not understanding customers needs and wants (Cooper, 1997b, 43). In this context, market research does not necessarily mean an ordinary market research that normally is made with the final customers of the new product. Market opportunity identification helps to find new market for the surplus technologies and new businesses based on the research surplus. The importance of strong market orientation can not be emphasized enough, but off course, it is not the only thing that matters. Other analyses are needed, too. In the thesis, the form of market research application is not considered in more detail, but it is essential to integrate it into RSP.
The broadly applied SWOT analysis about strengths, weaknesses, opportunities and threads could be applied in the surplus technologies evaluation, too. The traditional method offers a framework for the documentation and evaluation of the different aspects of the surplus. Technologies are evaluated internally (strengths and weaknesses) and externally (opportunity and threats) and, on the other hand, positive aspects (strengths and opportunities) and negative aspects (weaknesses and threats) are eyed. But the execution of a thorough analysis requires time and usually more than a one person. The subjective analysis is suitable for RSP, but if the analysis just lists strengths, weaknesses, opportunities and threats of a technology, the view might be too subjective.
PPM-solutions providers highlight the importance of a bird-eye view from all projects in the portfolio. Usually, bubble diagrams, different tables and maps are used. Even if those tools are meant primarily for strategic fit and balance check, which are not important in RSP, the bird-eye view can be applied to RSP, too. The front page of the RSP solution can provide general information, such as how many technologies are there in RSP, how many
57
from different laboratories and recently stored technologies. In addition, a diagram tool could be helpful in the technology search, if the tool would draw technologies to a coordinates with selected dimensions. It is possible that with some metrics, RSP could be divided into clusters.
The financial methods are the most popular tools for evaluating projects and technologies, but those are not very appropriate for the technologies in RSP. The financial tools are well known and can assist discussion and decisions about the new technology development, but if the results are wanted to be reliable, plenty of accrual financial information, such as cash flows from several becoming years, is needed. In many cases, information is impossible to get. If the financial information is available or some kind of financial analysis has already been done for example in project proposal, it can be included into RSP.
58
Table 5. Methods for evaluating the future potential of research surplus
Method Suitable for RSP Not suitable for RSP
+ Effective, simple and easy to use + Subjective assessment can be made
+ Qualitative and quantitative aspects are considered
+Lists questions for discussion
+ Suitable for evaluation of every kind of surplus
+ Identifies new business opportunities
+ usually used too little in companies
- Requires well defined
criteria/weights for criteria
- Too subjective approach
- Not a simple method for
implementation
- Options are difficult to define
- Valuation of the benefits of options
is hard
- Only market aspect
- Time consuming
- Too subjective approach
- Hard to evaluate immature
technology
- used mostly to visualize balance of portfolio
- Requires accrual financial data, which can be impossible to get - Probably negative outcome from the analysis
The methods introduced above do not rule each other out. There could and should be more than a one method integrated to RSP. The methods can be combined in several ways. For example the combination of scoring model and bubble diagram applications gives wider results from the evaluation and do not require extra work.
In the thesis, metrics and criteria for the methods are not discussed. To identify tools and metrics best suitable for the research surplus evaluation, workshop approach, like Delphi
59
method, could be used. Delphi method is based on an expert group, whose size is recommended to be 10-18 persons, answering anonymously a questionnaire designed to solve a problem. (Okoli & Pawlowsky, 2004, 19) The group could include for instance NRC’s researchers, managers, academic researchers and consultants.
5.7.3 Managing RSP
It is vital for communication and functionality that the RSP management team is a cross-functional group with researches that present the technical aspect, marketing/business personnel with the knowledge of market opportunities and legal counselors for understanding protection and enforcement issues. The team consists of members from as many laboratories as possible. In addition, there has to be a so called “knowledge broker” in a management team, who works as a link between project managers and RSP. The team does not have to be large; few persons could take the responsibility of RSP and searching new technologies from it.
The RSP management team is responsible for maintaining the portfolio. Even if data can be stored whenever projects become surplus and technologies can be searched and observed anytime, a seasonal reviewing is useful to do, at least when the strategic focus areas change or when big environmental changes occur. The portfolio could be gone through systematically with experts one to three times a year. The reviewing sessions could also be symposia for sharing information about the recently ended projects. Table 2 (chapter 3.3.4) gives some advice to management team how to reactivate the surplus technologies.
The portfolio can be open to everybody in NRC. Every researcher is free to observe non-core technologies, but the system is hierarchical. It means that only certain employees, such as project managers, are able to update the system. The thesis does not consider the data security and protection issues, because Nokia and NRC have their own methods and systems for managing those issues.
60
5.8 RSP Database
In this chapter, a short view to the database of RSP is given. The clearest alternative is to separate core technology and non-core database from each other. This is essential, if RSP is used as a market place for non-core technologies and some part of it will be opened outside NRC. If the database stays the way it is now, all projects are in the same database, the surplus has to be easy to recognize and distinguish from the core. However, a strong linkage between core technologies and surplus has to be kept no matter what kind of storage solution is chosen (Figure 22).
Core technologies can become non-core and should be moved into RSP. It might happen for example to old technologies that have been used to Nokia’s products but are now considered to license out. At the same time, non-core technologies can become core. The example of this could be open source technologies that might become core in some circumstances.
At the moment, projects and patents are documented into different databases, which are not linked with each other. While developing RSP, the linkage between patents, RSP and core technologies should be created (Figure 22). The utilization of RSP can benefit from the linkage, because from the database of the patents a person who is observing the surplus is able to estimate the commercial value of a non-core technology better. (Saarinen, 2006)
Figure 22. The connections between the databases of core technologies, patents and RSP
61
5.9 Implementation
RSP will be some kind of a computer system. The thesis does not include deeply consideration of the implementation of RSP, but some topics are discussed in this chapter and the appendix 2 gives the simplified example of the RSP system in practice. Following ideas for RSP are common suggestions and arguments that have been captured from the literature and different solution providers on the Internet.
There are three main issues that have to be considered when implementing the RSP system: functionality, technology and costs. If the software for the system is decided to be developed in-house, operating systems, databases, and programming languages have to be considered. Even if there are plenty of project portfolio management solution vendors, normal PPM-solutions are not suitable for research surplus management. So in this case, in-house solution may be the best alternative. The integrations of RSP into NRC’s other data systems must be considered, too.
Implementation requires a task force that consists from five to nine persons. Cooper (1997, 268) argues that it is the most appropriate size for the development team in implementing new innovation process. Two to three persons do not provide enough perspective and on the other hand, with over ten persons, it is hard to schedule meetings. Before actual implementation phase, it is essential to do proper detailed design. In addition, it is important to discuss with other personnel about the concept of RSP. They might give good suggestions and ideas related to RSP.
In any organizational change, the resistance of change is usually tough, especially in situations where workload or control of the work will increase. RSP does the both; workload will increase because the surplus has to be stored in the system and maintained, and from RSP, executives and other laboratories can observe how much surplus a laboratory or even a particular person has produced. The implementation of RSP does not succeed without the dedicated implementation team.
62
Commitment to use RSP is gained with communication and training. The development team must convince the organization that RSP is needed. Like in every reform, commitment of top management is important, too. User-friendly manual and a few training sessions ensure that the purpose of RSP is understood and it will be employed properly. In many cases, if the organization is big, like NRC, the pilot version of a new system is used. MMT laboratory is a good place to test RSP before it is extended into the whole organization.
After RSP is implemented, generated surplus can be documented into it anytime, but how the current research surplus material that already exists is moved to the new portfolio has to be considered during the implementation. There have to be general instructions about how old research material is still relevant enough for RSP, who can be responsible for the information transfer and when the surplus should be filed.
63
6 CONCLUSIONS
Changing business environment with shorter lifecycles, globalization and more intensive competition force companies to be more innovative, and look for new business models. The open innovation that emphasizes more open attitude in a company’s innovation process, collaboration and alternative paths to market, is adopted as an innovation philosophy in many industries. The development requires new pursuit patterns from R&D divisions. The goal of this thesis was to create the constructive concept to one of those new patterns in NRC - Research Surplus Portfolio. The purpose of the concept is to move human capital into intellectual assets that can be managed. The main research question was: what kind of system is suitable and effective for managing the research surplus (non-core technologies) in NRC. The second purpose of the thesis was to identify tools and techniques for RSP.
In the thesis, the framework for RSP is constructed. The thesis designs the concept of the portfolio and gives suggestions to how to utilize and manage it in NRC. The framework gives guidelines for the research organization to implement the portfolio for their non-core technologies better and advance the new business creation from the surplus. From the constructive approach point of view, the thesis introduces the term “Research Surplus Portfolio”, which is new to the portfolio management literature, and creates the framework for it. Even if models and tools are pretty similar to the ordinary project portfolio management, the essence of RSP is so different from the project portfolio that new management processes and procedures have to be considered. Table 6 collects the main differences between PPM and RSP management issues.
64
Table 6. The comparison of the project portfolio management and RSP management
PPM Management RSP Management
Organizational approach Top-down or bottom-up approach Bottom-up approach
Goals Value maximization, balance,
strategic fit
Management process Selection, evaluation, prioritization, resource allocation, balancing the portfolio and strategic fit consideration, decision making (go/kill decisions), maintaining Effective use of innovations, finding opportunities, new business creation
Storage, evaluation, maintaining, technology search
Time perspective Present projects Future opportunities
Project selection Projects are not placed into the portfolio automatically. Selection is based on the evaluation of value, strategic fit and balance
Projects prioritizing Projects are prioritized for resource
allocation
Evaluation methods Combination of financial method,
strategic approach, scoring model and bubble diagram All surplus projects will be placed to the portfolio
No need for prioritization or resource allocation
Combinations of scoring model, bubble diagram and market research
The approach to the project management could be either bottom-up or top-down, but in RSP management, the approach needs to be bottom-up, because project managers and researchers know their needs and what they have done the best. There are three goals for the ordinary project portfolio management: value maximization, balance and strategic fit, and the portfolio management processes are designed to meet these goals. The RSP management has different objectives: the increased utilization of research resources, increased innovativeness and new business creation. But even if the goals of the RSP management differ from the goals of the project portfolio management, some PPM tools and techniques are suitable for RSP management, too. For example, the technology evaluation methods in both systems are similar. Overall, RSP management process is much lighter than PPM process. It concentrates on the future opportunities and deals with the more complex and uncertain environment.
65
NRC produces a lot of research material from numerous projects. Typically, the information documentation and sharing is a challenge for research organizations. That is why RSP concept fits to NRC well. It is clear that it would ease the management of non-core technologies, but in addition, it could help the new business creation, as well. RSP could offer several benefits for the research organization:
New business opportunities
Applications for out-licensing
Advancement of venture and spin-off activities
Visibility and transparency
Increased managerial efficiency
Increased utilization of resources and technologies
Learning
Increased innovativeness
Moving towards open innovation paradigm
RSP could be used in several ways. The thesis gives four examples on how to use the portfolio: It could function as the technology search engine, the idea bank, the communication tool and the market place for technologies. Its utilization depends on whether the system is internal or external. If it was created just for the internally use, an efficient search engine application might be enough. Then RSP would function as a researchers’ extra memory, from where earlier projects could be reviewed. However, if the purpose of RSP is to create new businesses from the research surplus or market the waste technologies outside the company, RSP must be developed further.
To make the system functional, RSP solution must be simple and it has to have the approval of the executives and other employees. When assessing the usability of the RSP, the most important elements are the documentation of the surplus to the portfolio and
66
searching it from that. Documentation is done by project managers. It is essential that it becomes a part of their basic work and a routine task during a research project. The attributes stored from the surplus need to be categorized and evaluated effectively. The organization has to name a few persons, who review the database regularly and are responsible for RSP. Advanced searching facility must be applied to RSP. It should be able to accept several different search words and search types.
One problem in the utilization of RSP is that the reviewing of the research waste is certainly not the core activity in NRC. Even if an effective application for RSP is found, decision making has still strong human aspect that should not be forgotten. How much Research Surplus Portfolio will be exploited and new opportunities from the waste be found, depends on the organization and the people behind RSP. The danger is that it becomes an unused data system. It is clear that RSP demands at least little extra work, and looking back may seem to be worthless, but benefits from exploring old research results could be significant, if new business could be created from already wasted research.
In the thesis, the first step towards the implementation of Research Surplus Portfolio is taken. The next step will be a more detailed design of the portfolio system and its testing, first in paper and then in practice. Further research is needed for instance to select the evaluation tools, construct the suitable applications with metrics to RSP and design the physical entity in more detail. Hargaron & Sutton (1997) discuss the innovation process with technology brokering. The results of the thesis could also be concerned from their survey’s point of view and examine possibilities to combine their survey and the results of the thesis.
The contraction of RSP framework has been made with close cooperation with different persons in NRC. That is why it can be argue that the result is reliable. If the usefulness of the framework constructed in the Thesis is validated with the weak market test (Kasanen et al.1993), it is justified to state that it is gained. Responsible manager has expressed the usefulness of the RSP framework for the further development of the management system of
67
the research surplus. On the other hand, developed framework is constructed in general level and it is easy to generalize to other research organizations as well.
68
REFERENCES
Aalto, T. 2001. Strategies and Methods for Project Portfolio Management. In Artto, K., Martinsuo, M., & Aalto, T. (eds) Project Portfolio Management: Strategic Management through Projects. Project Management Association Finland, Helsinki. 176 p.
ISBN: 951-22-5594-4
Allio R. 2005. Interview with Henry Chesrough: Innovating Innovation. Strategy & Leadership, Chicago, vol.33, iss.1, pp. 19-25
Archer, NP. & Ghasemzadeh, F. 1999. An Integrated Framework for Project Portfolio Selection. International Journal of Project Management, Kidlington, vol. 17, iss. 4, pp. 207-216
Berkhout, A.J., Hartmann, D., Van der Duin, P. & Ortt, R. 2006. Innovating the Innovation Process. International Journal of Technology Management, Geneva, vol. 34, iss 3/4, pp. 390-404
Boer, F.P. 1999. The Valuation of Technology: Business and Financial Issues in R&D. John Wiley & Sons, Inc, USA. 403 p. ISBN: 0-471-31638-5
Boer, F.P. 2000. Valuation of Technology Using “Real Options”. Research Technology Management, Washington, vol. 43, iss. 4, pp. 26-30
Bordley, R. 1998. R&D Project Selection versus R&D Project Generation. IEEE Transactions and Engineering Management, New York, vol. 45, iss. 4, pp. 407-413
Chesbrough, H. 2003a. Open Innovation: The New Imperative for Creating and Profiting
from Technology. Harvard Business Press, Boston, Massachusetts. 227 p.
ISBN 1-57851-837-7
69
Chesbrough, H. 2003b. The New Rules of R&D.Harvard Management Update, Boston, vol. 8, iss. 5, pp. 3-4
Chesbrough, H. 2003c. The Logic of Open Innovation: Managing Intellectual Property. California Management Review, California, vol. 45, iss. 3, pp. 33-58
Chesbrough, H. 2004. Managing Open Innovation. Research Technology Management, Washington, vol. 47, iss. 1, pp. 23-26
Chesbrough, H. 2006. Open Innovation: A New Paradigm for Understanding Industrial Innovation. In Chesbrough, H., Vanhaverbeke, W. & West, J. (eds). Open Innovation: Researching a New Paradigm. Oxford University Press, Oxford. 400 p. ISBN 0-19¬929072-5
Chien, C. 2002. A Portfolio-evaluation Framework for Selecting R&D Projects. R&D Management, Oxford, vol. 32, iss. 4, pp. 359-368
Christensen, J., Olesen, M. & Kjaer, J. 2005. The Industrial Dynamics of Open Innovation – Evidence from the Transformation of Consumer Electronics. Research Policy, Amsterdam, vol. 34, iss. 10, pp. 1533-1549
Cohen, W. & Levinthal, A. 1990. Absorptive Capacity: A New Perspective on Learning and Innovation. Administrative Science Quarterly, Ithaca, vol. 35, iss. 1, pp. 128-153
Cooper, R. 1990. State-Gate System: A New Tool for Managing New Products. Business Horizons, Greenwich, vol. 33, iss. 3, pp. 44-55.
Cooper, R., Edgett, S. & Kleinschmidt, E. 1997a. Portfolio Management in New Product Develompent: Lessons from the Leaders II. Research Technology Management, Washington, vol. 40, iss. 6, pp. 43-52
70
Cooper, R. 1997b. Winning at New Products: Accelerating the Process from Idea to
Launch. Second Edition. Addison-Wesley Publishing Company, Inc., USA. 358 p.
ISBN: 0-201-56381-9
Cooper, R., Edgett, S. & Kleinschmidt, E. 1999. New Product Portfolio Management: Practices and Performance. The Journal of Product Innovation Management, New York. vol. 16, iss. 4, pp. 333-351
Cooper, R., Edgett, S. & Kleinschmidt, E. 2000. New Problems, New Solutions: Making Portfolio Management More Effective. Research Technology Management, Washington, vol. 43, iss. 2, pp. 18-33
Cooper, R., Edgett, S. & Kleinschmidt, E. 2001. Portfolio Management for New Product Development: Results of an Industry Practices Study. R&D Management, Oxford, vol. 31, iss. 4, pp. 361-380
Doering, D.S. & Parayre, R. 2000. Identification and Assessment of Emerging Technologies. In Day, G & Shoemaker, P. (eds). Wharton on Managing Emerging Technologies. John Wiley & Sons, Inc., USA. 460 p. ISBN: 0-471-36121-6
Dooley, L., Lupton, G. & O’Sullivan, D. 2005. Multiple Project Management: A Modern Competitive Necessity. Journal of Manufacturing Technology Management, Bradford, vol. 16, iss. 5/6. pp. 466-482
Dye L. & Pennypacker, J. (eds). 1999. Project Portfolio Management: Selecting and Prioritising Projects for Competitive Advantage. Center for Business Practices, West Chester. 421 p. ISBN: 1-929576-00-5
Edvinsson, L & Sullivan, P. 1996. Developing a Model for Managing Intellectual Capital. European Management Journal, London, vol. 14, iss. 4, pp. 356-364
71
Foreier, B. 2002. Roadmapping: Critical Thinking for Interconnect Products and Technologies. CircuiTree, Troy, vol. 15, iss. 9, pp. 50-61
Garud, R. & Nayyar, P. 1994. Transformative Capacity: Continual Structuring by Intertemporal Technology Transfer. Strategic Management Journal, Chichester, vol. 15, iss. 5, pp. 365-385
Goldheim, D., Slowinski, G., Daniele, J., Hummel, E. & Tao, J. 2005. Exracting Value from Intellectual Assets. Research Technology Management, Washington, vol. 48, iss. 2, pp. 43-48
Gulati, R. 1998. Strategic Alliances and Networks. Strategic Management Journal, Chichester, vol. 19, iss. 4, pp. 293-317
Hamilton, W. 2000. Managing Real Options. In Day, G & Shoemaker, P. (eds.) Wharton
on Managing Emerging Technologies. John Wiley & Sons, Inc., USA. 460 p.
ISBN: 0-471-36121-6
Hargadon, A.B. & Sutton, R.I. 1997. Technology Brokering and Innovation in Product Development Firm. Administrative Science Quarterly, Ithaca, vol 42, iss. 4, pp. 716-749
Harrison, S. & Sullivan, P. 2000. Profiting from Intellectual Capital: Learning from Leading Companies. Industrial and Commercial Training, Guilsborough, vol. 32, iss. 4, pp. 139-148
Henderson, B. 1970. Growth-Share Matrix. The Boston Consulting Group. 2006. [Internet document][Accessed:October12th]Available:
http://www.bcg.com/this_is_bcg/mission/growth_share_matrix.html
Hogan, J. 2005. Open Innovation or Open House: How to Protect Your Most Valuable Assets. Medical Device Technology, Chester, vol. 16, iss. 3, pp. 30-31
72
Kasanen, E., Lukka, K. & Siitonen, E. 1993. The Constructive Approach in Management Accounting Research. Journal of Management Accounting Research, Sarasota, vol. 5. pp. 343-264
Koen, P., Greg, A., Boyce, S., Clamen, A., Fisher, E., Fountoulakis, S., Johnson, A., Puri, P. & Seibert, R. 2002. Fussy Front End: Effective Methods, Tools and Techniques. In Belliveau, P., Griffin, A. & Somermayer, S. The PDMA Toolbook for New Product Development. John Wiley & Sons, Inc., USA. 480 p. ISBN: 0-471-20611-3
Liito. 2006. Liito – Uudistuva liiketoiminta ja johtaminen. [Internet document] [Accessed: June 27th 2006] available: http://www.tekes.fi/liito/.
Lukka, K. 2000. The Key Issues of Applying the Constructive Approach to Field Research. In Reponen, T. (ed.). 2000. Management Expertise for the New Millenium. In Commemoration of the 50th Anniversary of the Turku School of Economics and Business Administration. Publications of the Turku School of Economics and Business Administration, A-1:2000
MacMillan, I. & McGrath, R. 2002. Crafting R&D Projects Portfolios. Research Technology Management, Washington, vol. 45, iss. 5, pp. 48-59
Martinsuo, M. 2001. Project Portfolio Management: Contingencies, Implementation and Strategic Renewal. In In Artto, K., Martinsuo, M. & Aalto, T. (eds). Project Portfolio Management: Strategic Management through Projects. Project Management Association Finland, Helsinki. 176 p. ISBN: 951-22-5594-4
Miller, W. L & Morris, L. 1999. Fourth Generation R&D: Managing Knowledge, Technology, and Innovation. John Wiley & Sons, Inc., USA. 347 p. ISBN 0-471-24093-1
Nokia, 2006a. Nokia-Research. [Internet document] [Accessed: August 7th 2006]
73
available: http://www.nokia.com/link?cid=EDITORIAL_4070.
Nokia. 2006b. NRC General Presentation/February 2006. Power-point presentation slides.
Nokia. 2006c. Nokia and Open Source. [Internet document] [Accessed: August 7th 2006] available: http://opensource.nokia.com/contributions.html
Okoli, C & Pawlowsky, S. 2004. The Delphi Method as a Research Tool: An Example, Design Considerations and Applications. Information & Management, Amsterdam., vol 42, iss. 1, pp. 15-29
Parhankangas, A., Holmlund, P. & Kuusisto, T. 2003. Managing Non-Core Technologies: Experiences from Finnish, Swedish and US Corporations. Technology Review 149/2003. Tekes, Helsinki. 77 p. ISBN 952-457-142-0
Poskela, J., Korpi-Filppula, M. & Mattila, V. 2001. Project Portfolio Management Practices of a Global Telecommunications Operator. In Artto, K., Martinsuo, M. & Aalto, T. (eds). Project Portfolio Management: Strategic Management through Projects. Project Management Association Finland, Helsinki. 176 p. ISBN: 951-22-5594-4
Rothwell, R. 1992. Successful Industrial Innovation: Critical Factors for the 1990s. R&D Management, Oxford, vol. 22, iss. 3, pp. 221-240.
Saint-Onge, H. 1996. Tacit Knowledge: The Key to the Strategic alignment of Intellectual Capital. Strategy & Leadership, Chicago, vol. 24, iss. 2, pp. 10-14
Schoemaker, P. 1995. Scenario Planning: A Tool for Strategic Thinking. Sloan Management Journal, Massachusetts, vol.36, iss. 2, pp. 25-40
74
Shoemaker, P. & Mavaddat, V. 2000. Scenario Planning for Disruptive Technologies. In Day, G & Shoemaker, P. (eds). Wharton on Managing Emerging Technologies. John Wiley & Sons, Inc., USA. 460 p. ISBN: 0-471-36121-6
Spradlin, T. & Kutoloski, D. 1999. Action-Oriented Portfolio Management. Research Technology Management, Washington. vol. 42, iss. 2, pp. 26-32
Stevens, T. 1997. Balancing Act. Industry Week, Cleveland, vol. 246, iss. 6, pp. 40-44
Stewart, T. 1994. Your Company’s Most Valuable Asset: Intellectual Capital. Fortune, New York, vol. 130, iss. 7, pp. 68-73
Stewart, T. 2001. Ten Years Later, How Far We’ve Come. Fortune, New York, vol. 143., iss. 11, pp. 192-194
Sullivan, P. 1998. Profiting from Intellectual Capital: Extracting Value from Innovation. John Wiley & Sons, Inc., USA. 366 p. ISBN: 0-471-19302-X
Sullivan, P. 2000. Value-Driven Intellectual Capital: How to Convert Intangible Corporate Assets into Market Value. John Wiley & Sons, Inc., USA. 276 p. ISBN: 0-471-35104-0
Tao, J., Daniele, J., Hummel, E., Goldheim, D. & Slowinski, G. 2005. Developing an Effective Strategy for Managing Intellectual Assets. Research Technology Management, Washington, vol. 48, iss. 1, pp. 50-58
Tidd, J., Bessant, J. & Pavitt, K. 2001. Managing Innovation: Integrating Technological, Market and Organizational Change. Second Edition. John Wiley & Sons, Inc., England, 388 p. ISBN 0-471-49615-4
von Hippel, E. 1994. The Sources of Innovation. Oxford University Press, USA. 232 p. ISBN: 978-0-19-509422-0
75
Yet2.com. 2006. Company’s homepage [Internet document] [Accessed: September 28th 2006] available: http://www.yet2.com/app/about/home
Interviews
Saarinen, J. 2006. Head of Multimedia technology laboratory in NRC. Tampere 7.9.2006
Karlsson, M. 2006a. Project Manager in NRC. E-mail 3.8.2006
Karlsson, M. 2006b. Project Manager in NRC. E-mail 8.8.2006
76
APPENDIX 1: List of Key Characteristics of Surplus
General data:
Project name
Code number
Laboratory
Class (audio, video, games...)
Type (licensing case, parked technology, idea...)
Status (defensive, offensive, breakthrough)
Documentation day
Modification day
Contacts (inventor, project manager..)
Competitors
Key words
Project information:
Duration
Start date
Financier(s)
Problems/ Kill variables
Costs
Stage of the development
Size of the project (resources, personnel)
Technical information:
Description/abstract
Related projects/technologies
Status of the legal protection
Related patents/invention publications
Deliverables (results, materials)
Source of idea
Market opportunities:
Suggestions for later use
Feasibility
Market attractiveness
Business/utilization idea
Competitive advantage
Link to the business strategy
Market segment
Competitive impact of technology
Market position
Evaluation results and probabilities:
Rating (0-10) on key criteria
Ranking (1-N) based on some ranking criteria
Attractiveness scores
Related financial analyses: NPV, ECV, IRR, Payoff...
Probability of technical success
Probability of commercial success
Market research (for example opportunity analysis based on Delphi method)
Risk analysis (scenario planning, sensitive analysis, Monte Carlo simulation)
SWOT
Real options (definition, valuation)
Bubble diagrams/maps (could be drawn from other information gathered)
Other methods and analyses (Business case analysis, investment opportunity analysis, trend analysis)
APPENDIX 2: Example of the RSP system in practice
This simplified example gives an overview about RSP system in practice. Note that, the example illustrates only a one possible search event.
1. The most important element of RSP is the searching function. It should be very sophisticated and efficient. It should allow very different searches and search words, such as certain time period, key words, certain stored attribute or searches based on evaluation. The search facility is related to the other databases, too; the system also allows searching core technologies and patents.
2. The front page of the interface of the portfolio could contain general information. It could summarize how much surplus is there in the database, latest documentations, latest results from the utilization and other current news.
3. Some kind of graphics to support the general information would increase the usability and intelligibility. It could be useful to add a function, which allows changing the dimensions of the diagram.
4. In this case, non-core technologies that are produced over the last five years are searched. The system shows a list of those surplus items and basic information about them.
5. A searcher can review a single technology in more detail and get more information about it. The goal is that this page would contain all the information that is needed to identify the future opportunities. From this page, the managers and researchers have access to review technical information, related material and, patents and invention publications related to the particular technology.
Automatic Vandalism Detection in Wikipedia:
Towards a Machine Learning Approach
Koen Smets and Bart Goethals and Brigitte Verdonk
Department of Mathematics and Computer Science
University of Antwerp, Antwerp, Belgium
{koen.smets,bart.goethals,brigitte.verdonk}@ua.ac.be
Abstract
Since the end of 2006 several autonomous bots are, or have been, running on Wikipedia to keep the encyclopedia free from vandalism and other damaging edits. These expert sys¬tems, however, are far from optimal and should be improved to relieve the human editors from the burden of manually reverting such edits. We investigate the possibility of using machine learning techniques to build an autonomous system capable to distinguish vandalism from legitimate edits. We highlight the results of a small but important step in this di¬rection by applying commonly known machine learning al¬gorithms using a straightforward feature representation. De¬spite the promising results, this study reveals that elemen¬tary features, which are also used by the current approaches to fight vandalism, are not sufficient to build such a system. They will need to be accompanied by additional information which, among other things, incorporates the semantics of a revision.
Introduction
Since the inception of Wikipedia in 2001, the free encyclo¬pedia, which is editable by everyone, has grown rapidly to become what it is today: one of the largest sources of ad¬equate information on the Internet. This popularity trans¬lates itself to an ever growing large amount of articles, readers consulting them, editors improving and extending them ... and unfortunately also in the number of acts of van¬dalism committed a day. By vandalism we understand every edit that damages the reputation of articles and/or users of Wikipedia. Priedhorsky et al. (2007) provide a survey of the typical categories of damages together with an empiri¬cally determined likeliness of occurrence. We list them here in decreasing order of appearance: introducing nonsense, offenses or misinformation; the partial deletion of content; adding spam (links); mass deletion of an article ...
To fight vandalism, Wikipedia relies on the good faith of its users that accidentally discover damaged articles and, as in practice turns out, on the time-consuming efforts of its ad¬ministrators and power users. To ease their job, they use spe¬cial tools like Vandal Fighter to monitor the recent changes and which allow quick reverts of modifications matching regular expressions that define bad content or are performed
Copyright c 2008, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
by users on a blacklist. Since the end of 2006 some vandal bots, computer programs designed to detect and revert van¬dalism have seen the light on Wikipedia. Nowadays the most prominent of them are ClueBot and VoABot II. These tools are built around the same primitives that are included in Van¬dal Fighter. They use lists of regular expressions and consult databases with blocked users or IP addresses to keep legit¬imate edits apart from vandalism. The major drawback of these approaches is the fact that these bots utilize static lists of obscenities and ‘grammar’ rules which are hard to main¬tain and easy to deceive. As we will show, they only detect 30% of the committed vandalism. So there is certainly need for improvement.
We believe this improvement can be achieved by apply¬ing machine learning and natural language processing (NLP) techniques. Not in the very least because machine learning algorithms have already proven their usefulness for related tasks such as intrusion detection and spam filtering for email as well as for weblogs.
The remainder of this paper is as follows. First, we give a brief overview of related work, followed by a motivation for using machine learning to solve the problem. Next, we complement the most recent vandalism studies by dis¬cussing the performance results of the bots currently active on Wikipedia. Thereafter, we present the preliminary results of using a Naive Bayes classifier and a compression based classifier on the same features that serve as raw input for those bots. Finally, we formulate conclusions and outline the approach we plan to investigate next.
Related Work
Wikipedia has been subject to a statistical analysis in sev¬eral research studies. Vi´egas, Wattenberg, and Dave (2004) make use of a visualization tool to analyze the history of Wikipedia articles. With respect to vandalism in particular, the authors are able to (manually) identify mass addition and mass deletion as jumps in the history flow of a page. Buriol et al. (2006) describe the results of a temporal analysis of the Wikigraph and state that 6 percent of all edits are reverts and likely vandalism. This number is confirmed by Kittur et al. (2007) in a study investigating the use of reverting as the key mechanism to fight vandalism. They also point out that only looking for reverts explicitly signaling vandalism is not strict enough to find evidence for most of the vandal
43
ism in the history of articles. The most recent study, to the best of our knowledge, by Priedhorsky et al. (2007) catego¬rizes the different types of vandalism and their occurrence rate in a subset of 676 revision chains that were reverted. They confirm that reverts explicitly commented form a good approximation to spot damages, with a precision and recall of respectively 77% and 62%. Our work complements this last one, as we investigate a yet more recent version of the English Wikipedia history, and also analyse the decisions made by two bots. We also try to respond to the authors’ re-quest to investigate the automatic detection of damage. The authors believe in intelligent routing tasks, where automa¬tion directs humans to potential damage incidents but where humans still make the final decision.
There is strong cross-pollination possible between Wikipedia and several research areas. Wikipedia can ben¬efit from techniques from the machine learning, information retrieval and NLP domains in order to improve the quality of the articles. Adler and de Alfaro (2007) build a content-driven system to compute and reflect the reputation of au¬thors and their edits based on the time span modifications re¬main inside an article. Priedhorsky et al. (2007) use a closely related measure but they do not take into account the lifetime but the expected viewing time to rate the value of words. Rassbach, Pincock, and Mingus (2007) explore the feasibil¬ity of automatically rating the quality of articles. They use a maximum entropy classifier to distinguish six quality classes combining length measures, wiki specific measures (number of images, in/out links ... ) and commonly used features to solve NLP problems (part-of-speech usage and readability metrics). The problem to detect damages is related to ours in the sense that we need to rate the quality of a single revision instead of the whole article. The cross-pollination also holds for the other way around as machine learning, information retrieval and NLP can benefit from the use of Wikipedia. Gabrilovich and Markovitch (2007) use an explicit seman¬tic interpreter built using articles from Wikipedia which is capable of measuring the semantic relatedness between text documents.
Recently, Potthast, Stein, and Gerling (2008) also use ma¬chine learning to detect vandalism in Wikipedia. Compared to their work, we have a larger labeled data set, use different classifiers, and most importantly, use different features. We aim to summarize an edit by focusing on the difference be¬tween the new and old version of an article, while Potthast, Stein, and Gerling use a set of 15 features that quantify the characteristics of vandalism.
Vandalism Detection and Machine Learning
The particular task to detect vandalism is closely related to problems in computer security: intrusion detection or filter¬ing out spam from mailboxes and weblogs. It is a specific kind of web-defacement, but as the accessibility allows ev¬eryone to contribute, there is no need for crackers breaking into systems. We can see it as a form of content-based access control, where the integrity constraint on Wikipedia enforces that “All article modifications must be factual and relevant” as stated by Hart, Johnson, and Stent (2007). The prob¬lem also shares characteristics intrinsic to computer security
problems. We need to deal with a skew and ever changing class distribution as the normal edits outnumber vandalism and both vandalism and legitimate edits are likely to change, due to respectively the adversarial environment and the rise of new articles or formatting languages.
Machine learning provides state of the art solutions to closely related problems. We put two techniques from the world of spam detection to the test. On one hand we use a well-known Naive Bayes classifier and on the other hand, as results from Naive Bayes models are significantly improved by state-of-the-art statistical compression models, a classi¬fier based on probabilistic sequence modeling provided by Bratko et al. (2006).
Although we are aware that we will not be capable of identifying all types of vandalism (e.g. detecting misinfor¬mation in the pure sense is regarded as impossible without consulting external sources of information), we believe that machine learning might cope with this interesting, but far from trivial, problem.
Performance Analysis of Bots on Wikipedia
In this section we complement the work done by Pried-horsky et al. (2007) by analysing the results of the bots on one hour of data from the English version of Wikipedia. We show that there is still significant room for improvement in the automatic detection of vandalism. Furthermore, we pro¬vide additional evidence that the labeling procedure based on edit reverts, is quite sound. Next, we introduce the Sim¬ple English Wikipedia and present the results of a modified version of ClueBot on this data set, which we also use in our machine learning experiments later on. We start however with a short introduction to ClueBot’s inner working.
ClueBot
ClueBot, written by Carter (2007), uses a number of simple heuristics to detect a subset of the types of vandalism men¬tioned above. First, it detects page replaces and page blanks relying on an auto-summary feature of MedaWiki software. Next, it categorizes mass delete, mass addition and small changes based on absolute difference in length. For the last three types, vandalism is determined by using a manually crafted static score list with regular expressions specifying the obscenities and defining some grammar rules which are hard to maintain and easy to by-pass. Negative scores are given to words or syntactical constructions that seem impos¬sible in good articles, while wiki links and wiki transcludes are considered as positive. The difference between the cur¬rent and the last revision is calculated using a standard diff algorithm. Thereafter, the inserted and deleted sentences are analysed using the score list and if this value exceeds a cer¬tain threshold vandalism is signaled. ClueBot further relies on the user whitelist for trusted users and increases its pre¬cision by only reverting edits done by anonymous or new users.
English Wikipedia (enwiki)
We analyse one hour of data from the first of March 2008 (00:00:00 - 00:59:59), restricting ourselves to the recent
44
45
for evaluation purposes. We do not aim to statistically anal¬yse the different approaches but use it more as a guide to conduct our search towards a machine learning based van¬dalism detection tool.
BOW + Naive Bayes
As a first attempt we use the Naive Bayes implementation from the ‘Bow’ toolkit (McCallum 1996) as learning mech¬anism to tackle the problem. This tool treats each feature as a bag of words and uses Porter’s stemming algorithm and stop word removal to decrease the size of the feature space. Next, we train a Naive Bayes classifier on each of the fea¬tures separately. Our final classifier combines the results of the individual classifiers by multiplying the obtained proba¬bility scores.
Table 5: Censored feature list of revision 29853 from the Vandalism page in Simple Wiki English.
Experiments
In this section, we will discuss the setting for our machine learning experiment conducted on simplewiki, the Simple English version of Wikipedia. We first consider the data representation. Thereafter we give a brief description of two learning algorithms put to test: a Naive Bayes classifier on bags of words (BOW) and a combined classifier built using probabilistic sequence modeling (Bratko et al. 2006), also referred to in the literature as statistical compression mod¬els.
Revision Representation
In this case study we use the simplest possible data repre¬sentation. As for ClueBot and VoABot II, we extract raw data from the current revision and from the history of previ¬ous edits. This first step could be seen as making the static scoring list of ClueBot dynamic. This should provide a base¬line for future work. In particular, for each revision we use its text, the text of the previous revision, the user groups (anonymous, bureaucrat, administrator ... ) and the revision comment. We also experimented with including the lengths of the revisions as extra features. The effect on overall per¬formance is however minimal and thus we discard them in this analysis. Hence the focus lies here more on the content of an edit.
As the modified revision and the one preceding it differ slightly, it makes sense to summarize an edit. Like ClueBot, we calculate the difference using the standard diff tool. Pro¬cessing the output gives us three types of text: lines that were inserted, deleted or changed. As the changed lines only dif¬fer in some words or characters from each other, we again compare these using wdiff. Basically, this is the same as what users see when they compare revisions visually using the MediaWiki software. Table 5 gives an example of the feature representation used throughout this paper, applied to a vandalized revision.
To evaluate our machine learning experiments we use 60% of the labeled data for training and the remaining 40%
Probabilistic Sequence Modeling
Probabilisitic sequence modeling (PSM) forms the founda¬tion of statistical compression algorithms. The key strength of compression-based methods is that they allow construct¬ing robust probabilistic text classifiers based on character-level or binary sequences, and thus omit tokenization and other error-prone pre-processing steps. Nevertheless, as clearly stated by Sculley and Brodley (2006), they are not a “parameter free” silver bullet for feature selection and data representation. In fact they are concrete similarity measures within defined feature spaces. Commonly used statistical compression algorithms are dynamic Markov compression (DMC) and prediction by partial matching (PPM), both de¬scribed in detail by Bratko et al. (2006). Basically these are n-gram models where weights are implicitly assigned to the coordinates during compression. Empirical tests, in above references, show that compression by DMC and PPM out¬performs the explicit n-gram vector space model due to this inherent feature weighting procedure. For the implementa¬tion we use PSMSlib (Bratko 2006), which uses the PPM algorithm.
During the training phase a compression model Mcf is built (Bratko et al. 2006) for each feature f in Table 5 and for each class c (vandalism or legitimate). The main idea is that sequences of characters generated by a particular class will be compressed better using the corresponding model. In theory, an optimal compression can be achieved if one knows the entropy given that model. In order to classify a revision r, we estimate for each of its feature values x the entropy H by calculating,
IIx
Hf 1
c (r) = |x| log p(xi|xi1
ik, Mcf ),
i=1
where p(xi|xi1
ik, Mcf) is the probability assigned by model
Mcf to symbol xi given its k predecessors. In order to score
the revision, we combine all features by summing over the
entropies,
~Sc(r) = Hcf (r)
f
46
47
Figure 1: Precision/Recall curves: Naive Bayes versus Probabilistic Sequence Modeling for revision diff features with(out) user groups and comment.
information than is apparent from the current results, where it appears to be merely a noise factor. To exploit the useful¬ness of this feature, we will take into account its effect on the semantic level by measuring the text life, i.e. the value of the deleted words, as suggested by Adler and de Alfaro (2007).
Conclusions and Future Work
As far as we know, we are among the first to try machine learning techniques to answer the need of improving the re¬call of current expert systems, which are only capable of identifying 30% of all vandalism. We demonstrate that, by applying two machine learning algorithms, a straight for¬ward feature representation and using a set of noisy labeled examples, the accuracy of the actual running bots can be im¬proved. We feel confident that this study is merely a starting point and that there is much room for improvement. In the end almost all vandalism that is not related to misinforma¬tion should be detectable automatically, without consulting third-party information.
For future work, we will combine the ideas from Gabrilovich and Markovitch (2007) and Adler and de Al-faro (2007) to enhance the feature representation. We aim to rebuild their explicit semantic interpreter and use it for semantic comparison between the current modified revision and the previous versions of an article. We will compare the concepts related to text inserted and deleted, and weight these features using respectively the authority of authors and the value of words expressed in text life or expected viewing rate. In this context, we plan to compare our effort to the work of Potthast, Stein, and Gerling (2008).
Acknowledgements
We thank Walter Daelemans for suggesting statistical com¬pression models, Jacobi Carter and Aaron Schulz for provid¬ing the source code of ClueBot and VoAbot II. Koen Smets is supported by a Ph. D. fellowship of the Research Founda¬tion - Flanders (FWO).
References
Adler, B. T., and de Alfaro, L. 2007. A Content-Driven Reputation System for the Wikipedia. In Proceed¬ings of the 16th International World Wide Web Conference (WWW). ACM Press.
Bratko, A.; Cormack, G. V.; Filipiˇc, B.; Lynam, T. R.; and Zupan, B. 2006. Spam Filtering using Statistical Data Compression Models. Journal of Machine Learning Re¬search 7(Dec):2673–2698.
Bratko, A. 2006. PSMSLib: Probabilistic Sequence Mod¬eling Shared Library. Available at http://ai.ijs.si/ andrej/psmslib.html.
Buriol, L. S.; Castillo, C.; Donato, D.; Leonardi, S.; and Stefano, M. 2006. Temporal Analysis of the Wikigraph. In Proceedings of the IEEE/WCIC/ACM International Con¬ference on Web Intelligence (WI), 45–51.
Carter, J. 2007. ClueBot and Vandalism on Wikipedia. Unpublished. Available at http://24.40.131.153/ ClueBot.pdf.
Gabrilovich, E., and Markovitch, S. 2007. Computing Semantic Relatedness using Wikipedia-Based Explicit Se¬mantic Analysis. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI), 1606– 1611.
Hart, M.; Johnson, R.; and Stent, A. 2007. Content-Based Access Control. Submitted to the IEEE Symposium on Pri¬vacy and Security.
Kittur, A.; Suh, B.; Pendleton, B. A.; and Chi, E. H. 2007. He Says, She Says: Conflict and Coordination in Wikipedia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
McCallum, A. K. 1996. Bow: a Toolkit for Statis¬tical Language Modeling, Text Retrieval, Classification and Clustering. Available at http://www.cs.cmu.edu/ ˜mccallum/bow.
Potthast, M.; Stein, B.; and Gerling, R. 2008. Automatic Vandalism Detection in Wikipedia. In Proceedings of the 30th European Conference on IR Research (ECIR), 663– 668.
Priedhorsky, R.; Chen, J.; Lam, S. T. K.; Panciera, K.; Ter-veen, L.; and Riedl, J. 2007. Creating, Destroying, and Restoring Value in Wikipedia. In Proceedings of the Inter¬national ACM Conference on Supporting Group Work.
Rassbach, L.; Pincock, T.; and Mingus, B. 2007. Explor-ing the Feasibility of Automatically Rating Online Article Quality. In Proceedings of the International Wikimedia Conference (Wikimania). Wikimedia.
Sculley, D., and Brodley, C. E. 2006. Compression and Machine Learning: a New Perspective on Feature Space Vectors. In Proceedings of the Data Compression Confer¬ence (DCC), 332–341.
Vi´egas, F. B.; Wattenberg, M.; and Dave, K. 2004. Study¬ing Cooperation and Conflict between Authors with history flow Visualizations. In Proceedings of the SIGCHI Confer¬ence on Human Factors in Computing Systems.
48
10 Considerations for a Cloud
Procurement
March 2017
© 2017, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Notices
This document is provided for informational purposes only. It represents AWS’s current product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether express or implied. This document does not create any warranties, representations, contractual commitments, conditions or assurances from AWS, its affiliates, suppliers or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.
Amazon Web Services –
Contents
Purpose 2
Ten Procurement Considerations 2
1. Understand Why Cloud Computing is Different 2
2. Plan Early To Extract the Full Benefit of the Cloud 3
3. Avoid Overly Prescriptive Requirements 3
4. Separate Cloud Infrastructure (Unmanaged Services) from Managed
Services 4
5. Incorporate a Utility Pricing Model 4
6. Leverage Third-Party Accreditations for Security, Privacy, and Auditing 5
7. Understand That Security is a Shared Responsibility 6
8. Design and Implement Cloud Data Governance 6
9. Specify Commercial Item Terms 6
10. Define Cloud Evaluation Criteria 7
Conclusion 7
Page 1
Amazon Web Services –
Purpose
Amazon Web Services (AWS) offers scalable, cost-efficient cloud services that public sector customers can use to meet mandates, reduce costs, drive efficiencies, and accelerate innovation.
The procurement of an infrastructure as a service (IaaS) cloud is unlike traditional technology purchasing. Traditional public sector procurement and contracting approaches that are designed to purchase products, such as hardware and related software, can be inconsistent with cloud services (like IaaS). A failure to modernize contracting and procurement approaches can reduce the pool of competitors and inhibit customer ability to adopt and leverage cloud technology.
Ten Procurement Considerations
Cloud procurement presents an opportunity to reevaluate existing procurement strategies so you can create a flexible acquisition process that enables your public sector organization to extract the full benefits of the cloud. The following procurement considerations are key components that can form the basis of a broader public sector cloud procurement strategy.
1. Understand Why Cloud Computing is Different
Hyper-scale Cloud Service Providers (CSPs) offer commercial cloud services at massive scale and in the same way to all customers. Customers tap into standardized commercial services on demand. They pay only for what they use.
The standardized commercial delivery model of cloud computing is fundamentally different from the traditional model for on-premises IT purchases (which has a high degree of customization and might not be a commercial item). Understanding this difference can help you structure a more effective procurement model. IaaS cloud services eliminate the customer’s need to own physical assets. There is an ongoing shift away from physical asset ownership toward on-demand utility-style infrastructure services. Public sector entities should understand how standardized utility-style services are budgeted for, procured, and used and then build a cloud procurement strategy that is
Page 2
Amazon Web Services –
intentionally different from traditional IT—designed to harness the benefits of the cloud delivery model.
2. Plan Early To Extract the Full Benefit of the Cloud
A key element of a successful cloud strategy is the involvement of all key stakeholders (procurement, legal, budget/finance, security, IT, and business leadership) at an early stage. This involvement ensures that the stakeholders can understand how cloud adoption will influence existing practices. It provides an opportunity to reset expectations for budgeting for IT, risk management, security controls, and compliance. Promoting a culture of innovation and educating staff on the benefits of the cloud and how to use cloud technology helps those with institutional knowledge understand the cloud. It also helps to accelerate buy-in during the cloud adoption journey.
3. Avoid Overly Prescriptive Requirements
Public sector stakeholders involved in cloud procurements should ask the right questions in order to solicit the best solutions. In a cloud model, physical assets are not purchased, so traditional data center procurement requirements are no longer relevant. Continuing to recycle data center questions will inevitably lead to data center solutions, which might result in CSPs being unable to bid, or worse, lead to poorly designed contracts that hinder public sector customers from leveraging the capabilities and benefits of the cloud.
Successful cloud procurement strategies focus on application-level, performance-based requirements that prioritize workloads and outcomes, rather than dictating the underlying methods, infrastructure, or hardware used to achieve performance requirements. Customers can leverage a CSP’s established best practices for data center operations because the CSP has the depth of expertise and experience in offering secure, hyper-scale, IaaS cloud services. It is not necessary to dictate customized specifications for equipment, operations, and procedures (e.g., racks, server types, and distances between data centers). By leveraging commercial cloud industry standards and best practices (including industry-recognized accreditations and certifications), customers avoid placing unnecessary restrictions on the services they can use and ensure access to innovative and cost-effective cloud solutions.
Page 3
Amazon Web Services –
4. Separate Cloud Infrastructure (Unmanaged Services) from Managed Services
There is a difference between procuring cloud infrastructure (IaaS) and procuring labor to utilize cloud infrastructure or managed services, such as Software as a Service (SaaS) cloud. Successful cloud procurements separate cloud infrastructure from “hands-on keyboard” services and labor, or other managed services purchases. Cloud infrastructure and services, such as labor for planning, developing, executing, and maintaining cloud migrations and workloads, can be provided by CSP partners (or other third parties) as one comprehensive solution. However, cloud infrastructure should be regarded as a separate “service” with distinct roles and responsibilities, service level agreements (SLAs), and terms and conditions.
5. Incorporate a Utility Pricing Model
To realize the benefits of cloud computing you need to think beyond the commonly accepted approach of fixed-price contracting. To contract for the cloud in a manner that accounts for fluctuating demand, you need a contract that lets you pay for services as they are consumed.
CSP pricing should be:
• Offered using a pay-as-you-go utility model, where at the end of each month customers simply pay for their usage.
• Allowed the flexibility to fluctuate based on market pricing so that customers can take advantage of the dynamic and competitive nature of cloud pricing.
Allowing CSPs to offer pay-as-you-go pricing or flexible pay-per-use pricing gives customers the opportunity to evaluate what the cost of the usage will be instead of having to guess their future needs and over procure. CSPs should provide publicly available, up-to-date pricing and tools that allow customers to evaluate their pricing, such as the AWS Simple Monthly Calculator: http://aws.amazon.com/calculator. Additionally, CSPs should provide customers with the tools to generate detailed and customizable billing reports to meet business and compliance needs.
Page 4
Amazon Web Services –
CSPs should also provide features that enable customers to analyze cloud usage and spending so that customers can build in alerts to notify them when they approach their usage thresholds and projected/budgeted spend. Such alerts enable organizations to determine whether to reduce usage to avoid overages or prepare additional funding to cover costs that exceed their projected budget.
6. Leverage Third-Party Accreditations for Security, Privacy, and Auditing
Leveraging industry best practices regarding security, privacy, and auditing provides assurance that effective physical and logical security controls are in place. This prevents overly burdensome processes and duplicative approval workflows that are often unjustified by real risk and compliance needs. There are many security frameworks, best practices, audit standards, and standardized controls that cloud solicitations can cite, such as the following:
• Federal Risk and Authorization Management Program (FedRAMP)
• Service Organization Controls (SOC) 1/American Institute of Certified Public Accountants (AICPA): AT 801 (formerly Statement on Standards for Attestation Engagements [SSAE] No. 16)/International Standard on Assurance Engagements (ISAE) 3402 (formerly Statement on Auditing Standards [SAS] No. 70), SOC 2, SOC 3
• Payment Card Industry Data Security Standard (PCI DSS)
• International Organization for Standardization (ISO) 27001, ISO 27017, ISO 27108, ISO 9001
• Department of Defense (DoD) Security Requirements Guide (SRG)
• Federal Information Security Management Act (FISMA)
• International Traffic in Arms Regulations (ITAR)
• Family Educational Rights and Privacy Act (FERPA)
• Information Security Registered Assessors Program (IRAP) (Australia)
• IT-Grundschutz (Germany)
• Federal Information Processing Standard (FIPS) 140-2
Page 5
Amazon Web Services –
7. Understand That Security is a Shared
Responsibility
As cloud computing customers are building systems on a cloud infrastructure, the security and compliance responsibilities are shared between service providers and cloud consumers. In an IaaS model, customers control both how they architect and secure their applications and the data they put on the infrastructure. CSPs are responsible for providing services through a highly secure and controlled infrastructure and for providing a wide array of additional security features. The respective responsibilities of the CSP and the customer depend on the cloud deployment model that is used, either IaaS, SaaS, or Platform as a Service (PaaS).Customers should clearly understand their security responsibilities in each cloud model.
8. Design and Implement Cloud Data Governance
Organizations should retain full control and ownership over their data and have the ability to choose the geographic locations in which to store their data, with CSP identity and access controls available to restrict access to customer infrastructure and data. Customers should clearly understand their responsibilities regarding how they store, manage, protect, and encrypt their data. A major benefit of cloud computing as compared to traditional IT infrastructure is that customers have the flexibility to avoid traditional vendor lock-in. Cloud customers are not buying physical assets, and CSPs provide the ability to move up and down the IT stack as needed, with greater portability and interoperability than the old IT paradigm. Public sector entities should require that CSPs: 1) provide access to cloud portability tools and services that enable customers to move data on and off their cloud infrastructure as needed, and 2) have no required minimum commitments or required long-term contracts.
9. Specify Commercial Item Terms
Cloud computing should be purchased as a commercial item, and organizations should consider which terms and conditions are appropriate (and not appropriate) in this context. A commercial item is recognized as an item that is of a type that has been sold, leased, licensed, or otherwise offered for sale to the general public and generally performs the same for all users/customers, both commercial and government. IaaS CSP terms and conditions are designed to reflect how a cloud services model functions (i.e., physical assets are not being
Page 6
Amazon Web Services –
purchased, and CSPs operate at massive scale to offer standardized commercial services). It is critical that a CSP’s terms and conditions are incorporated and utilized to the fullest extent.
10. Define Cloud Evaluation Criteria
Cloud evaluation criteria should focus on system performance requirements. Select the appropriate CSP from an established resource pool to take advantage of the cloud’s elasticity, cost efficiencies, and rapid scalability. This approach ensures that you get the best cloud services to meet your needs, the best value in these services, and the ability to take advantage of market-driven innovation. The National Institute of Standards and Technology (NIST) definitions of cloud benefits are an excellent starting point to use for determining cloud evaluation criteria: http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-146.pdf.
Conclusion
Thousands of public sector customers use AWS to quickly launch services using an efficient cloud-centric procurement process. Keeping these ten steps in mind will help organizations deliver even greater citizen-, student-, and mission-focused outcomes.
Page 7
Methods for large scale SVD with missing values*
Miklós Kurucz András A. Benczúr Károly Csalogány
Data Mining and Web search Research Group, Informatics Laboratory
Computer and Automation Research Institute of the Hungarian Academy of Sciences
{realace, benczur, cskaresz}@ilab.sztaki.hu
ABSTRACT
We compare recommenders based solely on low rank ap-proximations of the rating matrix. The key difficulty lies in the sparseness of the known ratings within the matrix that cause expactation maximization algorithms converge very slow. Among the prior publicly known attempts for this problem a gradient boosting approach proved most success¬ful in spite of the fact that the resulting vectors are non-orthogonal and prone to numeric errors. We systematically explore expectation maximization methods based both on the Lanczos algorithm and power iteration; novel in this pa¬per is the efficient handling of the dense estimate matrix used as input to a next iteration. We also compare sequence transformation methods to speed up convergence.
Categories and Subject Descriptors
J.4 [Computer Applications]: Social and Behavioral Sci-ences; G.1.3 [Mathematics of Computing]: Numerical Analysis—Numerical Linear Algebra
General Terms
recommender systems, singular value decomposition
Keywords
dimensionality reduction, missing data
1. INTRODUCTION
Recommender systems predict the preference of a user on a given item based on known ratings. In order to evaluate methods, in October 2006 Netflix provided movie ratings from anonymous customers on nearly 18 thousand movie titles [3].
*This work was supported by the Mobile Innovation Cen-ter, Hungary, a Yahoo Faculty Research Grant and by grant ASTOR NKFP 2/004/05
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
KDDCup.07, August 12, 2007, San Jose, California, USA. 2007
Copyright 2007 ACM 978-1-59593-834-3/07/0008 ...$5.00.
In this paper we concentrate on recommenders based solely on low rank approximation and compare various implemen¬tations and parameter settings. The low rank approximation of the rating matrix as a recommendation is probably first described in [5, 24, 17, 27] and many others near year 2000.
The key difficulty in computing the low rank approxima¬tion lies in the abundance of missing values in the rating matrix: the Netflix matrix for example consists in 99% of missing values. While several authors describe expectation maximization based SVD algorithms dating back to the sev¬enties [13] and [7, 28, 30] describes the method for a recom¬mender application, we aware of no systematic studies on large scale problems. In particular all these results consider small, few thousands by few thousands submatrices of the EachMovie or Jester databases, of several orders of magni¬tude smaller than handled by our algorithms.
A successful approach to a low rank recommender is de¬scribed by Simon Funk in [12] is based on an approach rem¬iniscent of gradient boosting [11]. The algorithm opens a number of theoretic questions including its relation to pub¬lished results that solve SVD with missing values as well as the effect of the parameters on convergence speed and over-fitting. One of the main intents of this paper to understand the relation of his method to existing missing value SVD approaches.
Our main contributions are:
9 The implementation of SVD based recommenders for large scale problems with specific attention to the scal-ability issues of handling full matrix imputation values. Note that previous results except for [12] handle data of several orders of magnitude smaller than ours.
9 The comparison of various methods in terms of recom-mendation accuracy and convergence rate, with em-phasis on the explanation of parameters that speed up convergence.
1.1 Our results and organization
The rest of the paper is organized as follows. In the rest of the Introduction we describe related approaches, the experi¬mental setup, and the SVD algorithm implementation used. In Section 2 we measure the effect of filling missing values by zeroes, by averages and finally be the output of an item-item similarity based recommender, a challenging task since the full recommendation matrix has several billion entries.
After imputation by external values we turn to expec-tation maximization approaches that compute a low rank approximation in one iteration and impute the outcome for
a next iteration. In Sections 3 and 4 we use the Lanczos al¬gorithm and power iteration, respectively. In both cases we resolve the implementational challenge of handling the full matrix arising by the previous iteration. We observe slow convergence of the methods; we evaluate methods to speed up by combining partial results.
Finally in Section 5 we describe a least squares based ex¬pectation maximization approach to directly optimize a low rank solution for small error. In this algorithm we optimize each user vector separately, thus enabling a user-by-user adaptive control on the number of dimensions used. Our key observation is that for a user with r ratings, roughly r/25 dimensions should be used.
In all cases we investigate the effect of the dimensional-ity of the low rank approximation; we observe best perfor-mance at a few dimensions and overtraining as the number of dimensions approaches 100 with good performance on the training but deterioration on the test (probe) set. All meth¬ods are compared in Section 6.
1.2 Data set, evaluation and experimental setup
Netflix provided over 100 million ratings from n over 480 thousand randomly-chosen, anonymous customers on m nearly 18 thousand movie titles [3]. The company withheld certain portion of the ratings as a competition qualifying set that we will not use in this report. Netflix also identified a probe subset of the complete training set; we refer the remaining known ratings as the training data.
We use the root mean squared error
RMSE2 = (wij ˆwij)2
ijR
as the single evaluation measure, where wij is the actual rating, an integer in the range 1–5, given by user i to movie j, and ˆwij is the prediction given by the recommender system. We present RMSE values for the train and the separated test set but not the qualifying set.
The experiments were carried out on a cluster of 64-bit 3GHz P-D processors with 4GB RAM each and a multipro¬cessor 1.8GHz Opteron system with 20GB RAM.
1.3 Related work
Recommenders based on the rank k approximation of the rating matrix based on the first k singular vectors are prob¬ably first described in [5, 24, 17, 27] and many others near year 2000.
The Singular Value Decomposition (SVD) of a rank ρ ma¬trix W is given by W = UT ΣV with U an m × ρ, Σ a ρ × ρ and V an n × ρ matrix such that U and V are orthogonal. By the Eckart-Young theorem [16] the best rank-k approxi¬mation of W with respect to the Frobenius norm is
||WUk T ΣkVk||2 F = (wij σkukivkj)2. (1)
ij k
where Uk is an m × k and Vk is an n × k matrix contain¬ing the first k columns of U and V and the diagonal Σk containing first k entries of Σ.
The RMSE differs from the above equation only in that summation is over known ratings
RMSE2 = err2ij where errij = wij σkukivkj (2)
ijR k
where R denotes either the training or the test set. To sim-plify notation we extend errij with value 0 for ij / R.
As already emphasized in one of the early works [27], the crux in using SVD for recommenders lies in handling miss¬ing values in the rating matrix W. Goldberg et al. [15] for example require a gauge set where all ratings are known, an assumption clearly infeasible on the Netflix data scale. Azar et al. [2] prove asymptotic results on replacing missing values by zeroes and scaling known ratings inversely proportional to the probability of being observed.
The expectation maximization (EM) algorithm proceeds as follows. Given the output Uk, Σk and Vk matrices of sizes m × k, k × k and n × k, respectively, produced by SVD in the maximization step, the expectation step produces a matrix with entries
wij if ij R [UkΣkVk]ij otherwise
= errij + [UkΣkVk]ij (3)
where the last equality follows by the definition of err as in (2). The algorithm alternates between SVD computation (maximization) and the expectation equation (3) until con-vergence. It is easy to see that U, Σ and V minimizing (2) is a fixed point of this iteration; up to our best knowledge, this is the only theoretical result known about the convergence properties of the above EM missing value SVD algorithm. While in our implementation k is typically fixed as input parameter, variants of this algorithm may increase or even decrease k as the iterations proceed.
This algorithm is perhaps first used for recommenders by Canny [7] and then several others [28, 30]. Canny [7] con¬centrates on privacy issues; he reports experiments on much smaller scale such as a subset of the EachMovie data. Srebro and Jaakkola [28] compare methods that fill missing entries by zeroes, also by scaling known entries as in [2], using a gauge set as in [15] as well as a variant of the EM procedure. They also give a number of hints related to the convergence of the EM method. First of all they observe the algorithm may reach local optimum; it is unclear whether this may happen in the missing value case as well. They also show that different rank solutions are non-orthogonal; for this end they propose starting out with a large rank approximation and gradually reduce the rank in the EM iterations.
The EM algorithm for solving SVD with missing values dates back to the seventies; [14] gives a more recent descrip¬tion. In early results, the generic idea of filling missing values by expectation maximization to our knowledge appears first in [19] and is perhaps best described by [9] with the explicit mention of factor analysis as an application but apparently no references between another line of work [25, 13]. To our knowledge, the first paper that presents the missing value problem is [25]; [13] generalizes the missing value problem to a weighted regression and solves it by EM.
More recently several authors reinvented EM for SVD. In [6] the idea of representing the missing data imputation ma¬trices by their known SVD U and V appears that is key in our sparse implementation. Zhang et al. [30] give an ap¬proximate SVD algorithm with theoretical analysis, however tests are only shown on small scale data.
Theoretical works on SVD based recommenders exist [10] but we are aware of none that address the missing value problem. In particular we aware of no results on the conver
Table 1: Running times in the form of minutes or hours:minutes for a single iteration over the Netflix Prize matrix with n over 480 thousand, m nearly 18 thousand and over 100 million non-zeroes. For Lanc-zos and Power the top column k gives dimensionality while for Adaptive the dimensionality is 1 + r/k for a user with r ratings, i.e. here unlike for the other two algorithms the running time decreases with k. For Lanczos we use 40 iterations altogether and for Power we use 100 for a single dimension, hence here we get linear dependency on k for Power.
gence except for the negative experimental findings of Srebro and Jaakkola [28].
Dimensionality reduction is investigated for gene expres-sion data as well. Several authors [20, 29] compare imputa¬tion methods including nearest neighbor as well as the EM approach with controversial findings for accuracy but a def¬inite identification of the very slow convergence for EM.
1.4 SVD implementation
In our implementation we used the Lanczos code of svd-pack [4] and compared it with a power iteration developed from scratch. Lanczos appears precise and efficient; in con¬trast power iteration used by several results [21, 8] is slightly faster but much less accurate; computing more than two di¬mensions is numerically very unstable due to the orthogonal projection step. Running times are shown in Table 1.
We implemented a key modification over svdpack that enables missing data imputation as well as very large in-put handling. After removing the obsolete condition on the maximum size of an input matrix, we abstracted data ac¬cess within the implementation to computing the product of a vector with either the input matrix or its transpose.
While implementation issues of SVD computation are be¬yond the scope of the paper, we compare the performance of the Lanczos and block Lanczos code of svdpack [4] and our implementation of a power iteration algorithm. Hagen et al. [18] suggest fast Lanczos-type methods as robust basis for computing heuristic ratio cuts; others [21, 8] use power iteration.
We also measure the number of dimensions of the ap-proximation. Typically in SVD use the dimensionality is restricted by efficiency considerations and for example for spectral clustering [1, 23] suggest more eigenvalues produce better quality cuts. However we observe that as the num¬ber of dimensions increase beyond roughly 10, we overtrain and prediction quality deteriorates; for this reason we also test an algorithm that adaptively selects more dimensions for users with more ratings in Section 5.
2. MISSING DATA IMPUTATION FROM EX-TERNAL RESULTS
In the simplest approach we use external sources of data to fill missing ratings and optimize for error in Frobenius norm as in equation (1) in the hope that external data fit well and optimization for Frobenius yields good approximation for
Figure 1: The rmse as the function of the iterations for the simple Lanczos EM algorithm, the missing values are filled with zeros in the initial matrix.
the RMSE equation (2) as well. First, as expected, we show filling missing data with zeroes as suggested for example by [2] badly fail over the rare Netflix data by providing recom¬mendations near 0 due to the abundance of zeroes in the matrix after imputation. We improve performance first by using averages, then by the outcome of a more sophisticated recommender based on item-item similarities. Surprisingly user averages perform better than the output of the recom¬mender in this case.
Imputation by zeroes and averages are fairly straightfor-ward given control over data access within the SVD algo-rithm as described in Section 1.4. It is however a challenging question for a full recommendation matrix that we describe in Section 2.1.
We show RMSE values for imputation with zeroes and av-erages as the first iterations in Fig. 3. We observe very poor performance; in particular by filling with zeroes we are so far off from optimum that even a large number of EM iterations remain insufficient to converge.
2.1 Output of an item-item similarity based recommender
We implemented the adjusted cosine similarity [26] for an item-item similarity based recommender that recommends an unrated movie j to a given user i by the weighted average of the nearest N movies to i rated by the user. Here N is a parameter; roughly speaking, this approach increases the fraction of known values by a factor of N.
The Lanczos implementation of svdpack [4] accesses the matrix by in one step computing a product of a vector with either the matrix or its transpose. The SVD implementation may hence access the nearest neighbor lookup table when¬ever a matrix multiplication is needed. The implementation requires space to store the rating matrix and the nearest neighbor index. In the running time however the matrix multiplication time becomes dependent on the size of the full matrix mn instead of the much smaller number of known ratings. While this implementation is memory efficient, it is so slow that we had to give up tests in this direction.
We may however give an efficient item-item similarity based imputation by slightly regressing the item-item simi¬larity based output towards the user average ˆu, as follows. We form the submatrix S of the item-item similarities where for efficiency considerations we only keep the top 100 largest
Figure 2: The rmse as the function of the number of iterations for the simple Lanczos EM algorithm with the first iteration imputed with the output of the simplified item-item similarity based recommender.
entries in each row. We even discard those of the values be¬low 0.5. When predicting a rating for user i and movie j, we then compute the sum of w, ˆu weighted by the similarity of j and j' for all j' where both the similarity and the rating w~ are known. Next in order to give a prediction we have to add the normalized value to ˆu. In order to be efficiently computable, we simply normalize by 100, even though typ¬ically there are less than 100 j' terms in the sum and their similarity values may be as low as 0.5.
The algorithm proceeds as follows. We let H be an n × m matrix where each row contains the user averages ˆu as identical values and
(w ˆu)/100 if (ij) E R
F =
0 otherwise.
The product of vector x with the imputed matrix W' can be efficiently computed as
x • W' = x • H + x • F • S + x • E
where
W' H F • S if(ij)ER
E =
0 otherwise
removes the effect of the similarity based recommendation where the actual rating is known. In Fig. 2 we see the RMSE for a 10-dimensional SVD started by these values.
3. SPARSE LANCZOS IMPLEMENTATION WITHIN AN EM FRAMEWORK
When using the Lanczos algorithm after an expectation step, we face the same difficulty of imputing a full matrix as in Section 2.1. We provide a similar solution below, with careful analysis of the number of operations used. Note that unlike in the previous section, we need a large number of iterations until convergence hence not just the space but also the speed of handling the dense input is crucial.
Since in a Lanczos iteration we require the product of vec¬tor x = (x1, x2, ...) with wˆ (or similarly with its transpose), by the EM algorithm equation (3) we compute
err • x + UΣV • x.
The space required by this algorithm is equal to O(kn+km) for multiplying x with the imputed low rank approximation term by term from right to left, in addition to the number of non-missing values in the rating matrix. Hence the addi¬tional work due to imputation is negligible in the Lanczos computation.
3.1 Speeding up convergence
In order to speed up convergence we apply the generic method of finding an optimal linear combination of the val-ues wˆ = σkukivkj in the current and w(t1) in the pre¬vious iterations: we minimize the quadratic expression of λ in the RMSE equation (2):
(w λ ˆw (1 λ)w(t1)
)2.
We then let w() = λ wˆ (1 λ)w(t1) for the λ value at the minimum.
In the above naive form however matrix values in the next iteration will arise as a linear combination of a rank k matrix and the previous w(1), which in turn depends on w(2) and inductively on all previous partial results. Since it is infeasible to store either full matrices or all partial results, we have to relax the above algorithm. We give two versions next that obey the scalability requirements.
Our two implementations of linearly combining current and previous results use the last two low rank approxima¬tions U(), Σ(t)k ,Vk(t) and U(t1)
, Σ(1) , V (1) . In the first
algorithm we combine as
λU(t)
Σ() V ()
+ (1 λ)U(1)
Σ(1) V (1)
,
i.e. using only the low rank matrix instead of the com-bined iteration t 1 approximation. The minimum of the quadratic expression in λ is attained, with the notation of A = Uk (t)Σ(t) k V (t)
and A' = U(1)
Σ(1) V (1)
, at
(A A')(A' w)
λ = PijR(Aij A'ij)2
In the second variant we combine the low rank decomposi-tion elements: from U()
, V ()
and the previous result we get
ˆU()
and ˆV(). We ignore Σ(1)
based on the observation
that the Σ converges fast. With the simplified notation Uand U' for the i-th row of U() and U(1)
, respectively and
the same notation for the columns of the V, we minimize (λ(Ui Ui')+ Ui')Σ(λ'(Vi Vi') +Vi') wij2.
Here for each λ' there is a corresponding optimum λ' given by
X = λ'(Ui Ui')(Vj Vj') + (Ui Ui' )Vj',
λ =
λ'UΣ(V V ' ) + U ' V ' w)X
;
X2
we select the best by substituting a low number of probe λ
values.
4. POWER ITERATION WITHIN AN EM FRAMEWORK
Figure 3: The rmse as the function of the iterations for the simple Lanczos EM algorithm and the two convergence boosting variants. Curves correspond to different dimensionality with V1 and V2 denoting the two convergence boosting variants.
For a full matrix W, power iteration proceeds by repeat-edly letting
u(+1) = W • v()/||v() ||, (4)
v(+1) = W • u()/||u()||. (5)
The algorithm converges to the first singular vectors also called the “hub” and “authority” vectors [22]; due to numeric errors this holds even if we start out with an initial v(0) orthogonal to the first vector V1 unless we orthogonalize, i.e. project each or some v() onto the hyperplane orthogonal to V1. By orthogonalization to the first k 1 singular vectors V1 however we may obtain the next V by iteration (4–5).
In the presence of missing data we may use (4–5) in the expectation maximization framework by filling ij / R by
w = σ1ui • v
() . First we observe that if v() is a good ap
()
proximation of V1, then ||v(+1)|| σ1, hence the iteration turns to
Figure 4: The rmse as the function of the number of iterations for the basic power iteration method given by equations (6–7).
v(+1) = W • u()/||u() ||. (7)
This approach is split into two different heuristic implemen¬tations in the next two subsections. The RMSE for the basic implementation (6–7) is shown in Fig. 4.
4.1 Method of individual increments
We rewrite (6) as
u(+1) =
: w
σ1 v(t)
• u()v()
+
2
v()
• u . (8)
()
Given the assumption that the v are normalized that we may enforce in our algorithm, the second term is simply u()
. As a heuristic speedup, we split (8) into increments over u for individual j, replacing u by a new value in each step. This yields an algorithm with a cycle over
u u + (w/σ1 vj • ui)vj (9)
very closely reminiscent of Simon Funk’s steps [12]
u (1 lRate)u + K(w σ1uivj)vj. (10)
We use an idea similar to the convergence acceleration in Section 3: we multiply the increment in (9) by a factor δ that
Figure 5: The rmse as the function of the number of iterations for power iteration with individual in-crements given by equation (9).
Figure 6: The rmse as the function of the number of iterations for power iteration with repeated hub and authority steps given by equation (12).
then t iterations of (6) give
u (1 h)u + (1 (1 v))v1Δ. (11)
Best results are obtained in Fig. 6 if we use values k = 5 for the first singular vector computation and then decrease to 3, 2 and finally 1 for next dimensions. In addition we also combined this technique with individual increments as in (8):
u (1 v 2 )u + (1 (1 v2))v2Δ, (12)
a formula again reminiscent of (10) of [12].
We may repeat the idea of the previous subsection and
compute these steps individually for each i and j
5. A LEAST SQUARES APPROACH WITH ADAPTIVE DIMENSIONALITY
We give an algorithm that alternatedly computes and op¬timal U for a fixed V and then exchanges the role of U and
V, similar to the “hub” (4) and “authority” (5) steps of the power iteration. For fixed V, optimizing the RMSE equation (2) can be done separately for the columns of U as
(w σuv)2 (13)
as n regression problems.
The key idea in our algorithm is that once the regression is made separate for each user, we may adaptively select the right dimensionality for user i depending on the amount of ratings r given by her. Fig. 7 depicts RMSE for different val¬ues of constant K where we use the first 1+ r/K dimensions in the above expression. We observe overfitting on the train¬ing set: the more dimensions used, the better is the RMSE; however over the probe set values of K between 25... 30 per¬form the best; for a large number of iterations apparently K = 25 takes lead.
6. COMPARISON OF METHODS AND CON¬CLUSION
In our findings the best method is Lanczos with 10 di-mensions. Unfortunately the iterations are relative costly and convergence boosting approaches tend to give minor improvements only. Power iteration based methods, though performing very similar steps as Funk’s algorithm [12], tend to overfit the training set. We believe more careful tuning could improve performance. The runner up is the adaptive dimensionality least squares approach.
The most carefully tuned implementation of Funk’s al-gorithm (10) [12] reaches an RMSE slightly below 0.92 on the probe set with 95 dimensions in over 100 iterations and K = 0.015, lRate = .001. By setting K = 0 performance similar to ours is reported. We believe a thorough measure¬ment over our algorithms might find improvements, however our main goal here was to understand the behavior of the missing value problem by investigating a large number of related algorithms.
For further work we propose the implementation and com-parison of fast SVD approximations and experiments with graphs of even larger scale. We also plan to mix results, a method that is known to yield significant improvement and in addition sometimes prefer weaker recommenders and thus slightly redraw the picture.
7. REFERENCES
[1] C. J. Alpert and S.-Z. Yao. Spectral partitioning: the more eigenvectors, the better. In DAC ’95: Proceedings of the 32nd ACM/IEEE conference on Design automation, pages 195–200, New York, NY, USA, 1995. ACM Press.
[2] Y. Azar, A. Fiat, A. R. Karlin, F. McSherry, and J. Saia. Spectral analysis of data. In Proceedings of the 33rd ACM Symposium on Theory of Computing (STOC), pages 619–626, 2001.
[3] J. Bennett and S. Lanning. The netflix prize. In KDD Cup and Workshop in conjunction with KDD 2007, 2007.
[4] M. W. Berry. SVDPACK: A Fortran-77 software library for the sparse singular value decomposition. Technical report, University of Tennessee, Knoxville, TN, USA, 1992.
[5] D. Billsus and M. J. Pazzani. Learning collaborative information filters. In ICML ’98: Proceedings of the Fifteenth International Conference on Machine Learning, pages 46–54, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc.
[6] M. Brand. Incremental singular value decomposition of uncertain data with missing values. In ECCV (1),
0 20 40 60 80 100 120
0 10 20 30 40 50 60 70 80 90 100
Figure 7: The rmse as the function of the number of iterations for adaptive least squares given by equation (13). Different curves correspond to values of K where we compute least squares over the first 1 + r/K dimensions for a user with r ratings. Left: rmse over the test set. Right: rmse over the probe set.
pages 707–720, 2002.
[7] J. Canny. Collaborative filtering with privacy via factor analysis. In SIGIR ’02: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pages 238–245, New York, NY, USA, 2002. ACM Press.
[8] D. Cheng, S. Vempala, R. Kannan, and G. Wang. A divide-and-merge methodology for clustering. In PODS ’05: Proceedings of the twenty-fourth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 196–205, New York, NY, USA, 2005. ACM Press.
[9] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm.
[10] P. Drineas, I. Kerenidis, and P. Raghavan. Competitive recommendation systems. In Proceedings of the 34th ACM Symposium on Theory of Computing (STOC), pages 82–90, 2002.
[11] J. H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189–1232, 2001.
[12] S. Funk. Netflix update: Try this at home. http://sifter.org/˜ simon/journal/20061211.html, 2006.
[13] K. R. Gabriel and S. Zamir. Lower rank approximation of matrices by least squares with any choice of weights. Technometrics, 21:489–498, 1979.
[14] Z. Ghahramani and M. I. Jordan. Supervised learning from incomplete data via an EM approach. In J. D. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems, volume 6, pages 120–127. Morgan Kaufmann Publishers, Inc., 1994.
[15] K. Goldberg, T. Roeder, D. Gupta, and C. Perkins. Eigentaste: A constant time collaborative filtering algorithm. Inf. Retr., 4(2):133–151, 2001.
[16] G. H. Golub and C. F. V. Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, 1983.
[17] D. Gupta, M. Digiovanni, H. Narita, and K. Goldberg. Jester 2.0 (poster abstract): evaluation of an new linear time collaborative filtering algorithm. In SIGIR
’99: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 291–292, New York, NY, USA, 1999. ACM Press.
[18] L. W. Hagen and A. B. Kahng. New spectral methods for ratio cut partitioning and clustering. IEEE Trans. on CAD of Integrated Circuits and Systems, 11(9):1074–1085, 1992.
[19] H. O. Hartley. Maximum likelihood estimation from incomplete data. Biometrics, 14:174–194, 1958.
[20] T. Hastie, R. Tibshirani, G. Sherlock, M. Eisen, O. Alter, D. Botstein, and P. Brown. Imputing missing data for gene expression arrays. Technical report, Department of Statistics, Stanford University, 2000.
[21] R. Kannan, S. Vempala, and A. Vetta. On clusterings — good, bad and spectral. In IEEE:2000:ASF, pages 367–377, 2000.
[22] J. Kleinberg. Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5):604–632, 1999.
[23] J. Malik, S. Belongie, T. Leung, and J. Shi. Contour and texture analysis for image segmentation. Int. J. Comput. Vision, 43(1):7–27, 2001.
[24] M. H. Pryor. The effects of singular value decomposition on collaborative filtering. Technical report, Dartmouth College, Hanover, NH, USA, 1998.
[25] A. Ruhe. Numerical computation of principal components when several observations are missing. Technical report, UMINF-48, Ume˚a, Sweden, 1974.
[26] B. Sarwar, G. Karypis, J. Konstan, and J. Reidl. Item-based collaborative filtering recommendation algorithms. In WWW ’01: Proceedings of the 10th international conference on World Wide Web, pages 285–295, New York, NY, USA, 2001. ACM Press.
[27] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Application of dimensionality reduction in recommender systems–a case study. In ACM WebKDD Workshop, 2000.
[28] N. Srebro and T. Jaakkola. Weighted low-rank approximations. In T. Fawcett and N. Mishra, editors, ICML, pages 720–727. AAAI Press, 2003.
[29] O. G. Troyanskaya, M. Cantor, G. Sherlock, P. O.
Brown, T. Hastie, R. Tibshirani, D. Botstein, and R. B. Altman. Missing value estimation methods for dna microarrays. Bioinformatics, 17(6):520–525, 2001.
[30] S. Zhang, W. Wang, J. Ford, F. Makedon, and J. Pearlman. Using singular value decomposition approximation for collaborative filtering. In CEC ’05: Proceedings of the Seventh IEEE International Conference on E-Commerce Technology (CEC’05), pages 257–264, Washington, DC, USA, 2005. IEEE Computer Society.
SPIRITUAL FORMATION: RETRIEVING PERICHORESIS AS A MODEL FOR SHARED LEADERSHIP IN THE
MARKETPLACE
MICHAEL L. DAVIS
Abstract
Although shared leadership has been identified as a product of the leadership research in the last quarter of the twentieth-century, this article seeks to provide a theological basis for the practice of shared leadership in the marketplace by Christians. Weaving together perichoresis, union with Christ, and spiritual formation, I propose that maturing Christians may practice shared leadership as a reflection of the Trinitarian model.
Introduction
―Like everything else that evolves over time, a new type of leadership is emerging. And many of the old rules just don‘t apply anymore."1 Comments such as this abound in the leadership literature of the twenty-first century. Change is in the air. Daily researchers publish new leadership articles touting life-changing approaches to the ―world‘s oldest vocation."2 The study of leadership reminds one of the old wedding mantra: something old, something new, something borrowed, something blue. Shared leadership is one of the new models of leadership finding favor in nearly every sector of today‘s society.
Some brides attempt to fulfill tradition by having both something old and something new as part of their wedding day attire. Skillful researchers will find something old about
1 Patrick Sanaghan and Paulette A. Gabriel, ―Why Top-Down Leadership Doesn‘t Work Anymore," The Key Leadership Blog, entry posted August 28, 2012, http://keyleadership.com/wordpress/why-top-down-leadership-doesnt-work-anymore/ (accessed March 7, 2013).
2 Remy De Gourmont, ―Concepts of Leadership," in The Bass Handbook of Leadership: Theory, Research, and Managerial Applications, ed. Bernard M. Bass, (New York: Free Press, 2008), 3.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
106 DAVIS
shared leadership as well. What appears to be a new concept has roots in time and eternity. For Christian leaders, shared leadership is much more than an approach to leading others. Authentic shared leadership is a reflection of the Christian‘s sanctification process. Although shared leadership serves as a contemporary model for the marketplace, we may understand it more deeply as an expression of mature spiritual formation rooted in the biblically-based doctrine of perichoresis.3 According to James Torrance, this patristic word describes God as "the God who has his true being as the Father of the Son, and as the Son of the Father in the Spirit. God is love, and has his true being in communion, in the mutual indwelling of the Father, Son, and Holy Spirit."4 Further, the persons of the Godhead occupy the same infinite space because the Father and the Son are both fully God. The Father, Son, and Spirit are "in each other in indissoluble union." The union of the Father, Son, and Spirit does not detract from the uniqueness of either.5
The significance of this article is found in its contention that shared leadership is not a new model identified by modern theorists and practitioners, namely Craig Pearce and Jay Conger, but that it finds its genesis prior to the creation of humankind. Shared leadership is rooted in a retrieval6 of the doctrine of the perichoretic
3 The doctrine of perichoresis, in Millard J. Erickson, Christian Theology, (Grand Rapids: BakerAcademic, 1998), 366, refers to ―the teaching that the life of each of the persons flows through each of the others, so each sustains each of the others and each has direct access to the consciousness of the others."
4 James Torrance, "The Doctrine of the Trinity in Our Contemporary Situation," in The Forgotten Trinity, ed. Alasdair I. C. Heron (London: BCC/CCBI Inter-Church House, 1989), 15.
5 Gerald Bray, The Doctrine of God (Keicester, UK: InterVarsity Press, 1993), 158.
6 J. Todd Billings, Union with Christ: Reframing Theology and Ministry for the Church, (Grand Rapids: BakerAcademic, 2011), 4. Billings believes that ―a theology of retrieval" involves participating with the saints of past ages through reading of Scripture. The writings and teachings of old may be useful for modern readers to gain a better understanding of current theological issues through the retrieved wisdom of the past. In this paper, the retrieval of the doctrine of perichoresis related to union with Christ serves
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DaVIS 107
relationship of the persons of the Trinity and may be a result of the believer‘s union with Christ.
As Christians are positioned in Christ and Christ is a member of the Trinity, it stands to reason that Christians find their place in the perichoretic relationship of the Trinity. It is this sharing of the nature and practices found in the Trinitarian relationship of the Godhead that enables maturing disciples to practice intentionally shared leadership.7 My contention does not discount the work of common grace in the lives of non-believers who may decide to practice shared leadership from a pragmatic stance. However, I suggest that one outcome of maturing spiritual formation is Christ-likeness, particularly in the area of leadership.
Miroslav Volf describes the leadership style of the Godhead as polycentric reciprocity or a relationship "characterized neither by a pyramidal dominance of the one nor by a hierarchical bipolarity between the one and the many."8 It is this polycentric reciprocity that foreshadows the modern model of shared leadership, which takes into account the changing role of leaders and followers in organizations. The model does not focus on the position power of a leader but instead focuses on the critical functions of leadership as diagnosis and action taking. Any
as the basis for a biblical approach to shared leadership. Although hierarchical understandings of the Trinity may provide a model for hierarchical leadership the writer contends that the perichoretic, mutual, interpenetrating relationship as exemplified by the Persons of the Godhead serve as sufficient basis for a biblical understanding of shared leadership as a model for marketplace leaders. However, other leadership theorists may devise an alternative model of Trinitarian leadership based on alternative presuppositions of the nature of God.
7 Although the work of Leonardo Boff suggests that perichoresis may be a model for human community, I am not attempting to make that point. However, this article does suggest that maturing spiritual formation, as a result of the believer‘s union with Christ, positions her/him in a perichoresis with the Trinity and this position in Christ influences the leadership practices of the believer as they become more like Christ. See Leonard Boff, Trinity and Society (New York: Orbis Books, 1988), 7.
8 Miroslav Volf, After Our Likeness: The Church as the Image of the Trinity (Grand Rapids, Mich.: Eerdmans, 1998), 217.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
108 DAVIS
team member can perform the critical leadership functions to assess the current effectiveness of the team and then take appropriate action.9
My research of shared leadership in the marketplace by Christian leaders suggests that shared leadership finds real-life expression through:
1. Collaborative climate strategies
2. An understanding of the power of team vision
3. Removal of leadership walls
4. Safe communication
5. Enrichment of the lives of all stakeholders
6. Lived-out values
7. Recognition of the value of people.10
This article will review the current literature relevant to shared leadership as an expression of mature discipleship made possible by perichoresis and union with Christ. I will develop the argument in three phases reflecting the Trinitarian roots of shared leadership as spiritual formation: perichoresis via union with Christ, shared leadership via perichoresis, and mature discipleship via shared leadership. Finally, I will draw useful conclusions and applications for spiritual formation.
Literature Review
Theorists often view shared leadership as a twentieth-century phenomenon. This article is an attempt to articulate my belief that shared leadership is foundationally Christian in nature and character. One may find the roots of this leadership style in an understanding of the Trinitarian idea of perichoresis and union with Christ, as well as evidence of maturing spiritual formation. Development of this argument includes a review of literature related to shared leadership, perichoresis, and spiritual formation informed by the Christian‘s union with Christ. Although a wealth of
9 Susan E. Kogler Hill, "Team Leadership," in Leadership: Theory and Practice. 5th ed., ed. Peter G. Northouse (Los Angeles: Sage, 2010), 258.
10 Michael L. Davis, "Intentional Practice of Shared Leadership in the Marketplace by Christian Leaders: A Multi-case Study" (dissertation, The Southern Baptist Theological Seminary, 2014), 74–78.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DAVIS 109
material exists related to each of these topics, a review of selected materials will suffice to build the framework for my thesis: the intentional practice of shared leadership may be understood as an outgrowth of a mature spiritual formation process.
Shared Leadership
Shared leadership, particularly in the form of leadership teams, is the fastest growing style of leadership today.11 Craig Pearce and Jay Conger define shared leadership as ―a dynamic, interactive influence process among individuals in groups for which the objective is to lead one another to the achievement of group or organizational goals or both."12 Pearce and Conger contend that the study of leadership did not evolve until the industrial revolution and that a fascination with the concept of shared leadership was not present until the 1970s. According to Pearce and Conger, ―By the mid-1990s, several scholars began mining this fertile intellectual soil. These scholars, independently and simultaneously, developed models that directly addressed shared leadership. Conditions were finally right for the acceptance of this seemingly radical departure from the traditional view of leadership."13
Amy Edmondson offers an understanding of teaming, a particular type of shared leadership. Edmondson believes, ―Success requires a shift from organizing to execute to a new way of working that supports collaboration, innovation, and organizational learning."14 Edmondson sees teaming as a reaction to the work of Frederick Taylor and Henry Ford, and refers to her concept as a verb and rather than a bounded, static entity.15 Obviously, twentieth-century
11 Craig L. Pearce and Jay A. Conger, Shared Leadership: Reframing the Hows and Whys of Leadership, (Thousand Oaks, CA: Sage Publications, 2003), xi.
12 Pearce and Conger, 1.
13 Pearce and Conger, 13.
14 Amy Edmondson, Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy, (San Francisco: Jossey-Bass, 2012), Kindle Electronic Edition: Chapter 1, Location 604–612 .
15 Edmondson, Location 426–437.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
110 DaVIS
leadership scholars tend to hold an understanding that shared leadership is a modern construct.
Some contemporary writers articulate shared leadership from a theological stance. Peter Dickens buttresses the idea that shared leadership finds its roots in the activity of the Trinity as he writes,
The Trinity is an extraordinary model of distributed leadership. The concept of heroic leadership invariably suggests a single leader.... This [shared leadership model] can only happen when the leaders shift from power to service as their plausibility structure. They must fully embrace not only the concept of service, but intentionally lead from a position that is utterly devoid of power.16
Jimmy Long also suggests that the Jerusalem Council, described in Acts 15, serves as an early Church historical example of the modern shared leadership paradigm. According to Long, ―This shared leadership model seemed to be pervasive in the early days of the church.‖17
Perichoresis
In recent years, leadership theorists have seen a connection between perichoresis and shared leadership. George Cladis has written a popular work on the topic. Leading the Team-Based Church, completed in 1999, seeks to ―bring vital faith and church organization closer together to serve effectively the Kingdom of God in a postmodern culture.‖18 Cladis is one who has encouraged the Christian leadership community to consider the influence of the Trinity on Christian leadership. According to Cladis, John of Damascus used perichoresis in terms of a circle dance, emphasizing the circularity of the holy dance performed by the Father, Son, and Holy Spirit. Further, Cladis writes, ―A
16 Jimmy Long, The Leadership Jump: Building Partnerships between Existing and Emerging Christian Leaders, (Downers Grove, Ill.: IVP Books, 2009), 55.
17 Long, 98.
18 George Cladis, Leading the Team-Based Church: How Pastors and Church Staffs Can Grow Together into a Powerful Fellowship of Leaders, (San Francisco: Jossey-Bass, 1999), x.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DAVIS 111
perichoretic image of the Trinity is that of the three persons of God in constant movement in a circle that implies intimacy, equality, unity yet distinction, and love."19 This description of God is both a ―biblical and theological model for building meaningful ministry teams in the church of the twenty-first century."20
Other theologians see perichoresis in a different light. Gregory Nazianzen used the term ―to refer to the interpenetration of the two natures of Christ, both divine and human."21 According to James Womack, perichoresis originally ―derived from a Stoic term for `mixture‘ which literally meant `a mutual coextension of dissimilar parts entering into one another at all points.‘"22
Christian Formation
Robert Letham‘s work serves as the bridge that connects perichoresis and the doctrine of union with Christ. In his book, Union with Christ: In Scripture, History, and Theology, Letham refers his readers to John 14:10–11: ―Do you not believe that I am in the Father and the Father is in me? The words that I said to you I do not speak on my own authority, but the Father who dwells in me does his works. Believe me that I am in the Father and the Father is in me, or else believe on account of the works themselves." Letham concludes,
This is a reference to what in Trinitarian theology is termed the perichoresis, the mutual indwelling of
the three Trinitarian persons. In the words of Gerald Bray, the persons occupy the same infinite divine space. The Father and the Son are both fully God. All that can be said to be God is possessed by both. Yet they are distinct. The Father and the
19 Cladis, 4.
20 Cladis, 4.
21 James A. Womack, ―A Comparison of Perichoresis in the Writings of Gregory of Nazianzus and John of Damascus," (Master‘s thesis, Dallas Theological Seminary, 2005), 2.
22 Womack, 10–11.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
112 DaVIS
Son—and, by extension, the Spirit—are in each other in indissoluble union. This union does not
infringe the distinctness of either. Jesus goes on to
say that when the Spirit comes, he will indwell his disciples. Moreover, they will then know for
themselves that he and the Father are in each other. On top of this, they will also know that Christ is in them presumably by the Holy Spirit.23
Letham presents clear arguments that union with Christ is necessary for Christian formation to take place in the life of the believer. He writes, ―Union with Christ is the foundational basis for sanctification and the dynamic force that empowers it."24 Letham‘s work serves as a foundational source for this article.
Millard Erickson, in Whose Tampering with the Trinity, understands Bruce Ware‘s concept of the Trinity as an asymmetrical perichoresis, based on Jesus‘ words in John 14:10–11. Erickson writes, ―Here there is asymmetry, in which Jesus is in the Father and the Father is in him. The works Jesus does are not just his own, but the Father is doing his work through Jesus. Ware would strongly agree and probably would see this text as supporting his position, although he does not specifically cite it.."25
In Union with Christ: Reframing Theology and Ministry for the Church, J. Todd Billings describes his theology of retrieval, that is ―hearing the voices of the past in such a way that they are allowed to exceed and overcome the chatter of the present."26 Billings describes a theology of union with Christ as a contrast to the understanding of modern American society. He writes,
A theology of union with Christ centers Christian identity in Jesus Christ himself, and in the claim of
23 Robert Letham, Union with Christ: In Scripture, History, and Theology, (Phillipsburg, NJ: P&R Publishing, 2011), 48.
24 Letham, 6.
25 Millard Erickson, Whose Tampering with the Trinity: An Assessment of the Subordination Debate, (Grand Rapids: Kregel Publications, 2009), Kindle Electronic Edition: Chapter 6, Location 1880–1884.
26 Billings, 2.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DAVIS 113
the Triune God upon the Christian. Salvation is not self-centered but is a renewal and restoration of the
self precisely through orienting the self toward
God, toward the church as the body of Christ, and toward the neighbor. Individual believers discover
their true identity in communion rather than in a
pragmatic, individualistic approach to salvation, and tinkering is replaced by a posture of humble
gratitude before God. The God encountered in
union with Jesus Christ is at once more majestic and
more intimate than the deistic-tending God of the
West.27
As a transitional step to a preview of literature related to spiritual formation, Timothy Paul Jones and Michael S. Wilder, in Christian Formation, present a Christian model for spiritual formation. Jones and Wilder share their conception of formation in Christ as a circle, much like the rings of a tree. In this model, ―A dynamic interaction occurs among the believer‘s faith in God; the means of growth in faith through love of others, love for God, and suffering; and the shaping of one‘s identity in Christ."28 Jones and Wilder provide a framework for moving Christian formation beyond a mere theological understanding of union with Christ and its implications for Christian sanctification, toward practical implementation in the work-a-day world.
Dwight J. Zscheile in ―The Trinity, Leadership, and Power" provides a final contribution to this discussion of leadership, perichoresis, and Christian formation through union with Christ. In this article, Zscheile calls Christian leaders to remembrance of their identity in Christ and in the community of the church. Zscheile, writing from a deeply Trinitarian view, suggests ―The renewal of a historical, narrative, and eschatological understanding of God‘s self-revelation as three persons invites Christian leaders to help their communities place themselves within
27 Billings, 9.
28 Timothy Paul Jones and Michael S. Wilder, ―Faith Development and Christian Formation," in Christian Formation, ed. James R. Estep and Jonathan H. Kim, (Nashville: B&H Academic, 2010), 195.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
114 DaVIS
God‘s unfolding plot."29 Zscheile further suggests that ―when Christian leaders, in the power of the Spirit, cultivate and guide communities of unity and diversity, mutuality and openness, creativity and concern, passion and participation, they live into the promise of Jesus‘ prayer" in John 17:21– 22 .30
Synthesis of Ideas
Two additional sources advance the argument of this study of a perichoretic basis for shared leadership as a product of mature spiritual formation. Constantine R. Campbell, in Paul and Union with Christ, concludes that the result of union with Christ is positive Christian living. Campbell writes, ―It is clear that union with Christ relates to the full variety of believers‘ actions, characteristics, and status. This is such that in Christ language serves as shorthand for indicating that a person is a believer."31 Clearly, Campbell understands that Christians, because of their location in Christ, will become conformed to his image. This is spiritual formation at its finest. Campbell continues his thoughts by offering, ―Christian discipleship means identification with the crucified Lord."32
A second work that helps synthesize the concepts addressed in this article is Perichoretic Salvation. James D. Gifford presents a harmony of several doctrines as he argues that ―the soteriological union—the union of the believer and Christ—constitutes a third type of perichoretic relationship; that is, Christ and the believer mutually indwell and participate in one another analogously to the way the persons of the Trinity do."33 Gifford presents a robust argument to support the current thesis that shared
29 Dwight J. Zscheile, ―The Trinity, Leadership, and Power," Journal of Religious Leadership 6(2) (Fall 2007): 62.
30 Zscheile, 62.
31 Constantine R. Campbell, Paul and Union with Christ: An Exegetical and Theological Study, (Grand Rapids: Zondervan, 2012), 387.
32 Campbell, 387.
33 James D. Gifford, Perichoretic Salvation: The Believer’s Union with Christ as a Third Type of Perichoresis, (Eugene, OR: Wipf & Stock, 2011), 2.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DAVIS 115
leadership flows from mature Christian formation as modeled by perichoresis.
Perichoresis via Union with Christ
The process of Christian sanctification is rooted in the believer‘s union with Christ. Jesus taught his disciples the necessity of being in Christ (John 14:20, 15:1–17, 17:20–23). Either as dear children loved by the Father rather than orphans, as branches of the Vine, or as gifts of the Father to the Son, believers find their fulfillment in Christ. Amazingly, Jesus articulated a foundational principle of shared leadership—unity. Because Jesus knew what his disciples would face in the coming days, he wanted them to understand the importance of unity. However, his desire for unity moved beyond relationships with each other, and ultimately to unity with the Father.34
Andreas J. Kostenberger offers further insight concerning the believer‘s union with Christ as a unifying principle for Christian living based on Jesus teaching found in John 17:
Believers‘ unity is neither self-generated nor an end in itself. Believers‘ "complete‖ unity results from being taken into the unity of God, and, once unified, believers will be able to bear witness to the true identity of Jesus as the Sent One of God. Unless they are unified, how can they expect to give authentic, credible testimony to the Father, who is united with the Son and the Spirit in revealing himself and his salvation in Christ? Secure in the Father‘s love, the same love with which he loved his Son, believers will be able to express and proclaim the Father‘s love to a dark and hostile world.35
Gerald Borchert echoes Kostenberger‘s assessment of the purpose of unity in the body of Christ when he writes, "Oneness is a means to enable the world to realize what
34 Andreas J. Kostenberger, John, in Baker Exegetical Commentary on the New Testament, ed. Robert Yarbrough and Robert H. Stein, (Grand Rapids, Mich.: BakerAcademic, 2004), 497.
35 Kostenberger, 498–99.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
116 DAVIS
God has been doing."36
The believer‘s union with Christ, unity with the body of Christ, and oneness with the Father are reason enough to retrieve the doctrine of perichoresis as a model of shared leadership. Although often difficult to understand, some theologians define perichoresis simply as ―the intensely intimate presence of the Father, Son, and Holy Spirit."37 Further, it may be understood as ―mutual indwelling or, better, mutual interpenetration and refer to the understanding of both the Trinity and Christology."38 However, a deeper understanding is possible.
James Gifford understands perichoresis as more than union of the Trinity or the dual natures of Christ. Gifford articulates a third type of perichoresis as he writes,
The incarnation of Christ, in which the Christological variety of perichoresis finds its full and orthodox expression, would show that a perichoretic relationship between the divine and human would at least be possible—the union of two natures in the person of Jesus Christ shows that both a divine and a human nature can indwell the same physical person simultaneously. In addition, humanity is created in the image of the triune, perichoretic God. Therefore, creation and the incarnation guarantee the possibility of such a relationship.39
Jesus was not interested in uniformity for his growing group of disciples, but rather, unity. Jesus and the Father are distinct from each other, separate persons with different functions. They work together with a common mission, yet
36 Gerald L. Borchert, John 12–21, in The New American Commentary: An Exegetical and Theological Exposition of Holy Scripture, ed. E. Ray Clendenen, (Nashville, Tenn.: B&H Publishing Group, 2002), 208.
37 Robert Kress, ―Unity in Diversity and Diversity in Unity: Toward an Ecumenical Perichoretic Kenotic Trinitarian Ontology," Dialogue and Alliance 4 (1990), 67.
38 S.M. Smith, ―Perichoresis," in Evangelical Dictionary of Theology, ed. W.A. Elwell, (Grand Rapids, Mich.: Baker, 1984), 843.
39 James D. Gifford, Jr., Perichoretic Salvation: The Believer’s Union with Christ as a Third Type of Perichoresis, (Eugene, OR: Wipf and Stock, 2011), 22.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DAVIS 117
are separate persons while remaining unified as one. This is the mystery of the triune nature.40 Such is the nature of shared leadership. Therefore, because believers experience union with Christ as a result of salvation provided by Christ, they also participate in the soteriological perichoresis with the triune God, our biblical model of shared leadership. The evidence of Scripture and an understanding of this third type of perichoresis, rooted in the believer‘s union with Christ, support the writer‘s thesis and results in shared leadership via perichoresis. It is now appropriate to examine biblical and theological evidence to support the contention that shared leadership flows from a soteriological perichoresis.
Shared Leadership via Perichoresis
This paper proposes that the intentional practice of shared leadership of Christians is rooted in union with Christ, is a product of the perichoretic nature of the Trinity, and is evidence of mature discipleship. In his recent work, Robert Crosby refers to the baptism of Jesus as he describes the perichoretic nature of the Trinity in terms of the Divine Team. Crosby writes, ―In this moment, God the Father was honoring God the Son. It was a joyful moment; one not to be missed. Herein we are given a peek inside the ultimate honoring circle, the Divine Team.‖41 Additionally, Crosby references John 5:19–23, 14:15–17, and 16:12–14 as he suggests that the Trinity serves as the ―Ultimate Honoring Circle.‖42 In these passages, Jesus honors the Father, who honors the Son, just as the Holy Spirit honors the Son while Jesus honors the Spirit.
The Holy Spirit plays a prime role in the shared leadership of the Trinity, and serves as an amazing example of self-giving. Jesus teaches his disciples that the Holy Spirit will not seek his own glory, but will shine the spotlight on
40 Merrill C. Tenney, John and Acts, in the Expositor’s Bible Commentary vol.9, ed. Frank E. Gaebelein, (Grand Rapids, Mich.: Zondervan, 1981), 167.
41 Robert C. Crosby, The Teaming Church: Ministry in the Age of Collaboration, (Nashville: Abingdon Press, 2012), 129.
42 Crosby, 131.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
118 DAVIS
the works of the Son.
When the Spirit of truth comes, he will guide you into all the truth, for he will not speak on his own
authority, but whatever he hears he will speak, and
he will declare to you the things that are to come. He will glorify me, for he will take what is mine and
declare it to you. All that the Father has is mine; therefore, I said that he will take what is mine and declare it to you (John 16:13–14, ESV).
In essence, the Holy Spirit‘s ―chief purpose is not to make himself prominent but to magnify the person of Jesus. The Spirit interprets and applies the character and teaching of Jesus to the disciples and by so doing makes him central to their thinking. He makes God a reality to people."43
Genesis 1:26 serves as the foundational evidence for the imago Dei and connects the Trinity to the creation of humankind: ―Let us make humankind in our image, according to our likeness."44 Colin Gunton has written, ―To be a person is to be made in the image of God: that is the heart of the matter. If God is a communion of persons inseparably related, then it is in our relatedness to others that our being human consists."45
If God is a Trinity, and the imago Dei is present in humankind, it seems reasonable to accept the influence of the triune nature of God on the leadership capacity of humans. Because of humankind‘s fallen nature, the believer‘s union with Christ and sanctification process is foundational for sharpening the Trinitarian-ness of Christian leadership.46 As Christians become more like Christ who is
43 Tenney, 158.
44 Stephen Seamands, Ministry in the Image of God: The Trinitarian Shape of Christian Service, (Downers Grove, Ill.: IVP Books, 2005), 35.
45 Colin E. Gunton, The Promise of Trinitarian Theology, (New York: T&T Clark, 2003), 116.
46 Donald Fairbairn suggests God gave Adam and Eve the Holy Spirit (Gen. 2:7) and thus they were connected to the Son and Father, sharing in the fellowship of the Trinity. Although this relationship was lost after the Fall, God gives the Spirit anew through redemption, restoring people to the state ―akin to the original sharing in the life of the Trinity that humankind lost through the Fall" [Donald Fairbairn, Life in the Trinity: An Introduction to
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DAVIS 119
one Person of the Trinity, the believer‘s leadership style will become more like the unique life of the Trinity, the original model of shared leadership. Seamands describes the unique life of the Trinity as glad submission and mutual deference between the Father, Son, and the Holy Spirit. For Seamands ―each divine person is always denying himself for the sake of the others and deferring to the others.‖47
The Trinitarian model of shared leadership, the product of the perichoretic relationship of the members of the Godhead, does not necessarily dictate that there is never a ―leader among leaders.‖ According to Scripture, even the members of the Trinity have assigned responsibilities and they function within those responsibilities. The doctrine of appropriations ―appropriates a particular activity to one of the Three, in order that we might better understand its role in the overall divine plan, and thereby grow closer to God.‖48 Seamands reminds his readers ―we say that salvation is from the Father, through the Son, and by the Holy Spirit. Our response to what God has done, whether it involves repentance, prayer, gratitude, obedience or worship, is also Trinitarian in nature; by the Spirit, through the Son, and to the Father.‖49
Anson Seers, Tiffany Keller, and James M. Wilkerson believe that shared leadership ―within self-directing work teams cannot be based on formal hierarchy.‖50 Therefore, it is their understanding that emergent leaders within teams are as important to the accomplishment of tasks as are those who hold official positions of power. Emergent leaders are those ―group members who exert significant
Theology with the Help of the Church Fathers (Downers Grove, IL: IVP Academic, 2009), 62]. Thus, as believers progress through the process of spiritual formation, they become more like Christ. Thus, they should practice the Trinitarian form of leadership.
47 Seamands, 79.
48 David Cunningham, These Three Are One, (Oxford: Blackwell, 1998), 117.
49 Semands, 120.
50 Anson Seers, Tiffany Keller, and James M. Wilkerson, ―Can Team Members Share Leadership?‖ in Shared Leadership: Reframing the Hows and Whys of Leadership, ed. Craig L. Pearce and Jay A. Conger, (Thousand Oaks, CA: Sage Publications, 2003), 80.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
120 DAVIS
influence over other members of the group although no formal authority has been vested in them.‖51 Current research of shared leadership serves as a reminder to Christian leaders that God has gifted his creation with two books of Truth: general revelation (science) and special revelation (Scripture). Although the social science community has only recently articulated the concept of shared leadership, Christian practitioners of shared leadership can with assurance, find a biblical basis for shared leadership in the example of the pre-creation, perichoretic relationship of the Trinity.52 Furthermore, shared leadership practice may be one by-product of a mature discipleship process.
Mature Discipleship via Shared Leadership
It is not the intention of this article to articulate an understanding that mature discipleship is a result of shared leadership, but that the practice of shared leadership may be a reflection of a robust discipleship process. Of course, sanctification finds its source in our union with Christ, thus Christ is the one who produces mature disciples. This truth is implied in Paul‘s description of the maturing Christian. Paul writes,
But the fruit of the Spirit is love, joy, peace, patience, kindness, goodness, faithfulness, gentleness, self-control; against such things there is
no law. And those who belong to Christ Jesus have crucified the flesh with its passions and desires. If
we live by the Spirit, let us also keep in step with the Spirit (Gal. 5:22–25 ESV).
51 Seers, Keller, and Wilkerson, 81.
52 Fairbairn believes that "the fact that God revealed his oneness first and only later his threeness means that our articulation of the doctrine of the Trinity should begin with God's oneness rather than with threeness" (41). This idea may be implied from a close reading of John 1 and Gen 1. Fairbairn seems to suggest that the "three-ness" of God (Father, Son, Holy Spirit) are only possible because of the "one-ness" of God. Again, this description may be helpful in understanding the working dynamics of shared leadership.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DaVIS 121
Christians reflect a growing sanctification through the practice of shared leadership. By embracing the Trinity‘s example of leadership, Christians can do no less than reflect the personality of God to humankind. Selfishness, greed, and thirst for power are results of humanity's fall and are often expressed through Great Man, scientific management, and bureaucratic, hierarchical leadership paradigms. According to George Cladis, perichoretic leadership is committed to developing teams that are covenanting, visionary, culture-creating, collaborative, trusting, empowering, and learning. Cladis suggests that the covenant community is the heart of Trinitarian leadership and the essence of this community is love.53
Based on the Trinitarian model of shared leadership, Robert Crosby suggests that Christians must ask themselves ―How should we conduct ourselves as Christians, as church leaders, and as collaborative teams in ministry?" Crosby sees the retrieval of the doctrine of perichoresis as necessary for Christian leadership. He writes, ―The Trinity is the premier model, or the master image, of what Christian fellowship, community, and teamwork are to look like for the purposes of edification, evangelization, and ultimately, the glorification of God on earth."54 Is it not the Christian‘s sole purpose to glorify God?
How is a Trinitarian model of shared leadership expressed in the real world? Interstate Battery System (IBS) has long embraced shared leadership and intentionally seeks to express its business philosophy through the intentionally practice of shared leadership. From the office of the CEO to the lowest strata of team members, shared leadership is the foundation of the ecosystem of IBS. According to the corporate training manual shared leadership involves four basic practices:
1. All employees (team members) are part of the ecosystem through interdependent relationships and all team members contribute to the creation of value.
53 Cladis, 10.
54 Crosby, 137.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
122 DAVIS
2. Team members aim to meet the needs of all stakeholders as an end in itself and believe that this approach leads to more value creation. (Traditionally, businesses relied heavily on trade-off or zero sum thinking—the idea that if somebody wins someone has to lose.)
3. All stakeholders win as decisions are made by all for all.
4. Team members align personal purposes with corporate purposes and those of other team members.55
Apparently, Jesus expects his disciples to model perichoresis through love of people (John 14:21). Just as believers love the Son, the Father loves believers. Christians reflect the love of God through their love for others. According to Borchert, ―the commands of the expectations of Jesus for his disciples are fully integrated into the way those disciples live.‖56 Additionally, ―participation in divine life means primarily sharing in the life of the Trinity, sharing in the relationship that has characterized the Father, Son, and Spirit from all eternity past.‖57 Jesus‘ followers appear to have the capacity to reflect God's love because of relatedness to one another through union with Christ. This deep relationship with other believers allows Christians to love one another just as the Son loves his children.
The members of the Divine Circle Dance model for disciples the value of community. God is the source of community based on the perichoretic relationship he enjoys within the Trinity. Believers can model shared leadership, both in the church and in the marketplace. They must also model community. Jesus taught his followers that love for one another serves as strong evidence of discipleship (John 13:35). Catherine Mowry LaCugna sees the practice of community as an outgrowth of growing discipleship. She
55 Interstate Battery System, "Team Member Training Manual," (Dallas, TX: 2014), 10.
56 Borchert, 128.
57 Fairbairn, 12.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DaVIS 123
writes,
The goal of Christian community, constituted by the Spirit in union with Jesus Christ, is to provide a
place in which everyone is accepted as an ineffable,
unique, and unrepeatable image of God, irrespective of how the dignity of a person might
otherwise be determined: level of intelligence,
political correctness, physical beauty, monetary
value.58
Thus, shared leadership leads to an environment that promotes love and community and in the event conflict does arise, provides for resolution through relationship.
Crosby offers several values of shared leadership through Christian living based on the Trinitarian model. First, disciples reflect the Trinity‘s practice of self-giving.59 Jesus articulated this principle in Mark 8:35: ―For whoever would save his life will lose it, but whoever loses his life for my sake and the gospel's will save it." Dallas Willard believes the ―advantage of believing in the Trinity is that we then live as if the Trinity is real: as if the cosmos environing us actually is, beyond all else, a self-sufficing community of unspeakably magnificent personal beings of boundless love, knowledge, and power."60 Growing Christians practicing shared leadership will reflect this reality in the church and the marketplace.
A second value of practicing a Trinitarian model of shared leadership is that disciples and churches become the image of God in society. According to Crosby, ―Our observations of the relationships that exist among the members of the Trinity provide powerful insight into how God wants us to relate to one another in community. Living in the image of God is no isolated discipline; it is a sacrament of community."61 John Champion amplifies Crosby‘s thoughts through the following analogy: ―If He
58 Catherine Mowry LaCugna, God for Us, (San Francisco: HarperCollins, 1991), 299.
59 Crosby, 137.
60 Dallas Willard, The Divine Conspiracy, (San Francisco: Harper, 1998), 318.
61 Crosby, 138.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
124 DaVIS
were but One Person, He would be shut up within Himself to live a lifeless life, devoid of personal interest, intercourse and content. In one way a self-existent life is a contradiction in terms. Life must be correspondent: it must be reciprocity."62 In other words, the value of shared leadership is being imago Dei in the world.
A third value of shared leadership articulated by Crosby is that community serves as the context in which God has chosen to reveal Himself.63 Without the context of community, God cannot be who he claims to be. Trinity assumes community. Disciples grow in sanctification through the context of community, as a result of union with Christ. Within this context, believers learn to practice shared leadership as a reflection of the perichoresis of the Trinity. Shared leadership or teaming not only is the best method for getting the work of the church and world accomplished, according to Crosby, ―it does something deeper in the heart of the Christ follower. It becomes a prime opportunity to glorify God and to come closer to reflecting the image of God amidst our experiences of community and collaboration."64
Some Conclusions
This article has proposed that shared leadership serves as evidence of maturing discipleship. Because believers are in union with Christ, the perichoretic relationship of the Trinity serves as a biblical model of shared leadership. Additionally, the Trinitarian roots of shared leadership suggest that the construct is not a new model, but one that has existed since the beginning of time. The world‘s oldest vocation, leadership, is best practiced through the oldest model: shared leadership.65 Therefore, at least three conclusions or suggestions may be drawn from this study of shared leadership.
62 John B. Champion, Personality and Trinity, (London: Fleming H. Revell Co., 1935), 103.
63 Crosby, 139.
64 Crosby, 139.
65 De Gourmont, 3.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
DAVIS 125
First, since shared leadership is foundationally Trinitarian and traces its roots to pre-creation times, it seems logical to assume that God desires for his children to practice shared leadership. Additionally, since Christians are expected to live as imago Dei in the world, they should not only practice shared leadership in the context of the church, but also in the marketplace. As disciples are going into the world, one means of testifying of Christ's love is to practice perichoretic or shared leadership.
Second, since God expects his followers to practice shared leadership both in the church and in the world, the church should ensure that Christians understand the principles of shared leadership and equip believers to become practitioners.66 Few churches understand the value of equipping believers for shared leadership ministry as part of their discipleship process. However, if the sanctification process is to move Christians closer to the likeness of Christ, then it is imperative that Christian educators and churches develop leadership training that is perichoretic in nature. Christian formation is ―driven by personal allegiance to God as Father, Son, and Holy Spirit—coupled with assent to specific truths about God‘s self-revelation in Jesus Christ as revealed in Holy Scripture.‖67
Finally, Christian disciples may make a dynamic impact on the secular marketplace by articulating and practicing the Trinitarian model of shared leadership in their vocational
66 According to Pearce and Conger (2003), the basic principles of shared leadership include: interactive influence among group members, mutual leadership, achievement of group goals, peer involvement in leadership process, periodic up or down influence, and leadership as an activity that can be shared among group members. These characteristics seem to reflect the life of the Trinity. Walter Kasper describes a ―symmetrical representational model,‖ which depicts this Trinitarian life as a circular movement [Walter Kasper, The God of Jesus Christ, trans. Matthew J. O‘Connell (New York: Crossroads, 1984), 216]. Additionally, the concept of the Father as the greater among equals is maintained by Augustine‘s writings as he insists that the Spirit who proceeds from both the Father and the Son does so principally from the Father [Augustine, On the Trinity 15.17, 26].
67 Jones and Wilder, 194.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
126 DaVIS
life. By strategically identifying the source of leadership style to co-workers, employees, superiors, and clients, Christians will be able to glorify God in their work.
Paul‘s admonitions to ―seek the things that are above, where Christ is seated at the right hand of God" (Col. 3:1b), and to ―Set your minds on things that are above, not on things that are on earth" (Col. 3:2) elevate the work of Christians beyond the scope of wealth and success. Paul identifies the reason believers can comply with his commands: ―For you have died, and your life is hidden with Christ in God" (Col. 3:3). Christians can make their work about Christ because they are in unity with Christ and God through perichoresis. The Apostle Paul reminds Christians that they are to ―put off the old self with its practices" and to ―put on the new self which is being renewed in knowledge after the image of its creator" (Col. 3:9–10). No doubt, Christian leaders need to renew their knowledge of leadership practice as they grow in likeness of Christ through spiritual formation.
It is not unreasonable to believe that God has given Christians a perfect model for leadership through the example of the Trinity‘s perichoretic relationship and the believer‘s union with Christ. The spiritual formation of maturing Christians can lead to the practice of shared leadership as they become more like Christ. The old rules do still apply, at least for Christian disciples seeking to grow in their spiritual formation.
Michael L. Davis, Ed.D., is associate pastor at First Baptist Church in Bremen, Georgia, an adjunct professor at New Orleans Baptist Theological Seminary, and a church consultant.
Journal of Religious Leadership, Vol. 14, No. 1, Spring 2015
Feature Selection for Face Recognition Based on
Multi-Objective Evolutionary Wrappers
Leandro D. Vignolo,a, Diego H. Milonea, Jacob Scharcanskib
aGrupo de Investigaci´on en Se˜nales e Inteligencia Computacional, Departamento de
Inform´atica, Facultad de Ingenier´ıa y Ciencias H´ıdricas, Universidad Nacional del
Litoral, CONICET, Argentina.
bInstituto de Informatica and Dept. de Engenharia Eletrica, Universidade Federal do Rio
Grande do Sul, Caixa Postal 15064, 91501-970, Porto Alegre, RS, Brasil.
Abstract
Feature selection is a key issue in pattern recognition, specially when prior knowledge of the most discriminant features is not available. Moreover, in order to perform the classification task with reduced complexity and accept¬able performance, usually features that are irrelevant, redundant, or noisy are excluded from the problem representation. This work presents a multi-objective wrapper, based on genetic algorithms, to select the most relevant set of features for face recognition tasks. The proposed strategy explores the space of multiple feasible selections in order to minimize the cardinality of the feature subset, and at the same time to maximize its discriminative capacity. Experimental results show that, in comparison with other state-of-the-art approaches, the proposed approach allows to improve the classification per¬formance, while reducing the representation dimensionality.
Key words: wrappers, multi-objective genetic algorithms, feature selection, face recognition.
1. Introduction
Face recognition has received significant attention due to its promising applications in security systems and human-computer interaction, which has
Corresponding author. Tel.: +54(342)4575233 ext 191, FAX: +54(342)4575224. Email addresses: ldvignolo@fich.unl.edu.ar (Leandro D. Vignolo ), d.milone@ieee.org (Diego H. Milone), jacobs@inf.ufrgs.br (Jacob Scharcanski)
Preprint submitted to Expert Systems with Applications February 7, 2013
motivated important new developments in research areas such as image pro-cessing and artificial intelligence. In general, the methodologies are devel¬oped for face images acquired under controlled conditions, but in practical situations, face recognition systems usually must also deal with changing con-ditions like variations in pose, expression and illumination, which introduce intra-class variability in the extracted features with respect to the training data [1, 2, 3]. In a face recognition problem, a given face image is classified into K previously known face classes. This is usually done using a model trained with the feature vectors extracted from a database of face images [4, 5].
Two main approaches exist in face recognition, those which are based on holistic methods and the others based on analytic techniques [6]. Holistic methods, such as eigenfaces [7], use global characteristics of the face images. On the other hand, analytic techniques, like the Active Shape Models (ASM) [8, 9], extract face features related to the eyes, the nose, the mouth, etc.
In facial modeling with ASM, a number of points (i.e. image locations) are selected from an input image, but only some of these points are useful for characterizing the face, since the others have small contributions to dis-crimination, or are noisy. As the training of ASM converges towards salient edges, if these edges are distorted by noise or some other artifact, like local illumination variation, erroneous feature matchings might arise [10]. Despite recent improvements made to ASM techniques, the matching errors may be undesirably high at some face locations [11, 12]. Even after some new implementations that improve the landmark location accuracy, the detec¬tion of facial features with varying pose and illumination is still challenging [13, 1]. Usually, once a set of face image locations (i.e. points) is selected by the ASM method, a number of features describing each face location is ex¬tracted. Then, the resulting feature vectors representing the faces are usually of high dimensionality, which makes the classification task more difficult [14]. Also, large feature sets are prone to overfitting and, hence, to achieve poor generalization performance [15].
In [10], the authors proposed to improve the ASM performance in face recognition by weighting the facial features according to a method based on adjusted mutual information. As the authors shown, this criterion al¬lowed the selection of the most relevant landmark points, in order to im¬prove the face classification results. However, the flexibility provided by the full set of features obtained by the ASM approach has not yet been fully explored by means of feature selection techniques, in order to reduce the
2
dimensionality of the representation while improving the face classification results. On the other hand, significant progresses have been made with the application of different artificial intelligence techniques for feature selection. In particular, many works rely on evolutionary algorithms for feature sub¬set optimization [16, 17, 18, 19], and for the search of optimal representa¬tions [20, 21, 22, 23]. In [24] a genetic wrapper was proposed for the selection of the most relevant features for improving the accuracy of face recognition. Nevertheless, this wrapper was focused on classification accuracy improve-ment, which limits the proposed method since it overlooks other important issues in face classification (e.g. feature space dimensionality and class over-lap).
In order to guide the search within the space of feasible face classifica¬tion solutions, here we propose the use of a Multi-Objective Genetic Algo¬rithm [25]. This method allows to overcome the above mentioned limitations by maximizing the face classification accuracy, while minimizing the number of features and the mutual information. Two different strategies for the rep-resentation of the candidate solutions are proposed and compared, and the generalization performance of the feature subset selection is assessed using an independent data set.
The organization of this paper is as follows. First a brief introduction to the use of ASM for face modeling is given in Section 2, and next our multi-objective wrapper for the selection of features for face classification is presented in Section 3. Section 4 describes our experiments and discuss the results obtained for face classification. Finally, our conclusions and ideas for future work are presented in Section 5.
2. Active Shape Models for Facial Recognition Applications
The ASM approach is used to represent shapes and their expected ways of deforming as learned from a training set. For this, it uses flexible point distribution models (PDM), based on the positioning of selected points in the face image examples [11]. This PDM iteratively deforms to fit the shape of an object, constrained to vary in the way learned from a set of training examples. When applied to face recognition, the ASM is trained on a set of sample faces, and N points are used to represent the shape of each face within its class (see Figure 1(a)).
Nevertheless, matching errors may arise in the location of the PDM points, often called landmarks, in a face image (see Figure 1(b)) [10]. Then,
3
Figure 1: Illustration of the landmark points used to model a face (a) and their location on an image (b) [10].
considering a training image set with K face classes, each class k = 1, ..., K is represented by N landmark points Sk,ǫ = {pi(xi + ǫxi, yi + ǫyi)k}, where i = 1, ..., N, (xi, yi) are the coordinates of the landmark point pi and (ǫxi, ǫyi) are the respective location errors. Every relevant facial characteristic (e.g. eye centers, mouth contours, etc.) is represented by a set of landmarks pi, and the particularities of each point in the image are described by Q fea¬tures (e.g. chrominance, texture, etc.). The features at landmark pi will be denoted {Fj,i}, with j = 1, ..., Q.
In order to describe each one of the N landmark points pi, the mean µFj,i and the variance σ2Fj,i of the measurements of each feature j taken within a defined neighborhood of that point are commonly used [10]. These are computed for all features Fj,im, with m = 1, ... , M, where M is the number of training image samples,
µj,i(r, q), (1)
{ }
σ2 Fj,i = max σ2 j,i(r, q) , r,qW (2)
where (r, q) are the pixel coordinates within the window W (of size w × w), centered at the landmark point pi [10], µj,i(r, q) = 1 ~M m=1 F m
M j,i(r, q) and
σ2j,i(r, q) = 1_M(Fm )2.
M m=1 j,i(r, q) µj,i(r, q)
To consider the feature variability within the w × w neighborhood of land¬mark pi, the maximum window variance was used in (2). The window size was set to w = 2 max {σǫ}, where σǫ is the standard deviation of landmark location errors, measured during ASM training. The probability density of
4
location errors at each landmark point is assumed to be approximately Gaus¬sian [26].
In this work, the face detector proposed by Demirel et al. [27] is used, and the process applied to the database of face images in order to obtain the ASM-based set of features is described in detail in [24, 10].
3. Multi-objective Wrapper for Face Feature Selection
Genetic algorithms (GAs) are meta-heuristic optimization methods, in¬spired on the process of natural evolution, that are capable of finding global optima in complex search spaces [28]. These optimization algorithms need to evaluate a problem-dependent objective function to guide the search. How¬ever, in most real-world problems we may be interested in satisfying more than one objective, and the optimization of one objective may conflict with the other objectives. In general, the solution of a multi-objective optimiza¬tion problem is not a single point, but a set of points known as the Pareto optimal front [29].
Different modifications to the traditional GAs were proposed in order to tackle multi-objective problems [30]. One generic approach is to combine the individual objective functions into a single aggregative function, or to consider all but one objective as constraints. Another generic approach is to determine a Pareto optimal, or nondominated set of solutions. This means, a set of solutions for which none of the objective values can be improved without detriment in some of the other objective functions. This approach takes advantage of the population-based nature of GAs, which allows the generation of several elements of the Pareto set in a single run [25].
Particularly, the Multi-Objective Genetic Algorithm (MOGA) is a vari¬ation of the classical GA, in which the rank of an individual is the num¬ber of chromosomes in the population by which it is dominated [30]. This technique addresses the search toward the true Pareto front, while maintain¬ing diversity in the population [31]. A problem that arises in Pareto based multi-objective evolutionary algorithms is the difficulty to preserve diversity among Pareto optimal solutions. The population tends to scatter around the existing optima forming stable sub-populations, or niches. One approach to overcome this difficulty, which is based on the concept of niching around promising points, makes use of a sharing function as proposed by Fonseca and Fleming [30]. Fitness sharing allows the MOGA to maintain the population diversity while encouraging the search for solutions in unexplored sections of
5
Figure 2: General scheme of the proposed multi-objective wrapper.
a Pareto front. This is accomplished by reducing the fitness of solutions in densely populated areas of the search space [29]. The MOGA, as other fit-ness sharing techniques, uses the parameter σs to define the size of the niche around a point in the Pareto front [31]. In this way, the nearby solutions are penalized in order to maintain population diversity, and to promote the search around all the salient peaks in the domain of feasible solutions.
Here we propose and study three different wrappers for feature selection in face recognition applications. The first wrapper is a classical GA, in which each individual represents a particular selection of the set of facial features ex¬tracted from an input image by means of ASM. The second wrapper that we propose is a multi-objective GA with an aggregative fitness function, which combines classification accuracy and the number of features in a single equa¬tion. Finally, we propose a third wrapper which consists of a MOGA, with the same objective functions considered for the second alternative. Addition¬ally, in this case we also use mutual information as an additional objective, in order to minimize the interdependence of the selected features. The proposed multi-objective wrapper method is described as a diagram in Figure 2.
The selection of individuals is done considering the set of coefficients rep¬resented by each chromosome, using the tournament selection scheme. This consists on choosing a few individuals at random from the population in or¬der to run a competition, from which the winner is selected for reproduction. To evaluate a particular individual, a set of images is used to compute the objective functions. In order to perform the evaluation, first, the feature vec¬tors that represent the images are assembled with the coefficients indicated by the chromosome.
The classical mutation and one-point crossover are used, and an elitist replacement strategy is applied in order to maintain the best individual for the next generation.
6
3.1. Fitness Functions
In the proposed multi-objective wrappers, one of the target functions evaluates the feature set suggested by a given chromosome, providing a mea-sure of the face classification accuracy. Therefore, a classifier is used as the first objective function, so that the success classification rate is considered for each evaluated individual. In order to guide the search, while maintaining a low computational cost, a simple classifier algorithm was considered. This classifier assigns the test face image, represented by its feature vector, to the class with the closest prototype (mean feature vector). The mean is first computed based on the feature vectors in the training set, and the Euclidean norm was used as distance in our experiments. Then, after an optimized solution is found, the k-nearest neighbors (KNN) classifier [14] is used to evaluate the classification performance on the test set.
It shall be observed that it is also beneficial to obtain a face image rep-resentation containing the smallest number of coefficients, which should be help in face image classification task, as discussed next.
3.1.1. Aggregative Fitness Function
For the aggregative approach we used a fitness function that combines classification accuracy and the number of features in a single equation. The proposed aggregative fitness function is:
Fa = αF1 + 1 α F2 , (3)
where α is a parameter that assigns a relevance to each objective. The first term of Fa corresponds to the prediction accuracy, F1 (the fitness function used in the standard GA), and F2 accounts for the the number of selected features. In our experiments we adjusted α between 0.7 and 0.9. The second objective function is defined as
~ ~
F2 = 100 1 n , (4)
L
so that we obtain a number in the same range as the classification rate. Here, n is the number of coefficients selected by the chromosome, and L is the length of the chromosomes.
7
Algorithm 1: Population evaluation in the proposed wrapper.
for each individual in the population do
Re-parameterize the face images using the features selected by the chromosome
(given the complete set of ASM features obtained using (1))
Train the classifier with the training set
Test the classifier with the validation set
Assign classification rate as the current value for F1
Assign the current value for F2, based on the number of features (4)
Assign the current value for F3, based on MI (5)
In the case of the aggregative AG, compute the total fitness (3)
3.1.2. Proposed Multi-Objective approach
For the proposed MOGA we used the objective functions F1 and F2, defined in (3) and (4), respectively. Also, we used an additional objective function designed to minimize the mutual information (MI) of the selected coefficients. We computed the MI for every pair of coefficients on the training data using the method proposed by Peng et al. [32]. We defined this third objective function as
M*
F3 = 1 + LM/n, (5)
where M* is the sum of the MI calculated for all the available features (taken in pairs), and M is the sum of the mutual information calculated for the features selected by a chromosome.
Considering the three proposed target functions, all steps in the evalua¬tion of a population by the proposed multi-objective wrapper are detailed in the Algorithm 1.
3.2. Chromosome Codification
In this work, the mean of the color chrominance channels Cr and Cb of the YCbCr color space were used as features for describing each of the 68 ASM landmark points, meaning that the the dimensionality of a complete feature vector is N ×Q = 136 [10]. We considered two different approaches for coding the chromosomes, yielding search spaces of significantly different sizes. In the first case, each gene represents a particular feature, independently of the landmark point associated to it. Thus, in this approach the chromosome size is 136, and each feature associated to a given landmark point can be selected individually and independently. In the second chromosome coding
8
alternative, each gene in a chromosome represents one of the ASM landmark points, so the chromosome value indicates whether the corresponding features are used or not, and hence the chromosome size is reduced to 68. In both coding alternatives, the initialization consists on a random selection of the genes (values) in the chromosomes, since no restriction was applied to the re-combinations of features.
4. Experimental Results and Discussion
A set of face images from the Essex Face Database was used in our experi¬ments [33], which contains a significant diversity of individuals and expression changes. In order to make a comparative evaluation of our experimental re¬sults with respect to other approaches available in the literature, 100 face classes were used. Five face images per class were randomly selected for training and other fifteen face images per class were separated for the test set [10].
As stopping criteria for the optimization, we considered a maximum of 500 generations, and convergence was assumed after 100 generations without fitness improvement. After the optimization step, the classification perfor¬mance with the selected feature subsets was evaluated on the test set, which was not used for the feature selection process. That is, the data from the test set was not used for the fitness evaluation during the optimization, which allowed to estimate the generalization performance of the optimized feature subsets. This test was performed employing a KNN classifier (with k = 1). We carried out several optimization experiments, considering different al¬ternatives and combination of parameters, and here we discuss the most relevant.
The experimental results are presented and discussed next. Section 4.1 discusses the experiments performed with the most simple approach studied in this paper, using the single-objective wrapper. Then, in Section 4.2, the results obtained with the proposed multi-objective strategies (i.e. aggregative GA and MOGA) are addressed. Finally, Section 4.3 presents a comparative analysis of the obtained results.
4.1. Single-Objective Optimization
In this Section, we first describe the experiments that involve chromo-somes of length 136 (as explained before), which will be referred to as GA-136. The classifier described in Section 3.1 was used in the evolution, which
9
was evaluated on the training data set in order to compute the fitness of each candidate solution. The GA population consisted of 30 individuals, and crossover and mutation probabilities were set to 0.8 and 0.025, respectively. In this case, the proposed GA converged to a set of 62 features, and the KNN classifier achieved an accuracy of 97.20% on the test data set.
Another set of experiments were conducted with single-objective opti-mization and GA-136 chromosomes. In order to obtain a better general¬ization performance, we enlarged the training data set using the Smoothed Bootstrap Resampling (SBR) method [34]. When the amount of data is not enough to ensure statistically significance, this method can be used to create new samples by adding noise to the feature values of the original samples. In particular, zero mean Gaussian noise with σ = 0.1 was used in our exper¬iments, since this value allowed to preserve the variance of the original train data. Accordingly, in the next experiments (GA-136+SBR), 20 SBR exam¬ples were generated for each class in order to perform the fitness evaluations. After the convergence of the GA, 68 features were selected, which allowed the classifier to achieve an accuracy of 97.40% on the test data. Therefore, we can infer that the resampling of the training data allows better generalization.
However, compared with the previous case, a larger subset of features was selected. A plot of the maximum fitness value obtained as the number of generations is increased is shown in Figure 3(a). Note that the convergence of the GA required about 220 generations in this experiment.
The following approach tested, as explained in Section 3, consisted in re-ducing the length of the chromosomes to the number of landmark points (68). This means that, within each chromosome, the selection of a given landmark implies that both of the corresponding features are used. As a result of this experiment, referred to as GA-68+SBR, we obtained a reduced feature set of size 56. With this feature set we obtained 98.0% of classification accu¬racy on the test data set, suggesting that the reduction of the chromosome size simplified the search space, making the search easier for the GA. Figure 3(b) shows the evolution of the fitness value, and it can be verified that the best solution was found after only 63 generations. When compared to Figure 3(a), it suggests that the codification strategy with smaller chromosomes, in addition to the resampled training data set, allowed a faster convergence of the GA.
10
Generations
(a)
Generations
(b)
Figure 3: Convergence of the GA in the experiments: a) GA-136+SBR and b) GA-68+SBR.
4.2. Multi-Objective Optimization
In this section, we discuss the experimental results obtained by using the simultaneous optimization of multiple objectives. We first used a classical GA with the aggregative fitness function given in (3), taking into account the number of features besides the classification accuracy. As in the previous case, we studied both the codification alternatives with chromosome lengths 136 and 68, and used SBR samples for training.
Figures 4(a) and 4(b) show the convergence plots for the optimizations using chromosomes of length 136, and the aggregative fitness function with α = 0.8 and α = 0.85, respectively. In the first case, GA-Aggre-2ob-136+SBR (α = 0.8), the GA converged to a set of only 32 features, and the KNN classifier achieved an accuracy of 97.40% on the test data set. With a similar experiment but using α = 0.85, we obtained a set with ten additional features (42), which lead to a small improvement on classification accuracy of the test set (97.80%).
On the other hand, conducting the same experiments indicated above, but using chromosomes of length 136, we obtained a subset of 46 features with classification accuracy of 97.60%, and a subset of 56 features giving an accuracy of 97.80% on the test set, with α = 0.8 and α = 0.85, respec¬tively. For these experiments, the fitness behaviors for different generations are shown in Figures 4(c) and 4(d). It is noticeable that the convergence of the GA takes a longer time to optimize two objectives simultaneously, in contrast to the optimizations with a single objective discussed before.
The last group of experiments consists in using a MOGA to optimize two and three objectives simultaneously. In addition to classification accuracy
11
and the number of features, in these experiments we also considered the minimization of the mutual information between selected features as a third objective. For the problem in hand, we obtained the most interesting results when σ3 was set to 0.09 and 0.1.
Several optimization experiments were conducted with the MOGA, first combining classification accuracy and the number of features, and then also including the mutual information measurement. Performing the optimization with two objectives (MOGA-2ob, as with the aggregative GA) and chromo-somes of length 136, we obtained a subset of 37 features (σ3 = 0.09) giving an accuracy of 97.30% on the test set, and subset of 32 features (σ3 = 0.1) giving an accuracy of 96.67% on the test set. With chromosomes of length 68, we obtained a subset of 38 features giving an accuracy of 97.53%, and a subset of 30 features giving an accuracy of 97.30% on the test set. In this way, we compare the MOGA and the aggregative GA, showing that the per¬formances of both are similar, except for a slight improvement of the MOGA in the later case.
On the other hand, when we also consider the minimization of mutual information (MOGA-3ob). We obtained a subset of only 26 features giving an accuracy of 97.00% (σ3 = 0.09), and a subset of 30 features which obtained 97.53% of accuracy on the test set (σ3 = 0.09), with chromosomes of length 136. Finally, with chromosomes of length 68, we obtained a subset of 36 features giving an accuracy of 97.93% (σ3 = 0.09), and a subset of 38 features giving 98.00% of accuracy on the test set (σ3 = 0.09).
4.3. Comparative Analysis and Discussion
Table 1 summarizes the results of the aforementioned experiments, and compares the performances obtained by the optimized subsets of features with two different approaches representing the state of the art. The second column shows the classification accuracy achieved by the different feature sets, obtained with the proposed wrapper optimization method on the test data set, and the third column shows the number of features involved. The last column exhibits the relative error reduction (RER) with respect to the Enhanced ASM [10], meaning the percentage by which the error rate is re¬duced. As illustrated by this table, the optimized representations obtained by the evolutionary wrappers obtained better classification performances. It should be observed that these optimized representations provided larger fea¬ture sets when compared to the Enhanced ASM. However, the feature set
12
Table 1: Classification results obtained for the test data
Method
Accuracy Number of features Relative error reduction
DFBFR [27] 93.73% 2 × 1002 -
Enhanced ASM [10] 95.33% 54 (reference)
GA-136 96.93% 62 34.26%
GA-136+SBR 97.40% 68 44.33%
GA-68+SBR 98.00% 56 57.17%
GA-Aggre-2ob-136+SBR (α = 0.8) 97.40% 32 44.33
GA-Aggre-2ob-136+SBR (α = 0.85) 97.80% 42 52.89
MOGA-2ob-136+SBR (σ3 = 0.09) 97.30% 37 42.18
MOGA-2ob-136+SBR (σ3 = 0.1) 96.67% 32 28.69
MOGA-3ob-136+SBR (σ3 = 0.09) 97.00% 26 35.76
MOGA-3ob-136+SBR (σ3 = 0.1) 97.53% 30 47.11
GA-Aggre-2ob-68+SBR (α = 0.8) 97.60% 46 48.61
GA-Aggre-2ob-68+SBR (α = 0.85) 97.80% 56 52.89
MOGA-2ob-68+SBR (σ3 = 0.09) 97.53% 38 47.11
MOGA-2ob-68+SBR (σ3 = 0.1) 97.30% 30 42.18
MOGA-3ob-68+SBR (σ3 = 0.09) 97.93% 36 55.67
MOGA-3ob-68+SBR (σ3 = 0.1) 98.00% 38 57.17
Additionally, the minimization of the mutual information as a third objective provides solutions with a better compromise between classification error and the number of features. However, it is important to observe that in this experiments we favor solutions that provide high classification accuracy more than those with fewer features.
An interesting performance analysis can be obtained by changing the 100-class problem into a binary classification task, and then computing the ROC curve according to the methodology proposed in [35]. For this binary classification task we took the 15 test patterns of a given class and assigned them as the registered user class, and all of the remaining test patterns, from the other 99 classes, were assigned to the unregistered user class. This was repeated for each of the 100 classes (each time a different class was labeled as registered) and the classification results obtained were averaged. As the unregistered users are unknown, the training patterns corresponding to this class were not used in the classification (we used only the patterns corre¬sponding to the registered user class). Instead of using the KNN classifier, the rule to classify the test samples was based on the Euclidean distance to the training samples of the registered user class. This rule can be described simply as follows: if the distance from the test image to each of the train
14
FPR (1specificity)
Figure 5: ROC curve generated by varying the threshold ä in the binary classification task. The solid line corresponds to the MOGA-3ob-68+SBR, the dashed line to GA-68+SBR, and the dash-dot line to the complete feature set.
ing (registered) users is less than the threshold 6, it is labeled as registered; otherwise the test image is classified as unregistered.
Figure 5 shows the ROC curves constructed with the true positive rate (TPR) and false positive rate (FPR) indexes, obtained by averaging the re¬sults for the 100 binary classification tests. The classification performance obtained with the feature set MOGA-3ob-68+SBR (solid line), with the fea¬ture set GA-68+SBR (dashed line), and with the complete feature set (dash-dot line), for different values of threshold 6 (varying from 0 to 120) are shown. High TPR values indicate that most of the test samples that belong to the registered class are correctly classified. On the other hand, high values of FPR occur when unregistered samples are labeled as registered. As can be seen in Figure 5, to obtain the highest TPR we need to tolerate a FPR differ¬ent of zero. It is important to observe that our optimized feature sets allow to improve on the classification results obtained with the complete feature set, obtaining a higher TPR without increasing the FPR. Also, analyzing the ROC curves it can be noticed that the 38-feature set obtained with the MOGA shows a significant improvement in classification performance with respect to the 56-feature set obtained with the classical GA (the same obser¬vation applies to the complete feature set). This confirms our hypothesis that it is beneficial to minimize the size of the feature set. Also, it can be noticed that the use of the resampling method allowed to obtain better results.
15
5. Conclusions and future Work
This paper presented and compared multi-objective wrappers, based on evolutionary computation techniques, designed to optimize the feature se-lection process in face image classification. The proposed wrappers provide feature sets of different sizes and face class discrimination capabilities, and the choice of the most appropriate wrapper should be guided by the require¬ments of the problem in hand (e.g. reduced feature set combined with a high classification accuracy, or just focus on high classification accuracy). The experiments were performed on a well known face image data set, where the face images were represented using the ASM approach. These experiments revealed that the optimized feature sets offer improved classification accu¬racy in comparison with other state of the art approaches. Probably because these optimized face representations provide better class separability in the feature space, while simplifying the classification task. Furthermore, the dimensionality of the ASM-based representation was significantly reduced, which also helps to avoid overfitting. Hence, the proposed strategy provides a valid alternative for the selection of relevant features for face recognition.
In the future, we plan to perform experiments with a larger data set, with increased variability of pose and illumination, and explore other options in terms of feature set optimization. We would like to explore other multi-objective optimization algorithms such as PESA-II or NSGA-II [25], in order to compare the performance with the MOGA. Also, a measure of compactness [36] could be also considered as objective function in order to improve the clustering of classes in our evolutive wrapper. Moreover, we would consider the use of other heuristic search methods, such as particle swarm [37, 38] and scatter search [39].
Acknowledgment
The authors would like to thank SPU (Secretar´ıa de Pol´ıticas Univer-sitarias, Ministerio de Educaci´on, Argentina) and CAPES (Coordenadoria de Aperfei¸coamento de Pessoal de Ensino Superior, Brazil) for financial sup¬port, and the Vision Group from the University of Essex (UK), for providing the face image database. Also, the authors wish to acknowledge the sup¬port provided by Agencia Nacional de Promoci´on Cient´ıfica y Tecnol´ogica and Universidad Nacional de Litoral (with PAE-PICT 00052, CAID 012-72), and Consejo Nacional de Investigaciones Cient´ıficas y T´ecnicas (CONICET) from Argentina.
16
References
[1] S. Milborrow, F. Nicolls, Locating facial features with an extended ac¬tive shape model, in: D. Forsyth, P. Torr, A. Zisserman (Eds.), Com¬puter Vision - ECCV 2008, Vol. 5305 of Lecture Notes in Computer Science, Springer Berlin / Heidelberg, 2008, pp. 504–513, 10.1007/978¬3-540-88693-8-37.
[2] S. Z. Li, A. K. Jain, Handbook of Face Recognition, Springer, 2011.
[3] Y. Wen, An improved discriminative common vectors and support vector machine based face recognition approach, Expert Systems with Appli-cations 39 (4) (2012) 4628 – 4632. doi:10.1016/j.eswa.2011.09.119.
[4] H. Cevikalp, B. Triggs, Face recognition based on image sets, in: Com-puter Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, 2010, pp. 2567–2573. doi:10.1109/CVPR.2010.5539965.
[5] S.-K. Oh, S.-H. Yoo, W. Pedrycz, Design of face recognition al¬gorithm using PCA-LDA combined for hybrid data pre-processing and polynomial-based RBF neural networks: Design and its appli¬cation, Expert Systems with Applications 40 (5) (2013) 1451–1466. doi:10.1016/j.eswa.2012.08.046.
[6] S. G. Kong, J. Heo, B. R. Abidi, J. Paik, M. A. Abidi, Re¬cent advances in visual and infrared face recognition-a review, Com¬puter Vision and Image Understanding 97 (1) (2005) 103–135. doi:10.1016/j.cviu.2004.04.001.
[7] M. Turk, A. Pentland, Face recognition using eigenfaces, in: Com¬puter Vision and Pattern Recognition, 1991. Proceedings CVPR ’91., IEEE Computer Society Conference on, 1991, pp. 586–591. doi:10.1109/CVPR.1991.139758.
[8] T. Cootes, C. Taylor, D. Cooper, J. Graham, Active shape models - their training and application, Computer Vision and Image Understanding 61 (1) (1995) 38–59.
[9] C. Wang, Y. Li, X. Song, Video-to-video face authentication system ro-bust to pose variations, Expert Systems with Applications 40 (2) (2013) 722–735. doi:10.1016/j.eswa.2012.08.009.
17
[10] C. Behaine, J. Scharcanski, Enhancing the performance of ac¬tive shape models in face recognition applications, Instrumentation and Measurement, IEEE Transactions on 61 (8) (2012) 2330–2333. doi:10.1109/TIM.2012.2188174.
[11] A. Hill, T. Cootes, C. Taylor, Active Shape Models and the shape approximation problem, Image and Vision Computing 14 (8) (1996) 601–607, 6th British Machine Vision Conference. doi:10.1016/0262-8856(96)01097-9.
[12] J. Kim, M. C¸etin, A. S. Willsky, Nonparametric shape priors for ac¬tive contour-based image segmentation, Signal Processing 87 (12) (2007) 3021 – 3044, special Section: Information Processing and Data Manage¬ment in Wireless Sensor Networks. doi:10.1016/j.sigpro.2007.05.026.
[13] Z. Zheng, J. Jiong, D. Chunjiang, X. Liu, J. Yang, Facial feature local-ization based on an improved active shape model, Information Sciences 178 (9) (2008) 2215–2223.
[14] C. M. Bishop, Pattern Recognition and Machine Learning, 1st Edition, Springer, 2007.
[15] J. Handl, J. Knowles, Feature subset selection in unsupervised learning via multiobjective optimization, International Journal of Computational Intelligence Research 2 (3) (2006) 217–238.
[16] S. Chatterjee, A. Bhattacherjee, Genetic algorithms for feature selection of image analysis-based quality monitoring model: An application to an iron mine, Engineering Applications of Artificial Intelligence 24 (5) (2011) 786 – 795. doi:10.1016/j.engappai.2010.11.009.
[17] Y.-X. Li, S. Kwong, Q.-H. He, J. He, J.-C. Yang, Genetic algorithm based simultaneous optimization of feature subsets and hidden markov model parameters for discrimination between speech and non-speech events, International Journal of Speech Technology 13 (2010) 61–73, 10.1007/s10772-010-9070-4.
[18] H.-H. Hsu, C.-W. Hsieh, M.-D. Lu, Hybrid feature selection by com¬bining filters and wrappers, Expert Systems with Applications 38 (7) (2011) 8144 – 8150. doi:10.1016/j.eswa.2010.12.156.
18
[19] W. Pedrycz, S. S. Ahmad, Evolutionary feature selection via structure retention, Expert Systems with Applications 39 (15) (2012) 11801 – 11807. doi:10.1016/j.eswa.2011.09.154.
[20] L. D. Vignolo, D. H. Milone, H. L. Rufiner, Genetic wavelet packets for speech recognition, Expert Systems with Applications 40 (6) (2013) 2350 – 2359. doi:10.1016/j.eswa.2012.10.050.
[21] L. D. Vignolo, H. L. Rufiner, D. H. Milone, J. C. Goddard, Evolution¬ary Splines for Cepstral Filterbank Optimization in Phoneme Classifica¬tion, EURASIP Journal on Advances in Signal Processing Volume 2011, doi:10.1155/2011/284791.
[22] L. D. Vignolo, H. L. Rufiner, D. H. Milone, J. C. Goddard, Evolutionary Cepstral Coefficients, Applied Soft Computing 11 (4) (2011) 3419 – 3428. doi:10.1016/j.asoc.2011.01.012.
[23] C. Charbuillet, B. Gas, M. Chetouani, J. Zarader, Optimizing fea¬ture complementarity by evolution strategy: Application to automatic speaker verification, Speech Communication 51 (9) (2009) 724 – 731, special issue on non-linear and conventional speech processing.
[24] L. D. Vignolo, D. H. Milone, J. Scharcanski, C. Behaine, An Evo¬lutionary Wrapper for Feature Selection in Face Recognition Ap¬plications, in: Systems, Man, and Cybernetics (SMC), 2012 IEEE International Conference on, Seul (Corea), 2012, pp. 1286–1290. doi:10.1109/ICSMC.2012.6377910.
[25] C. Coello Coello, G. Lamont, D. Van Veldhuizen, Evolutionary Al-gorithms for Solving Multi-Objective Problems, 2nd Edition, Genetic and Evolutionary Computation, Springer, Berlin, Heidelberg, 2007. doi:10.1007/978-0-387-36797-2.
[26] J. Shi, A. Samal, D. Marx, How effective are landmarks and their geom-etry for face recognition?, Computer Vision and Image Understanding 102 (2) (2006) 117 – 133. doi:10.1016/j.cviu.2005.10.002.
[27] H. Demirel, G. Anbarjafari, Data fusion boosted face recognition based on probability distribution functions in different colour chan¬nels, EURASIP J. Adv. Signal Process 2009 (25) (2009) 25:1–25:10. doi:10.1155/2009/482585.
19
[28] H. Youssef, S. M. Sait, H. Adiche, Evolutionary algorithms, simulated annealing and tabu search: a comparative study, Engineering Applica¬tions of Artificial Intelligence 14 (2) (2001) 167 – 181. doi:10.1016/S0952-1976(00)00065-8.
[29] H. Kim, M.-S. Liou, New fitness sharing approach for multi-objective genetic algorithms, Journal of Global Optimization (2012) 1–17. doi:10.1007/s10898-012-9966-4.
[30] C. M. Fonseca, P. J. Fleming, Genetic algorithms for multiobjective optimization: Formulation discussion and generalization, in: Proceed¬ings of the 5th International Conference on Genetic Algorithms, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993, pp. 416–423.
[31] A. Konak, D. W. Coit, A. E. Smith, Multi-objective optimization us¬ing genetic algorithms: A tutorial, Reliability Engineering and System Safety 91 (9) (2006) 992 – 1007. doi:10.1016/j.ress.2005.11.018.
[32] H. Peng, F. Long, C. Ding, Feature selection based on mutual informa¬tion criteria of max-dependency, max-relevance, and min-redundancy, Pattern Analysis and Machine Intelligence, IEEE Transactions on 27 (8) (2005) 1226–1238. doi:10.1109/TPAMI.2005.159.
[33] Vision Group, University of Essex (UK), Face recognition data (2007). URL http://cswww.essex.ac.uk/mv/allfaces/faces94.html
[34] G. A. Young, Alternative smoothed bootstraps, Journal of the Royal Statistical Society. Series B (Methodological) 52 (3) (1990) pp. 477–484.
[35] R. Bolle, J. Connell, S. Pankanti, N. Ratha, A. Senior, The relation between the roc curve and the cmc, in: Automatic Identification Ad¬vanced Technologies, 2005. Fourth IEEE Workshop on, 2005, pp. 15 – 20. doi:10.1109/AUTOID.2005.48.
[36] G. Stegmayer, D. H. Milone, L. Kamenetzky, M. G. L´opez, F. Carrari, A biologically-inspired validity measure for comparison of clustering meth-ods over metabolic datasets, IEEE ACM Transactions in Computational Biology and Bioinformatics 9 (3) (2012) 706–716.
20
[37] C.-Y. Tsai, I.-W. Kao, Particle swarm optimization with selective par¬ticle regeneration for data clustering, Expert Systems with Applications 38 (6) (2011) 6565 – 6576. doi:10.1016/j.eswa.2010.11.082.
[38] J. Kennedy, R. Eberhart, Particle swarm optimization, in: IEEE Inter-national Conference on Neural Networks, Vol. 4, 1995, pp. 1942–1948. doi:10.1109/ICNN.1995.488968.
[39] R. Mart, M. Laguna, F. Glover, Principles of scatter search, Eu¬ropean Journal of Operational Research 169 (2) (2006) 359 – 372, feature Cluster on Scatter Search Methods for Optimization. doi:10.1016/j.ejor.2004.08.004.
21
Using Subgroup Discovery algorithms in R: The SDR
package
Angel M. Garcia
2016-01-16
Abstract
Subgroup Discovery is a data mining task between classification and description, this task is nowadays a relevant task for researchers due to it success in many fields, especially the medical field. The SDR package provide the posibility of use some of the algorithms that exists in the specialized bibliography by means of using datasets provided in the format specified by data mining tool KEEL without dependencies to other tools/packages in the R console. Also, the package provide a graphical interface to use the algorithms easily and to do basic exploratory analysis with datasets in KEEL format.
1 Introduction
Nowadays, a huge amount of information is generated everyday over the Internet. Such information contains implicit knowledge that is very important to many organizations because they give the possibility, for example, of improve their products or services.
In order to achieve this purpose, traditional techniques of knowledge extraction like OLAP(On-Line Analitical Processing) are not capable to extract knowledge as well as data mining do. Due to the success of data mining at extracting knowledge, a lot of techniques and methods have been designed. Such methods can be classified into two vast fields according to their final objective:
• Predictive data mining: which objective is to find a value of a predefined variable of interest in new instances that arrive at the system.
• Descriptive data mining: which objective is to find relationships between instances that are at the moment in the system.
This classification does not cover all the algorithms that exists in data mining, and subgroup discovery is one of those fields not covered by this classification because it has caracteristics of both fields. Now, R has in CRAN repository a package called rsubgroup [Martin Atzmueller, (2014)], this package is only an interface to use the subgroup discovery algorithms provided in VIKAMINE [M. Atzmueller, F. Lemmerich (2012)] data mining tool, and has a strong dependency with this tool and use the package rJava [Simon Urbanek (2013)] that also provided an interface to use Java files in R. So, this packages has a lot of external dependencies.
Our SDR package implements by now other subgroup discovery algorithms that is not available on VIKAMINE and those algorithms are implemented directly in R, so no dependencies with other tools/packages are necessary to perform a subgroup discovery task and the posibilities of growing the package by means of the adition of new algorithms is easier for every user of R (we have a public GitHub repository where everyone can contribute). Also our package provide a web interface available at http://sdrinterface.shinyapps.io and locally by calling the SDR_GUI() function. Such interface could, in one hand make a subgroup discovery task via web without having R installed in the system or in the other hand, make an easier task by using the visual controls of the web interface.
Further, our package provide a function to read KEEL [J. Alcala-Fdez et al.(2011)] datasets files, a dataset format that is not supported now in R.
2 Subgroup Discovery
Subgroup discovery could be defined as [Wrobel S. (2001)]:
In subgroup discovery, we assume we are given a so-called population of individuals (objects, customer, ...) and a property of those individuals we are interested in. The task of subgroup discovery is then to discover the subgroups of the population that are statistically “most
1
interesting”, i.e. are as large as possible and have the most unusual statistical (distributional) characteristics with respect to the property of interest.
Subgroup discovery tries to find relations between different properties of a set with respect of a target variable, but this relations must be statistically interesting, so it is not necessary to find complete relations but partial relations.
The representation of thes relations is in form of rules, one per relation. This rule can be defined as:
R : Cond Targetvalue (1)
Where Targetvalue is the value for the variable of interest (target variable) for the subgroup discovery task and Cond is normally a conjunction of attribute-value pairs which describe the characteristics of the subgroup induced.
Subgroup discovery is halfway between description and classification because subgroup discovery has a target variable but his objective is not predict but describe. Likewise the use of a target variable is not possible in description, because description find relations among all variables.
A key point to understand what subgroup discovery do is his difference with predictive data mining. Predictive data mining tries to split the entire search space in a normally complex way to find a value for a variable of interest in new incoming instances while subgroup discovery tries to find interesting relations between those instances to find what characteristics of those instances are interesting regarding the variable of interest.
An example could be a group of patients and our variable of interest is if those patients has or has not got heart disease. Predictive data mining objective is to predict if new patients will have a heart disease by their characteristcs while subgroup discovery tries to find what subgroup of patients are more likely to have a heart disease and with those characteristics, make a treatment against those characteristics.
2.1 Main elements of subgroup discovery algorithms
All the algorithms that are included into this data mining task must have the next characteristics:
• Type of target variable: The target variable could be binary (two posible values), categorical ( n posible values) or numerical (a real value within a range).
• Description language: Rules must be as simple as possible. Due to this, rules are composed normally by conjunctions of attribute-value pairs or in disyuntive normal form. Also, to improve the interpretability of the results, fuzzy logic could be used.
• Quality measures: Are very important because they define when a subgroup is "interesting". They will be briefly described below.
• Search strategy: The search space grows exponentially by the number of variables. Using a search strategy that could find a good solution or the optimal one without searching into the whole search space is important.
2.2 Quality measures for subgroup discovery
A lot of quality measures have been defined for this task and there is no consensus for what quality measure it is the best for the task. The most important quality measures for subgroup discovery are described below[F. Herrera et al. (2010)]:
• Nr: The number of rules generated.
• Nv: The average number of variables that the rules generated have.
• Support: It measures the frequency of correctly classified examples covered by the rule:
n(Cond Tv)
Sup(x) = (2)
ns
Where n(Cond Tv) means the number of correctly covered examples and ns is the number of examples in the dataset.
• Coverage: It measures the percentage of examples covered by the rule related to the total number of examples:
Cov (x) = n(Cond) (3)
ns
where n(Cond) means the number of covered examples by the rule.
2
• Confidence: It measure the percentage of examples correctly covered of the total of covered examples:
Conf (x) = n(Cond Targetvalue) (4)
n(Cond)
• Interest: It measures the interest of the rule determined by the antecedent and consequent:
Env i=1 Gain(Ai)
Int(x) = (5)
nv• log2(dom(Gk))
where Gain is the information gain, Ai is the number of values of the variable and dom(Gk) is the cardinality of the target variable.
• Significance: It reflects the novelty in the distribution of the examples covered by the rule regarding the whole dataset:
where p(Cond) = n(Cond)
ns
• Unusualness: It is defined as the weigthed relative accuracy of a rule
n(Cond) (n(Cond Tv)
WRAcc(x) = n(Cond) n(Tv) (7)
ns ns )
where n(Tv) is the number of examples that belong to the target variable.
3 Algorithms of the package
Now, we describe the algorithms that are available in our package. This contains three algo-rithms: SDIGA [M. del Jesus et al. (2007)], MESDIF [M. del Jesus et al. (2007)] and NMEEF-SD [C.J. Carmona et al. (2010)]. In chapter 4 we describe how to use the algorithms.
3.1 SDIGA (Subgroup Discovery Iterative Genetic Algorithm)
The algorithm SDIGA is a subgroup discovery algorithm that extract rules with two possible represen-tations: one with conjunctions of attribute-value pairs (called canonical representation) or one with a disyuntive normal form (DNF) in the antecedent. It follows an iterative schema with a genetic algorithm in his core to extract those rules, this genetic algorithm only extract one rule, the best of the population, and after the extraction a local optimization could be performed in order to improve the generality of the rule. As the algorithm follows an iterative schema, the genetic algorithm is executed one time after another until a stopping criteria is reached: the rule obteined by the genetic algorithm and after the local improvement phase must cover at least one example not covered by precedent rules and this rule must have a minimum confidence (see Equation 4).
SDIGA can work with lost data (represented by the maximum value of the variable + 1), categorical variables and numerical ones using fuzzy logic with the latter to improve the interpretability of the results.
3.1.1 Components of genetic algorithm of SDIGA
As we mentioned above, a genetic algorithm is the core of SDIGA. Such genetic algorithm is the responsible of extract one rule per execution and this rule it is the best of the population at the final of the evolutive process.
3.1.1.1 Representation schema Each chromosome in the population represents a possible rule but it only represents the antecedent part of the rule because the consecuent is prefixed. SDIGA can handle two types of representation as we mentioned above, canonical and DNF.
Canonical representation is formed by a numerical vector of a fixed length equal to the number of variables with possibles values in a range in [0, max] where max is the number of possible values for categorical variables or the number of fuzzy sets defined for numerical variables. This max value represents the no participation of that variable in the rule.
DNF representation is formed by a binary vector, also with fixed length equal to the sum of all number of values/fuzzy sets of the variables. Here, a variable does not participate in a rule when all of his values are equal to zero or one.
3
3.1.1.2 Crossover operator SDIGA follows a pure stationary schema which only crosses the two best chromosomes in an iteration of the evolutionary process. This crossover is performed by a two-point cross operator.
3.1.1.3 Mutation operator Mutation operator is applied over all the population of parents and chromosomes generated by crossover. The mutation probability is applied at every gene. This operator can be applied in two ways (both with the same probability):
• Eliminate the variable: it puts the no participation value in that variable.
• Put a random value: it puts a random value in that variable. The no participation value is also included.
3.1.1.4 Fitness function The function to define the quality of a chromose in SDIGA is a weighted average of three of the quality measures described in section 2.2. So the functions to maximize is:
where:
• Sop(x) is the local support. This support is a modification of support (see Equation 2) and can be crisp or fuzzy:
– Crisp Support:
Ne+(Ri)
Sopn(x) = (9)
NeNC
– Fuzzy Support:
Ek APC(Ek, Ri)
Sopd(x) = (10)
NeNC
where Ne+(Ri) is the number of examples covered by the rule and it is not covered by previous rules. NeNC is the number of examples of the target variable that it is not covered by any rule yet and APC is the Antecedent Part Compatibility equation calculted only with new covered examples (see Equation 12).
• Conf (x) is the confidence defined as in Equation 4 for the crisp case, but for fuzzy case is defined as the ratio of the sum of APC expresion of the correctly covered examples and the sum of APC all examples covered by the rule:
EECC APC(Ek, Ri)
Confd(x) = E (11)
EC APC(Ek, Ri)
where EC C are the correctly covered examples and EC are the covered examples.
• Obj3(x) is other quality measure of section 2.2
• wi is the weight of objective i
As this rules uses fuzzy logic, we need an expresion to determine when an example is covered or not by a rule and also determine the belonging degree of that example to the rule. This function is determined by the expression APC (Antecedent Part Compatibility) and it is calculate by the expresssion:
APC(e, Ri) = T(TC(µ11(e1), ..., µ1 n(ei)), ..., TC(µi1(e1), ..., µi n(ei))) (12)
where:
• ei is the value of the example for the variable i
• µin is the belonging degree to the set n of the variable i
• TC is the fuzzy t-conorm. In this case is the maximum t-conorm.
• T is the fuzzy t-norm. In this case is the minimum t-norm.
µin will be one or zero if the variable is categorical and its value is the same of the rule or not. In case of numerical variable, µin will be calculated following the triangular belonging degree function.
4
3.1.1.5 Replace operator To get next population for the next iteration a replace operator must be performed. This operator sort the population by fitness value and keep only the n best chromosomes, being n the size of the population.
3.1.2 Local optimization
After the genetic algorithm returns a rule, this rule could be improved by means of a local optimization based on a hill climbing first best local search. The algorithm eliminate one by one a variable and if the rule has a support and confidence greater than or equal the actual, the process starts again with that variable eliminated.
3.2 MESDIF (Multiobjetive Evolutionary Subgroup DIscovery Fuzzy rules)
MESDIF is a multi-objective genetic algorithm that extract fuzzy rules. The representation used could be canonical or DNF (see Section 3.1.1.1). This algorithm follows the SPEA2 [E. Zitzler et al. (2001)] approach where an elite population is used along the whole evolutive process. At the end of the process, the rules stored in the elite population where returned as result.
The multi-objective approach is based on a niches schema based on the dominance in the Pareto front. This schema allows to find non-dominated individuals to fill the elite population, that has a fixed size. In Figure 1 we see a basic schema of the algorithm.
Figure 1: MESDIF operations schema
3.2.1 Components of the genetic algorithm.
Now we describe briefly the components of the genetic algorithm to show how it works.
5
3.2.1.1 Initial Population generation The initial population operator performed by MESDIF produce random chromosomes in two ways: one percentage of chromosomes are produced ramdomly and the rest are produced also randomly, but with the difference that only a maximum number of variables could participate in the rule. This produce more generality in the generated rules.
3.2.1.2 Fitness function To obtain the fitness value of each chromosome, MESDIF follows this steps:
• First, we look for dominated and non-dominated individuals in both populations. We call strength of an individual the number of individuals that this domain.
• An initial fitness value Ai is calculated for each individual, such value is the sum of the strength of all dominators of individual i. So, we have to minimize this value and for non-dominated individuals is zero.
• Due to this system could fail if there are a lot of non-dominated individuals, additional information about density is included. This density is computed by the nearest neighbour approach and it is calculate by
1
D(i) = (14) σk + 2
where σk is the k-th nearest neighbour.
• The final adaptation value is the sum of initial fitness and density information
Fitness(i) = D(i) + A(i) (15)
3.2.1.3 Truncation operator As the elite population has a fixed size, we need a truncation operator to truncate the elite population if all non-dominated individuals can not fit in elite population. To make this truncation, the operator take all non-dominated individuals and calculate the distance among every one. Then, the two closest individuals are taken and eliminate the individual with his k-th nearest neighbour with a minor distance. This process is repeated until non-dominated indivuals fit in elite population.
3.2.1.4 Fill operator If the number of non-dominated individuals are less than the size of the elite population, we need to fill elite population with dominated individuals. The operator sort the individuals by its fitness value and copy the n first individuals to elite population, where n is the size of the elite population.
3.2.1.5 Genetic operators The genetic operators of MESDIF are:
• A two-point crossover operator (see Section 3.1.1.2)
• A biased mutation operator, the functionality is the same operator of SDIGA (see Section 3.1.1.3) but it is applied over a population of selected individuals.
• A selection operator based in a binary tournament selection. This selection is only applied over the individuals of elite population.
• A replacement of the selected population based on the direct replace of the k worse individuals of the population. k is the number of individuals returned after crosses and mutations.
3.3 NMEEF-SD (Non-dominated Multi-objective Evolutionary algo-rithm for Extracting Fuzzy rules in Subgroup Discovery)
NMEEF-SD is another multi-objectve genetic algorithm based in the NSGA-II [K. Deb et al. (2000)] approach. This algorithm has a fast sorting algorithm and a reinitialisation based on coverage if the population does not evolve for a period.
This algorithm only works with a canonical representation (see Section 3.1.1.1). In [C. Carmona et al. (2009)] a study is presented where it reflects the low quality of rules obtained with a DNF representation.
6
3.3.1 Evolutionary process
The evolutionary process follows this steps
• An initial biased population Pt is generated (see Section 3.2.1.1).
• The genetic operators are applied over Pt obtaining Qt with the same size as Pt.
• Pt and Qt are joined obtaining Rt and the fast sorting algorithm is applied over Rt. The individuals are sorted forming different fronts in the following way: "The first front (F1) is composed of non-dominated individuals, the second front (F2) is composed by individuales dominated by one individual; the third front (F3) is composed by individuals dominated by two, and so on."
• After that, the algorithm generates the population of the next generation (Pt+1). First, the algorithm checks if the Pareto front covers new examples as it can be show in Figure 2. If this condition is not satisfied during a period of the evolutionary process, a reinitialisation based on coverage is performed. Otherwise, the algorithm gets the next population (Pt+1) introducing, in order, the first complete fronts of Rt. If the last front does not fit completely in Pt+1 then, the front is sorted by the crowding distance and first individuals are copied until Pt+1 is filled.
• At the final of the evolutionary process the individuals in the Pareto front are returned.
Figure 2: Operation schema of NMEEF-SD
3.3.2 Genetic operators
The genetic operators of NMEEF-SD are a two-point crossover operator (see Section 3.1.1.2) and a biased mutation operator (see Section 3.1.1.3).
3.3.3 Reinitialisation operator
This operator is applied if the Pareto front does not cover any new example during a 5% of the total number of evaluations. Then, the algorithm copy the non duplicated individuals in the Pareto front to Pt+1 and the rest of individuals are generated by means of trying to cover one example of the target class with a maximum number of variables.
3.4 FuGePSD
FuGePSD [C. Carmona et al. (2015)] is another genetic algorithm that finds interesting subgroups of a given class. It uses a programming genetic approach, where individuals are represented as trees with variable length instead of vectors. Also, the consecuent of the rule is represented. This schema has the advantage of get rules for all possible values of a given target variable in one execution. Furthermore, FuGePSD has a variable population length, which can change over the evolutive process based on an competitive-cooperative approach, the Token Competition.
7
3.4.1 Evolutionary process
The evolutionary process of FuGePSD works as follow:
1. Create a random population of a fixed length.
2. Create a new population called Offspring Propulation, which is generated via genetic operators.
3. Join original population and offspring and execute the token competition procedure
4. Get global fitnees and replace best population if necessary.
5. return to 2 until number of generations is reached.
6. Apply to the best population a Screening function and return the resulting rules.
The key function on this scheme is the token competition procedure, which promove diversity on the population, and some of the genetic operator will bring specifity to the rules.
3.4.2 Genetic operators
The genetic operators used by FuGePSD are:
• Crossover: it takes to parents and to random subtrees of each one. The crossover will cross the subtrees if the grammar are correct.
• Mutation: Change randomly a variable of an indiviual with a random value.
• Insertion: Inserts a new node on an indivudual, This node is a variable with a random value.
• Dropping: Remove a subtree of an individual.
This genetics operators will be applied with a probability given by the user.
3.4.3 Token Competition procedure
The token competition is the key procedure of FuGePSD, this brings diversity on the population keeping the best individuals. The algorithm works as follows: let a token be an example of the datasets, each individual can catch a token if the rule can cover it. If its occur, a flag is setted at that token and this token cannot be catchet by other rules.
So, the token competition works as follows:
1. Sort the population in descending order by their individual fitness value.
2. In order, each individual takes as much token as it can. This action is storied in a value for each individual called penalized fitness:
PenalizedFitness(Ri) = nusualness(Ri) count(Ri) (16)
ideal(Ri)
Where count(Ri) is the number of tokens the rule really seized and ideal(Ri) is the maximum number of tokens the rule can seize.
3. Remove individuals with PenalizedFitness = 0.
3.4.4 Screening Function
At the end of the evolutive process, the screening function is launched to get the users quality rules only. This rules must reach a minimum level of confidence and sensitivity. The function has an external parameter (ALL_CLASS) which if true, the algorithm will return, at least, one rule per value of the target variable, or at least the best rule of the population if false.
4 Use of SDR package
In this section we are going to explain how to use this package. This package tries to use in a really simple way subgroup discovery algorithms and also without any dependencies.
8
4.1 Installing and load the package.
The package SDR is now available at CRAN servers, so it can be installed as any other package by simply typing:
install.packages("SDR")
Also, the develop version is available into GitHub at http://github.com/aklxao2/SDR, feel free to clone and contribute if you wish. If you wish to use the development version you need to install the package devtools using the commando install_github:
devtools::install_github('aklxao2/SDR')
SDR depends only on the package shiny [Chang(2015)]. This package is neccesary to use the user interface in a local way and if the package is not installed, it asked to the user to be install when running the function SDR_GUI(). Once installed the package has to be loaded before use it. The package can be loaded through library() or require(). After loading the package there are six datasets available: carTra, carTst, germanTra, germanTst, habermanTra and habermanTst that corresponds to car, german and haberman training and test datasets.
This package use an internal representation assigned to the "keel" class . All the information about the datasets are available in this class.
4.2 Load a KEEL dataset
To use the algorithms available in this package, we could load a dataset if it is different of the three examples availables. Assuming the files 'irisTra.dat' and 'irisTst.dat', corresponding to the classical iris dataset, the load of this files will be as follows:
irisTraining <- read.keel("irisTra.dat")
irisTest <- read.keel("irisTst.dat")
As we mentioned above, the algorithms in the package uses fuzzy logic, and the definitions of the fuzzy sets are implicit to every dataset. This fuzzy sets definitions are defined with the same length and the same type (all are triangular fuzzy sets). By default, the function read.keel() creates three fuzzy sets for every variable. To change the number of sets per variable, you can use the argument 'nLabels':
irisTraining <- read.keel("irisTra.dat", nLabels = 5)
irisTest <- read.keel("irisTst.dat", nLabels = 5)
If you want to get more KEEL datasets format, please visit: http://sci2s.ugr.es/keel/datasets.php This function can read also ARFF the same way a KEEL file.
4.3 Creating a KEEL object from a data.frame
The other way to load a dataset on SDR is by a data.frame. This is possible by using the function keelFromDataFrame():
irisTraining <- keelFromDataFrame(data = iris, relation = "iris",)
Also we can specify the number of fuzzy labels to by used, the names of the variables if it is not on the columns of the data.frame, the type of variables that has the dataset (‘c’ for categorical, ‘r’ for real and ‘e’ for integer) and a vector indicating the names of the target class
irisTraining <- keelFromDataFrame(data = iris, relation = "iris", nLabels = 5, names = c("Sepal.Length, Sepal
4.4 Executing Subgroup Discovery algorithms
Once our datasets are ready to be used, it is time to execute one subgroup discovery algorithm. For example we want to execute the algorithm MESDIF. For the rest of the algorithm this steps are equal and only a few parameters are diferent, if you need help with the parameters, refer to the help of the function using help(function) or ?function.
The subgroup discovery algorithms have two ways of execution: one of them is by indicating the path of a
9
parameters file, with all the neccesary parameters. The other is by putting in the function all parameters one by one. The first mode of use is indicated when we have a dataset prepared for executing directly because this way does not allow us to modify the properties of the dataset, so the second way is indicated when we load and/or modify a dataset before execute the algorithm.
When we execute the algorithm, the results are shown in the console. This results are divided in three fields:
• First, an echo of the parameters used in the execution are shown.
• Second, the subgroups or rules genereted by the algorithm.
• Finally, some quality measures about the rules generated applied to test data are shown. The final value are a summary of this quality measures for all rules
As the results line could be extremly large, the algorithms also save this results into three files, divided in the same categories described above.
MESDIF(paramFile = "MESDIFparameters.txt")
library("SDR")
MESDIF( paramFile = NULL,
training = habermanTra,
test = habermanTst,
output = c("optionsFile.txt", "rulesFile.txt", "testQM.txt"),
seed = 0,
nLabels = 3,
nEval = 300,
popLength = 100,
eliteLength = 3,
crossProb = 0.6,
mutProb = 0.01,
RulesRep = "can",
Obj1 = "CSUP",
Obj2 = "CCNF",
Obj3 = "null",
Obj4 = "null",
targetClass = "positive"
)
##
## Algorithm: MESDIF
## Relation: haberman
## Training dataset: training
## Test dataset: test
## Rules Representation: CAN
## Number of evaluations: 300
## Number of fuzzy partitions: 3
## Population Length: 100
## Elite Population Length: 3
## Crossover Probability: 0.6
## Mutation Probability: 0.01
## Obj1: CSUP (Weigth: )
## Obj2: CCNF (Weigth: )
## Obj3: null (Weigth: )
## Obj4: null
## Number of examples in training: 244
## Number of examples in test: 62
## Target Variable: Survival
## Target Class: positive
##
##
##
10
## Searching rules for only one value of the target class...
##
##
## GENERATED RULE 1
## Variable Age = Label 2 ( 56.5 , 83 ,109.5 )
## Variable Year = Label 0 ( 52.5 , 58 ,63.5 )
## Variable Positive = Label 0 ( -26 ,0 ,26 )
##
## THEN positive
##
##
##
## GENERATED RULE 2
## Variable Positive = Label 0 ( -26 ,0 ,26 )
## THEN positive
##
##
##
## GENERATED RULE 3
## Variable Year = Label 0 ( 52.5 , 58 , 63.5 )
## Variable Positive = Label 2 ( 26 ,52 , 78 )
##
## THEN positive
##
##
##
##
##
## Testing rules...
##
##
## Rule 1 :
## - N_vars: 4
## - Coverage: 0
## - Significance: 0
## - Unusualness: 0
## - Accuracy: 0.5
## - CSupport: 0
## - FSupport: 0.002481
## - CConfidence: 0
## - FConfidence: 0.070144
##
##
## Rule 2 :
## - N_vars: 2
## - Coverage: 0.903226
## - Significance: 0.223676
## - Unusualness: -0.023413
## - Accuracy: 0.241379
## - CSupport: 0.209677
## - FSupport: 0.186104
## - CConfidence: 0.232143
## - FConfidence: 0.218818
##
##
## Rule 3 :
## - N_vars: 3
## - Coverage: 0
## - Significance: 0
11
## - Unusualness: 0
## - Accuracy: 0.5
## - CSupport: 0
## - FSupport: 0.004399
## - CConfidence: 0
## - FConfidence: 0.639344
##
##
## Global:
## - N_rules: 3
## - N_vars: 3
## - Coverage: 0.301075
## - Significance: 0.074559
## - Unusualness: -0.007804
## - Accuracy: 0.413793
## - CSupport: 0.209677
## - FSupport: 0.064328
## - FConfidence: 0.309435
## - CConfidence: 0.077381
5 The user interface
As we mentioned at the begin of this paper, the SDR package provide to the user an interface to use the subgroup discovery task in a more easy way and also do some basic exploratory analysis tasks.
We can use the user interface by two ways: one of them is using the interface via web at: http: //sdrinterface.shinyapps.io/shiny . This has the advantage of use the algorithm without having R installed in our system and also avoid expending process time in our machine. The other way is to use the interface is in a local way, having our local server. This could be possible simply using:
SDR_GUI()
This function launch the user interface in our predetermined web browser. As we can see in Figure 3. The page are organized into an options panel at the left and a tabbed panel at the right. Here is where our results are shown when we execute the algorithms.
The first we have to do is to load a KEEL dataset. If we want to execute a subgroup discovery algorithm we must load a training and a test file having the same '@relation' field in the KEEL dataset file.
Once we select the dataset, automatically it shows a graph with information about the last variable defined in the dataset. The graph shows the distribution of examples having some values of the variable. At the rigth of this graphic we can see a table with some basic information, more extended if this variable is numerical.
12
Figure 3: Initial screen of the SDR user interface.
When loaded a dataset we can do a lot of things, in Figure 4 we can see all the possibilities we can do:
1. As we mentioned above, we can load a second dataset as a test (or training) file.
2. This lists have a double function. In one hand we could select the variable to visualize in the graph and in the other hand is to select the target variable for executing a subgroup discovery algorithm. Below the variable selection we can choose a target value of the variable to find subgroups about this value or search for all posible values of the target variable.
3. Here we can choose how the information will be visualized, for categorical variables we can choose show the information as a pie chart or as a historgram. For numerical variables, a histogram and a bloxpot are available.
4. Here we can choose the subgroup discovery algorithm and his parameters, that it is shown below. Below the parameters, we have the button to execute the subgroup discovery algorithm.
5. This section allows the selection, for categorical variables, the values of that variable has that we can see in the graph. It is important to remark that it only "hide" the values in the graph, so it does not eliminate any example of the dataset.
6. It allows to visualize information about the training or test file.
After the execution of a subgroup discovery algorithm, we go automatically to the tab 'Rules generated', this tab contains a table with all subgroups generated. If we want we could filter rules by variable, for example, typing into the 'Search' field.
The tab 'Execution Info' shows an echo of the parameters used for launch the algorithm. This echo is equal than the one we can see in R console.
The tab 'Test Quality Measures' shows a table with quality measures of every rule applied to test dataset. The last row its a summary of results and shows the number of rules we have and the average results of every quality measure.
6 Summary
In this paper the SDR package has been introduced. This package can use four subgroup discovery algorithms without any other dependencies to others tools/packages. Also, the posibility of read and load datasets in the KEEL dataset format is provided, but it can read dataset from ARFF format or load a
13
Figure 4: Screen of the user interface once loaded a dataset.
data.frame object. Finally, a web-based interface is developed for make the work easier, even if we do not have R installed in our system.
The devolpment of the package will continue in the future, including more functionality to work with KEEL datasets, adding new subgroup discovery algorithms and also improve the web interface. As we can see we have a great job ahead, so we encourage other developers to participe adding tools or algorithms into the package, as we will do in futures releases.
References
[Wrobel S. (2001)] Inductive logic programming for knowledge discovery in databases. Springer, chap Relational Data Mining, pp 74-101.
[Martin Atzmueller, (2014)] rsubgroup: Subgroup Discovery and Analytics. URL http://CRAN.R-pro ject. org/package=rsubgroup
[Simon Urbanek (2013)] rJava: Low-level R to Java interface. URL http://CRAN.R-project.org/ package=rJava
[Chang(2015)] W. Chang. shiny: Web Application Framework for R, 2015. URL http://CRAN.R-pro ject. org/package=shiny. R package version 0.11.
[M. Atzmueller, F. Lemmerich (2012)] VIKAMINE - Open-Source Subgroup Discovery, Pattern Mining and Analytics in Machine Learning and Knowledge Discover in Databases pp. 842-845
[J. Alcala-Fdez et al.(2011)] J. Alcala-Fdez, A. Fernandez, J. Luengo, J. Derrac, S. Garcia, L. Sanchez, F. Herrera. KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework. In Journal of Multiple-Valued Logic and Soft Computing 17:2-3 (2011) 255-287.
[F. Herrera et al. (2010)] F. Herrera, M. J. del Jesus, P. Gonzalez, C. J. Carmona. An overview on subgroup discovery: foundations and applications. In Knowledge and Information Systems. December 2011, Volume 29, Issue 3, pp 495-525.
[M. del Jesus et al. (2007)] M. del Jesus, P. Gonzalez, F. Herrera, M. Mesonero. Evolutionary fuzzy rule induction process for subgroup discovery: a case study in marketing. In IEEE Trans Fuzzy Syst. pp. 15(4):578-592, 2007.
14
[C.J. Carmona et al. (2010)] C. J. Carmona, P. Gonzalez, M. J. del Jesus, F. Herrera. NMEEF-SD: Non-dominated Multi-objective Evolutionary algorithm for Extracting Fuzzy rules in Subgroup Discovery. In Fuzzy Systems, IEEE Transactions. Volume:18, Issue:5, pp.958-970, 2010
[M. del Jesus et al. (2007)] M. del Jesus, P. Gonzalez, F. Herrera. Multiobjective genetic algorithm for extracting subgroup discovery fuzzy rules. In Proceedings of the IEEE symposium on computational intelligence in multi-criteria decision making. pp. 50-57, 2007
[E. Zitzler et al. (2001)] E. Zitzler, M. Laumanns, L. Thiele. SPEA2: Improving the Strength Pareto Evolutionary Algorithm. 2001
[K. Deb et al. (2000)] K. Deb, S. Agrawal, A. Pratap, T. Mayerivan. A Fast Elitist Non-Dominated Sorting Genetic Algorithm for Multi-Objective Optimization: NSGA-II 2000
[C. Carmona et al. (2009)] C. Carmona, P. Gonzalez , M. del Jesus, F. Herrera. An analysis of evolu¬tionary algorithms with different types of fuzzy rules in subgroup discovery. In IEEE International Conference on Fuzzy Systems (FUZZ-IEEE). ICC Jeju, Jeju Island, Korea, 2009
[C. Carmona et al. (2015)] C. Carmona, V. Ruiz-Rodado, M. del Jesus, A. Weber, M. Grootveld, P. Gonzalez, D. Elizondo A fuzzy genetic programming-based algorithm for subgroup discovery and the application to one problem of pathogenesis of acute sore throat conditions in humans. In Information Sciences Volume:298, pp. 180-197, 2015
15
STRATEGIC PLANNING
Preparing and Leading
the Planning Process
by Verne Harnish
and the team at Gazelles
EXECUTIVE SUMMARY: The key to success in most endeavors requires the right preparation, exe¬cution, and follow-through. In this bonus chapter accompanying Scaling Up (Rockefeller Habits 2.0), we share a dozen key steps to prepare the team for a quarterly or annual planning session; suggested agendas; and recommended follow-through after the offsite meetings. And at the back of this chapter is a sample completed One-Page Strategic Plan.
Preparation:
1. Set Dates: It’s advisable to set the dates for the quarterly and annual planning retreats (some people call them “advances”) well in advance. And it’s best if a specific rhythm is established (i.e., the second Friday and Saturday before the end of the quarter). The annual planning session is normally two to three days and the quarterly sessions one to two days. Specific agendas are detailed below.
2. Scan Scaling Up: Have the executive team scan Scaling Up (Rockefeller Habits 2.0), especially the three chapters in the Strategy section and The Priority chapter in the Execution section. The book is available on Amazon.com or you can save some money and order in bulk (20 to a box) at www.scalingup.com.
3. Complete 4D Assessment (optional): Have the executive team take 5 to 10 minutes to complete the 4D assessment to see which of the Four Decisions — People, Strategy, Exe¬cution, or Cash — needs the most attention in the upcoming planning session. Click here to start your assessment now.
4. Read Collins’ Articles: Read (and re-read) Jim Collins’ Harvard Business Review article titled “Building a Company Vision.” (Download for $6 at www.hbr.com). Do this in the first few annual planning sessions until you’re comfortable with your Core Values, Purpose, Prof-it/X, and BHAG — key elements of the first two columns of the One-Page Strategic Plan (OPSP). Also go to www.jimcollins.com where Collins has several free interactive tutorials to help discover Core Values, discern a Purpose, choose a BHAG, etc.
Gazelles copyright 2015 1
Scaling Up
5. Strategic Thinking “Council”: Form the council as discussed in Scaling Up and start meeting weekly to get some critical talk time around the strategic decisions driven by the Strengths, Weaknesses, Trends (SWT) and 7 Strata worksheets. Also discuss the 4Ps of mar-keting: Product, Price, Place, and Promotion. In most firms, marketing strategy = strategy. Search the internet for Ogilvy’s 4Es of marketing and add those to the ongoing discussions/ debates. Even if it’s just for a few weeks prior to the planning session, these weekly discus-sions will get the strategy juices flowing.
6. Employee Survey: A few weeks prior to the planning offsite, conduct an employee survey. Employees’ insights are helpful in determining quarterly or annual priorities since they are closer to the customers and are immersed in the daily processes of the business. Many firms use an online survey tool such as SurveyMonkey to make it easier to administer. We suggest three simple questions:
a. What should (enter company name) start doing?
b. What should (enter company name) stop doing?
c. What should (enter company name) keep doing?
7. Customer Input: Along with employee feedback, formally gather customer input. At a min-imum, ask them the same three “start, stop, and keep” questions. As discussed in The Data chapter in Scaling Up, it’s easier to pick up patterns and trends if there is a weekly rhythm of gathering input from customers and employees, but this simple three-question survey will get you started if you are new to the process.
8. Top Three Issues: Send out an email to those attending the planning session to ask them to send back the top three issues they feel MUST be addressed/explored/answered at the upcoming planning session for them to feel it was a success.” Compile these for review at the beginning of the planning session or just prior.
9. (Optional) Strengths, Weaknesses, Opportunities, and Threats (SWOT): If you want to dig deeper than what the “top three issues” question uncovers, lead a separate SWOT exercise with the broader management team prior to the planning session. Or simply send out an email to your team seeking their input on the SWOT and compile the results for the planning session.
10. (Advanced) SWT: As part of the activities leading up to the planning session, have the senior team complete the SWT worksheet as outlined in Scaling Up. Helpful resources in identifying important trends are Frost & Sullivan’s annual trends report and Peter Diamandis’s Abundance 360 annual event and quarterly updates. You can contact Gazelles about partici¬pating in Abundance 360.
11. (Optional) One-Page Personal Plan (OPPP): Encourage all team members to update their OPPPs. It’s best if one’s personal and professional goals are aligned.
2 Gazelles copyright 2015
Strategic Planning
12. Get Someone to Facilitate: Find someone outside the company to facilitate your planning sessions. Ask a colleague from another firm or bring in a professional facilitator like those we have at Gazelles. This allows everyone on the team, including the CEO, to actively participate rather than worry about facilitation. And trained facilitators will know how to discern Core Values, define a powerful Purpose, discover a key strategy, and help the team uncover the underlying constraints in establishing priorities and setting key performance indicators (KPIs).
OPSP Form Preparation:
1. Replace Logo: Feel free to take all references to Gazelles off the OPSP document, except the copyright, and replace them with your own logo and company information. There are versions of all the Growth Tools without the Gazelles logo you can download at www.scalingup.com.
2. Replace Headings: Call them Core Values, Core Purpose, Brand Promise, Rocks, etc. — OR NOT!!! Feel free to come up with your own unique language within the firm. HP calls its Core Values The HP Way. Some firms prefer the term principles or guidelines instead of Core Values. Similarly, some firms find the term Rocks as a label for quarterly priorities to be awkward. Again, it’s up to you. The document is meant to serve as a guideline.
3. Use Blank Documents: There’s a tendency to provide team members with completed or semi-completed one-page plans during planning sessions (i.e., with Core Values, Purpose, etc. already filled in). We highly suggest you pass out blank OPSPs at the beginning of the session and have everyone fill them in by hand. There’s something about re-writing the Core Values, Purpose, BHAG, etc. each quarter that helps hardwire them into the brain and better connects each person to what is said and decided. Besides, there’s not that much to write!
4. Project up on a Screen: To facilitate the process, project the OPSP on a large screen. Designate someone to fill it in electronically so it can be immediately emailed to all of the participants after the planning session. Doing this also helps people stay focused and makes it easier for everyone who is filling in documents by hand simultaneously.
5. SaaS Offering (optional): Gazelles has partnered with Aligntoday.com to bring you all the Gazelles Growth Tools ‘in the cloud’ to give you flexibility, visibility and accountability during and after your planning session. In preparation for your planning session setup a trial account — go to www.alignwithgazelles.com and click the ‘trial’ button. Once you have your account setup use the ‘One Page Strategic Plan WIZARD’ to load your plan contents as the team makes decisions. This will give you the ability to display the OPSP up on the screen during planning and make real time adjustments as you go, with everything safely stored and accessible in the cloud. We suggest you start by giving access to your leadership team to track KPIs and Priorities and then cascade down thru the organization as you expand implementation of the Rockefeller Habits (it can manage the entire process for you). The monthly fee is just $5 per user. Need help, email info@aligntoday.com.
Gazelles copyright 2015 3
Scaling Up
AGENDA (OVERVIEW):
Quarterly vs. Annual (timing and agendas): The annual planning session is typically two to three days; the quarterly (or trimester) planning sessions are one to two days.
Executive Team Dinner: We recommend the senior team have dinner together and then meet afterwards for a couple hours the night before the start of the planning session. Dinner allows time for catching up, and the two-hour session following dinner gives the team a chance to focus on reviewing the SWT and share stories from the last quarter where the company “lived” its Core Values and Purpose. It’s also a good time to do some blue sky thinking about the company with questions like “If our team were to launch a new business, what would we do instead of this business?” and “How would we put our own company out of business?” It also gives the team an extra night to sleep on the conversations generated that evening.
The first third of each planning session (day one of the annual planning session; three hours of a quarterly session) is spent reviewing the SWOT (optional), and the first three columns of the OPSP. Also update the Functional Accountability Chart (FACe) tool and focus on the one func¬tional position that needs support.
The next third (day two of the annual planning session; afternoon of the quarterly session) is spent on the 1 year column of the OPSP, including a review of the company’s financials and using the CASh and Power of One worksheets to focus on ways to improve cash flow.
The final third (day three of the annual session; day two of the quarterly) is focused on completing the entire right hand page of the OPSP. Review the Process Accountability Chart (PACe) tool and choose one process to design or redesign that supports achieving the Critical Number (#1 Priority). Last, review the Rockefeller Habits Checklist and choose one or two of the 10 habits to execute (or execute better) the next quarter.
WARNING: “When I go slow, I go fast” notes the Chinese proverb. There is a tendency in planning sessions to rush through or ignore the Strengths, Weaknesses, and Trends along the bottom of the OPSP and the first two or three columns of the OPSP (Core Values, Purpose, BHAG,
Sandbox, and Brand Promises), especially after team members feel like they’ve nailed down the decisions
in previous sessions. However, spending sufficient time reviewing and updating the SWT and first three columns almost always makes the decisions in the Annual and Quarterly columns come more quickly and effortlessly. Trust us on this!
4 Gazelles copyright 2015
Strategic Planning
Sample Quarterly Agenda:
Day 1:
17:30 - 18:00 — Reception/arrival
18:00 - 20:00 — Dinner (Snacks in Europe)
20:00 - 22:00 — Opening Session — Review core values and purpose stories, SWT, and host open discussion about the future (continue into the night!).
22:00 - ? — Dinner in Europe
Day 2:
8:30 - 10:00 — Opening Remarks by CEO, Good News Stor ies/O PPPs, and Top 3 Issues (what will make this a successful meeting for everyone)
10:30 - 12:00 — SWOT, FACe, and review first three columns of OPSP
13:30 - 15:00 — Review Annual column of OPSP
15:30 - 17:00 — CASh and Power of One
18:00 - ? — Dinner, finish up what didn’t get completed earlier (further work on 7 Strata)
Day 3: (invite middle management)
8:30 - 10:00 — Review previous day, and complete Quarterly column OPSP
10:30 - Noon — PACe (review key process supporting Quarterly Critical Number),
13:30 - 15:00 — Establish column 6 Quarterly Theme (leave for middle management to design/ drive) and review Rockefeller Habits Checklist. Choose one or two areas for improvement.
15:30 - 17:00 — Everyone updates their own column 7 — individual KPIs, Critical Number, and Priorities. Then go around the room and have everyone share their Critical Number (top 1 priority for the quarter).
AGENDA (DETAILS):
Opening Remarks by CEO: Reflect on the past quarter/year and then set the stage for the major conflict that will be resolved this planning session.
Gazelles copyright 2015 5
Scaling Up
Details: Great meetings are structured like great movies according to Pat Lencioni in his latest book Death by Meeting. At the heart of all movies is a “conflict, then resolution” structure. Rath¬er than open with something like “I’m glad all of you can be here to participate in this planning session (yawn),” instead set the stage with an opening line like “We face stiff competition from XYZ, the marketplace for our services is heating back-up, and we’re being hindered by our ... so this next two days are critical in figuring how we address these challenges and maximize our opportunities...” Or opt for something like “we’ve been offered the greatest opportunity to gain market share in five years. It is for us to figure out how to make it happen...” or “This is the year we must make the kinds of profits we expect from a great company.” Pick up your hints from the preparation work you’ve done — the employee survey, the customer feedback, top three issues lists, and the SWT/SWOT analysis.
Good News Stories and OPPP: Share a round of good news stories. Sometimes this precedes the opening remarks by the CEO or occurs the night before if you host the optional evening session. It is your decision. (Optional) Go around a second time and share a couple highlights from each executive’s OPPP — a key relationship, achievement, and ritual for the coming quarter or year.
Details: Following the opening remarks, you want to set a positive tone, loosen everyone up, and help the team connect as people by taking 10 to 20 minutes to have everyone share good news personally and professionally from the previous week and a reflection on the previous quarter or year. Keeping it current helps make it relevant and fresh. The professional good news allows the team to count its blessings and the personal good news always brings a laugh or two — a powerful way to de-stress, slow the brain down to the alpha state (7 to 14 brainwave cycles per second), and help keep even the most dreaded issues in perspective. Also use it as an early gauge if someone is particularly stressed or disturbed coming into the meeting.
(Optional) Go around a second time — and make it second round, not one combined with the good news round. Share a few key decisions from each executive’s OPPP for the coming quarter or year. This awareness will prove helpful in setting the company goals and strengthen bonds between the team members. Maintaining a healthy team dynamic (and handling conflict) starts with being vulnerable with each other according to Pat Lencioni’s book Five Dysfunctions of a Team.
New Team Members: Pat Lencioni suggests all teams complete a personality test (Myers Briggs or equivalent) and review the results. This helps them understand and appreciate each other’s differences (and generates a laugh of two). He also suggests reviewing each other’s lifeline: the five high points and five low points in their life that have shaped who they are. This is something members of Young Presidents’ Organization and Entrepreneurs’ Organization do to form a healthy forum. Here’s a link to learn more about drawing lifelines, a powerful exercise for bonding teams when you share your lifeline with each other.
NOTE: When a new executive is added to the team, the lifeline exercise should be repeated. Adding someone to a team makes it a new team. It’s not the old team plus one.
6 Gazelles copyright 2015
Strategic Planning
Start Filling in One-Page Tools: The rest of the planning session is using the Growth Tools to drive the right questions and discussions. Go back to Scaling Up (Rockefeller Habits 2.0) Strategy section for specific instructions in completing the OPSP. Be sure to also review the other one-page tools according to the suggested agenda overview above.
WARNING: It’s your call, but we would resist jumping in and reviewing the past quarter or year (columns 4 and 5 of the OPSP) in detail at the beginning of the meeting (beyond the brief opening remarks of the CEO as outlined above). Once you open that Pandora’s Box, it’s hard to get it shut. Teams tend to get sucked right into the minutia, getting caught up in the details and making it difficult to step back from the trees and talk more strategic about the direction of the firm (could we have incor-porated more clichés in one sentence!!). We suggest you start more broadly reviewing the SWT and the first three columns of the OPSP — after all, it’s a strategic planning session, not a weekly executive team meeting or monthly review session. And if those weekly and monthly meetings have been effective, the quarter has been covered and everyone should be well briefed on the current state of the company.
Quarterly/Annual Employee (Town Hall) Meeting
Gather all the employees (or travel around to various offices) and share the results from the last quarter and the theme/priority for the next quarter. This meeting is usually 30 — 45 minutes long.
The first half of the meeting is a review/celebration of the previous quarter. The key is to get the employees talking and sharing victory stories from the previous quarter. The CEO starts this dialogue by saying something like “Welcome to the quarterly meeting. We said we were going to do X, and we achieved Y — congratulations.” Now, rather than drone on about how everyone pulled together and worked hard (as if the CEO has a clue), the CEO should ask the most pow¬erful question you can ask anyone (team or child) after they’ve accomplished something — “how did you do it?”
The leader can seed the conversation by saying something like “Lisa, you were telling me how your team pulled an all-nighter to win that Acme deal. Please share that with everyone.” Obviously, choose someone you know likes to talk/share. This will then get the conversation started. Lisa will likely say, “yes, we pulled the all-nighter, but we couldn’t have done it without Sam’s team...” and now Sam is invited to share. The key if for people to relive what it took to accomplish the goals.
In turn, if you failed to reach the goal (discussed in Scaling Up) and the team’s trust levels are good, it’s worthwhile discussing as well.
After this past quarter review, it’s time to introduce the theme for the next quarter. Turn down the lights, fire up the music, and make a grand transition. Doug Greenlaw at VTC came running in wearing a red jumpsuit to the music from the movie “6 Million Dollar Man” (add $6 million to the sales pipeline in the next quarter). Appletree Answers typically opened with a video — this one introducing their Idea Flash initiative mentioned in Scaling Up.
Gazelles copyright 2015 7
Scaling Up
Let each executive explain how their function will support the Theme/Critical Number and the “rocks” which will need to be moved to achieve the goal and quarterly outcomes (Revenues, GM, Profit, etc.). Also discuss the one process that will be designed or redesigned to support achieving the Critical Number (Appletree built an app inside Salesforce.com to support Idea Flash). Then handout a copy of the Vision Summary to all employees and give everyone a few minutes, in the meeting, to begin filling in their individual goals at the bottom of the Vision Summary. Everyone’s immediate supervisor can follow-up after the meeting with their teams to coach them if they need help determining appropriate KPIs and priorities that align with the company’s vision.
Daily/Weekly Updates:
After the quarterly meeting, make sure scoreboards are posted that track the daily/weekly progress on the Theme/Critical Number. Many CEOs, like Larry Page at Google, keep employees updat¬ed weekly on progress either through email or an all-employee meeting. Page calls them TGIF meetings (used to be on Fridays, now on Thursdays).
For those using a SaaS offering like Align, everyone can access updated results via their mobile or tablet devices and viewed on large flat screens scattered throughout the office. Here’s a typical screenshot from Align:
Hope you find this helpful. If we can be of assistance or you would like someone to facilitate these meetings email jean@gazelles.com
Sample One-Page Strategic Plan
On the next two pages is a sample completed One-Page Strategic Plan for a fictitious company called TestCo (we know you’re not exactly like them!).
BEST OF LUCK with your strategic planning process.
8 Gazelles copyright 2015
Strategy: One-Page Strategic Plan (OPSP) Organization Name: Testco
People (Reputation Drivers)
Employees Customers Shareholders
Appreciation every 7 days KPI = 100% aDelivery - KPI = Daily report 10 minutes
1. 1. Cash increase daily - KPI = % of increase
Employee Net Promoter Score KPI = 60+ 2. Client 'contact'- KPI = 3 Contacts per CliRevenue Increase - KPI = 20%
2.
Voluntary A-Player Retention - KPI = 95NPS - KPI = Net Promoter Score at 8.5
3. 3. Gross Margin - KPI = 55%
COREVALUES/BELIEFS
(Should/Shouldn't)
We live to hear the word "speed"
We never say no!
We always give options
We cultivate trust with clients
PURPOSE
(Why)
EASY! - We make using data easy so that it helps rather than hurts people!
Profit per X
Profit Per Installation
2014-$125,000
2015-$155,000
2016 - $200,000
BHAG®
'GLOBAL #1' ~ The #1 Global Data Analytics Solution. 1,000 installations within the Fortune 2,500 - globally!
TARGETS (3–5 YRS.)
(Where)
Future Date 31 Dec 2018
Revenues $14,250,000
Profit $2,850,000
Mkt Cap/Cash $1,425,000
Sandbox
$500M+ Corporations - US, Canada, Europe
Key Thrusts/Capabilities
1 UK - Germany - France launch and profitable
2
Database developed for automated marketing
3
Zero human touch in
request fullfillment
4
Named the Best Place to
Work Award
5
Brand Promise KPI’s
10 days or less - installation NPS scoring 60+
7 business days to measurable results
Brand Promises
Speed of installation
Easy to do business with
Results in a week
GOALS (1 YR.)
(What)
Strengths/Core Competencies Weaknesses:
Arrogance -- we're good and we know it
Sales Capabilities -- lacking, so better be the best
1
Automate proposal process
2
Dashboard/KPI metrics
updated and reviewed
kl
3
Hiring/onboarding plan and
process in place- filled
4
Ensure every TestCo team
member'understands' and
''
5
BHAG is a Registered Trademark of Jim Colins and Jerry Porras.
Your Name: Jack "Bullseye" Harris Date: 01 Jan 2015
Process (Productivity Drivers)
Make/Buy Sell Recordkeeping
ACTIONS (QTR)
(How) THEME
(QTR/ANNUAL) YOUR ACCOUNTABILITY
(Who/When)
Your KPIs Goal
Qtr # 1 ending 31 March 2015 Deadline: 3/31/2015
1 Proposals Closed 50
Revenues $2,185,000 Measurable Target/Critical #
Profit $37,000 1,200 face to face meetings with decision makers, clients
and partners
$1,201,750
Mkt Cap
2 Prospects Contacted 250
Gross Margin
Theme Name
Race to 1,200
Scoreboard Design
Describe and/or sketch your design in this space
Cash $145,000
A/R Days 35
3 Face to Face Meetings 125
~nv. Days 18
Rev./Emp $225,000
Your Quarterly Priorities Due
Rocks Who
1 Hire 3 Sales Associates 3/31/15
1 Training Program -created -implemented - every team member Lisa
2 Increase Google Presence 3/31/15
2 Complete Business Plan John
3 1200 Client Meetings with Whole Team 3/31/15
3 Increase Google Presence Jane
4 Get 10 Stories from Clients on Core Values 3/31/15
4 Reduce A/R Time - to 60 days less than 10% Tom
5
5 1,200 client meetings with whole team Angel
Critical #: People or B/S
22 Service Contracts 20 Service Contracts Between green & red 18 Service Contracts Celebration
A BIG dinner where we will bring our families - loved ones - together to celebrate the accomplishment Critical #: People or B/S
1500 Client Meetings
1200 Client Meetings
Between green & red
900 Client Meetings
Critical #: Process or P/L
89% Utilization Rate 85% Utilization Rate Between green & red 75% Utilization Rate Reward
$12,000 donated to the groups top 12 favorite charities - $1,000 per charity! $10 for every contact made! Critical #: Process or P/L
150 Referrals
100 Referrals
Between green & red
75 Referrals
Artificial intelligence and how it will evolve
The speed at which technology is evolving
Generational views on the use of our solutions
Communication mediums shifting -- social Information flows (speed and type) changes The value being placed of data as a tool
BHAG is a Registered Trademark of Jim Colins and Jerry Porras.
–
Date: 20.01.2019
In addition to part-I (General Handout for all courses appended to the timetable) this portion gives further specific details regarding the course.
EEE F435
DIGITAL IMAGE PROCESSING
DR. JAGADISH NAYAK
DR. JAGADISH NAYAK
:
The course introduces the students to the fundamentals of digital images and various processing techniques that are applied to them so as to improve their quality. These techniques are image enhancement, image restoration and image compression. It also briefly introduces automatic image classification and recognition.
(if any) Given in the Bulletin 2017-2018
[T1]:Gonzalez, R. C. & R. E. Woods, Digital Image Processing, Pearson Education , 3rd edition. 2008.
[R1]: Anil K Jain, Fundamentals of Digital Image Processing, Prentice –Hall Inc, 1989, Reprint 2004
[R2]: S Jayaraman , E Essakkirajan, T Veerakumar, Digital Image Processing , Tata McGraw Hill New Delhi
1-3 To introduce fundamental concepts and terms associated with digital images. A simple image formation model, image sampling,
quantization and interpolation, 2D signals and
systems. 2.3.4 -2.4.4
4 To introduce the
concept of image
enhancement Spatial Domain enhancement techniques 3.1
5-6 To study image enhancement by gray level transformations Some basic gray level transformations: image negatives; log, power-law and piecewise linear 3.2.1-3.2.4
7-8 To study Histogram processing of an image Histogram processing: equalization matching,
local enhancement 3.3-3.3.3
9-10 To study Histogram processing of an image Histogram statistics; arithmetic/logic operations 3.3.4
11-12 To learn image enhancement by filtering in the spatial domain Spatial filtering: smoothing and sharpening 3.4-3.6.4
13-15 Image Transforms DFT, DCT, 4.2-4.3
16-17 Image Transforms Walsh-Hadamard Transform 4.4-4.5
18-19 Image Transforms K-L Transform, Sine transform 4.6
20-22 To learn image
enhancement by
filtering in the
frequency domain Filtering in the frequency domain 4.7
23-24 Frequency domain filtering Smoothing, sharpening and selective filtering 4.8-4.10
25 Frequency domain filtering Filter banks and wavelets 4.11
26-27 Image degradation Image degradation model, 5.1-5.2
27-28 Degradation Estimating the degradation 5.6
29 To learn inverse
filtering Inverse filtering 5.7
30-31 To introduce the fundamentals of image compression Fundamentals of image compression 8.1
32-34 Basic Compression methods Huffman, arithmetic and LZW coding. 8.2.1-8.2.4
35-36 Basic Compression methods Run-Length, symbol based, Bit plane and
predictive coding JPEG 8.2.5-8.2.9
37-38 A brief introduction to segmentation techniques Image segmentation 10
39-40 To study Image
reconstruction from the projections Image reconstruction from projections, Principle of computer tomography (CT), Projection and Radon
Transform, The Fourier slice theorem,
Reconstruction Using Parallel-Beam Filtered
Backprojections.. 5.11
41-42 To understand
automatic image
recognition Object recognition, Pattern and Pattern classes, Image recognition based on Decision-Theoretic Methods. 12.1-12.2
43 To learn where the image processing techniques are applied *Image Processing Applications such as Character
recognition, Bio-medical application, Remote
sensing. On-line
materials
* The lectures may be slightly diverge from aforesaid plan based on students ‘background & interest in the topic, which may perhaps include special lectures and discussions that would be planned and schedule notified accordingly.
:
EC No
1 Quiz 1 20 Mins 7 To be
announced
in the
respective
notice
boards
2 Test 1 (Closed Book) 50 Mins 20
3 Assignement (Matlab) Continuos 18
4 Test 2 (Open Book)** 50 Mins 20
5 Comprehensive (Closed Book) 3 Hours 35
** Only prescribed text book(s) and hand written notes are permitted.
: The Assignment / Practical will be given / conducted on
either some or all of the above mentioned topics. Case studies, interpretation of data and then analysis, will form a part of all evaluation components. Assignments(s) may include seminar presentation and viva.
Details will be intimated through a separate notification or announced in the class and the deadlines would be indicated therein. However all assignments/reports would be completed by 1st week of May, 2019. It is necessary that all students stick to time schedule and do not postpone submission of assignments/reports. This will prevent extra load during last two weeks of class work. No make-ups would be allowed for submission of assignments / practical reports.
: Students are advised to read, collect additional information on the above mentioned topics as
per given schedule. In addition, awareness w.r.t. latest developments in the area would be an added advantage
:
Mid-sem grading will be displayed after two evaluation components or earlier when- ever about 40 % of
evaluation components are completed.
likely to get “NC”, if he / she
• Doesn’t appear / appear for the sake of appearing for the evaluation components / scoring zero in pre-compre total.
:
are not given as a routine. It is solely dependent upon the genuineness of the circumstances under which a student fails to appear in a scheduled evaluation component. In such circumstances, prior permission should be obtained from the Instructor-in-Charge (I/C).The decision of the I/C in the above matter will be final.
Every student is expected to be responsible for regularity of his/her attendance in class rooms and laboratories, to appear in scheduled tests and examinations and fulfill all other tasks assigned to him/her in every course. A student should have a minimum of 60% of attendance in a course to be eligible to appear for the Comprehensive Examination in that course. For the students under the purview of Academic Counseling Board (ACB), the Board shall prescribe the minimum attendance requirement on a case-to-case basis. Attendance in the course will be a deciding factor in judging the seriousness of a student which may be directly / indirectly related to grading.
:
Will be announced later in the class.
:
Students should come prepared for classes and carry the text book(s) or material(s) as prescribed by the
Course Faculty to the class.
:
All notices concerning the course will be communicated through institute Email ID and displayed on the EEE
department notice boards.
EEE F435
Dr. Jagadish Nayak Associate Professor ,
Contact details: Chamber No:145 , First floor
Email: jagadishnayak@dubai.bits-pilani.ac.in
Mobile No: 055 4907979
Advanced Studies in Biology, Vol. 9, 2017, no. 1, 1 - 7
HIKARI Ltd, www.m-hikari.com
https://doi.org/10.12988/asb.2017.61143
NSGA-II for Biological Graph Compression
A. N. Zakirov and J. A. Brown
Innopolis University, Innopolis, Russia
Copyright c 2016 A. N. Zakirov and J. A. Brown. This article is distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract
Examinations of a common biological reference organism, (E. coli), demonstrate that NSGA-II is able to provide a series of compressions at various ratios, allows a biologist to examine the organism’s connective networks with a measure of certainty of connectiveness. This is due to a novel method of scoring the similarity of the compressed network to the origional during the graph’s creation based on the number of false links added to the graph during the compression method.
Keywords: bioinformatics, genetic algorithm, graph compression
1 Introduction
Nowadays, graphs form the foundation of many real-world datasets: computer networks, social networks, biological networks. Development of technologies leads to larger and larger graphs. Some graphs contain millions or even billions of nodes and edges. Therefore, storing and processing their information has too high a cost. The interest in graph data is increasing [4], [18], [21] and many algorithms have been proposed for graph compression. Feder and Motwani [7] consider transforming a graph into a smaller one (in terms of the number of vertices and edges) that preserves certain properties of the original graph, such as connectivity. Adler and Mitzenmacher [3] and Suel and Yuan [19] consider losslessly compressing the Web Graph for efficient search engine storage and re¬trieval. In [6] there is proposed an query preserving graph compression. Work [8] a Web graph compression algorithm which can be seen as engineering of the Boldi and Vigna (2004) method is presented. Graph compression method based on representing communities with compact data structures is proposed
2 A. N. Zakirov and J. A. Brown
in [8]. Many of the ideas are similar to those proposed in [16] and [14]. They are based on hierarchical, agglomerative clustering and different methods for efficient implementation.
Comparison of various kinds of biological data is one of the main problems in bioinformatics and systems biology. The difference between biological graphs and, for example, web graphs is in weights of connections, which such graphs as E. coli regulatory network has. In [20] the weighted graph compression problem is proposed, and some initial solutions are provided, which can be used for processing biological and social graphs. Due to increased interests in systems biology, extensive studies have recently been done on comparison of biological networks. In [17] clustering based method for metabolic net¬works compression in presented. In [12], [15] data compression methods have been applied to comparison of large sequence data and protein structure data [13], [22] In [9] CompressEdge and CompressVertices methods for comparing large biological networks are proposed. [11] introduces a genetic algorithm for graphs (social and biological) compression that is based on the similarity of nodes, also, genetic algorithm approach was utilized in order to develop a compressed graph for a single compression ratio on a number of biological and non-biological graphs. This paper study the application of the Non-dominated Sorting Genetic Algorithm (NSGA-II) [5] for biological graph compression. We target a good compression ratio as well as keeping as much natural biological information in compressed graph as possible.
2 Materials and Methods
The NSGA-II, see [5], is a Multiple Objective Optimization (MOO) algorithm and is an instance of an Evolutionary Algorithm from the field of Evolutionary Computation. NSGA-II is an extension of the Genetic Algorithm for multiple objective function optimization. The objective of the NSGA algorithm is to improve the adaptive fit of a population of candidate solutions to a Pareto front constrained by a set of objective functions. The algorithm uses an evolutionary process with surrogates for evolutionary operators including selection, genetic crossover, and genetic mutation. The population is sorted into a hierarchy of sub-populations based on the ordering of Pareto dominance. Similarity between members of each sub-group is evaluated on the Pareto front, and the resulting groups and similarity measures are used to promote a diverse front of non-dominated solutions.
2.1 Objectives and fitness functions
On problem of biological graph compression we have an objectives of compress¬ing the graph and still save enough information about its origin. The fitness
NSGA-II for biological graph compression 3
Figure 1: Example of fake link appearance after compressing original graph a) by merging nodes 3 and 4 on b). Fake link 2-4 on c) appears after decompres-sion
function is based on the notion of similarity of original and compressed graphs. The similarity definition is related to fake links, that appear after compression and then decompression of the graph (fig 1). So the similarity is
Fr
S = 1 FT
where Fr is number of fake links that appears after decompression for tested graph and FT is total number of all possible fake links in the graph. FT is calculated from the original graph and its maximum compression in a one node, which gives as full graph after decompression. Amount of links in this full graph is n(n 1)/2 , where n is a number of nodes in graph. Amount of links in original graph is known. The difference of this two link number will be FT. We are targeting minimizing the fake links while maximizing the compression. Minimizing the fake links equals to maximizing the similarity S. The compression ratio is
N,
C =
No
where N, is number of nodes after compression and No is original number of nodes.
2.2 Dataset
Examining the dataset exampled for the compression of biological networks. Different biological networks are available at [2]. In this work, we are targeting the processing of the gene regulatory network of Escherichia coli (E. coli) [1]. This network is a relatively small network, it was cleaned of all duplicate links (nodes that indicate both activation and inhibition) and all unknown links. Our final graph consist of 1123 nodes and 2108 edges.
4 A. N. Zakirov and J. A. Brown
Figure 2: Results of algorithm after 1 (a), 25 (b) and 100 (c) populations
3 Results and Discussion
As evolution progresses we see a relatively steady increase in the relative fitness of the compression. As we can see on figure 2 (a) , first population is far away from the optimal condition. Nevertheless, after 25 generations (b) we can see how Pareto fronts began to appear and move towards optimum. The last results - 100 generations - is shown on (c). We run our tests with mutation rate 0.25 and crossover rate 0.9. This parameters have shown best results in [11]. Nevertheless, influence of mutation and crossover rates is field for further researches.
The results show that NSGA-II forms a thigh front to optimum similarity until a compression ratio of about 0.3. The Similarity degrades with increasing speed until at about 0.8 we have nearly half of the links in the graph being false. By developing not just a single good graph at a target compression ratio, such as in [11], but a series of graphs at many ratios, a biologist can select the highest amount of compression with minimal loss of information contained in the graph.
4 Conclusion
Taking into consideration biological background of graph, NSGA-II, as multi-objective genetic algorithm, provide more natural view of space, than the al-gorithms focusing only compression ratio as a measurement. We proposed similarity of original and compressed graph as a second objective. Further testing on other biological datasets is required in order to demonstrate the generality of the method.
NSGA-II for biological graph compression 5
Acknowledgements. This study was funded by RFBR according to the research project No. 16-31-60090.
References
[1] Avi Maayan: E.coli gene regulatory network dataset.
http://research.mssm.edu
[2] Avi Maayan: Network datasets for download.
http://research.mssm.edu/maayan/datasets/qualitative networks.shtml
[3] M. Adler, M. Mitzenmacher, Towards compressing web graphs, In Pro-ceedings of the Data Compression Conference, (2001), 203-212. https://doi.org/10.1109/dcc.2001.917151
[4] R. Agrawal, T. Imielinski, and A. Swami, Mining association rules be-tween sets of items in large databases, SIGMOD Rec., 22 (1993), 207-216. https://doi.org/10.1145/170036.170072
[5] K. Deb, A. Pratap, S. Agarwal and T. Meyarivan, A fast and elitist multi-objective genetic algorithm: NSGA-II, IEEE Transactions on Evolution¬ary Computation, 6 (2002), 182-197. https://doi.org/10.1109/4235.996017
[6] W. Fan, J. Li, X. Wang and Y. Wu, Query preserving graph compression, In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, (2012), 157-168. https://doi.org/10.1145/2213836.2213855
[7] T. Feder, R. Motwani, Clique partitions, graph compression and speeding-up algorithms, In Proceedings of the Twenty-third Annual ACM Sympo¬sium on Theory of Computing, (1991), 123-133. https://doi.org/10.1145/103418.103424
[8] S. Grabowski, W. Bieniecki, Tight and simple web graph compression for forward and reverse neighbour queries, Discrete Appl. Math., 163 (2014), 298-306. https://doi.org/10.1016/j.dam.2013.05.028
[9] M. Hayashida, T. Akutsu, Comparing biological networks via graph com¬pression, BMC Systems Biology, 4 (2010). https://doi.org/10.1186/1752-0509-4-s2-s13
[10] C. Hernandez, G. Navarro, Compressed representations for web and social graphs, Knowl. Inf. Syst., 40 (2014), 279-313. https://doi.org/10.1007/s10115-013-0648-4
6 A. N. Zakirov and J. A. Brown
[11] T. K. Collins, J. A. Brown, S. K. Houghten and Q. Qu, Evolving graph compression using similarity measures for bioinformatics applications, IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology, (2016).
[12] A. Kocsor, A. Kertesz-Farkas, L. Kajan and S. Pongor, Application of compression-based distance measures to protein sequence classifi¬cation: A methodological study, Bioinformatics, 22 (2005), 407-412. https://doi.org/10.1093/bioinformatics/bti806
[13] N. Krasnogor, D. A. Pelta, Measuring the similarity of protein structures by means of the universal similarity metric, Bioinformatics, 20 (2004), 1015-1021. https://doi.org/10.1093/bioinformatics/bth031
[14] K. LeFevre, E. Terzi, Grass: Graph structure summarization, Proceedings of the 2010 SIAM International Conference on Data Mining, (2010), 454¬465. https://doi.org/10.1137/1.9781611972801.40
[15] M. Li, J. H. Badger, X. Chen, S. Kwong, P. Kearney and H. Zhang, An information-based sequence distance and its application to whole mitochondrial genome phylogeny, Bioinformatics, 17 (2001), 149-154. https://doi.org/10.1093/bioinformatics/17.2.149
[16] S. Navlakha, R. Rastogi and N. Shrivastava, Graph summarization with bounded error, In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, (2008), 419-432. https://doi.org/10.1145/1376616.1376661
[17] H. Ogata, W. Fujibuchi, S. Goto and M. Kanehisa, A heuristic graph comparison algorithm and its application to detect functionally re¬lated enzyme clusters, Nucleic Acids Research, 28 (2000), 4021-4028. https://doi.org/10.1093/nar/28.20.4021
[18] Q. Qu, J. Qiu, C. Sun and Y. Wang, Graph-based knowledge repre¬sentation model and pattern retrieval, 2008 Fifth International Confer¬ence on Fuzzy Systems and Knowledge Discovery, 5 (2008), 541-545. https://doi.org/10.1109/fskd.2008.7
[19] T. Suel, J. Yuan, Compressing the graph structure of the web, In Pro-ceedings of the IEEE Data Compression Conference, (2001), 213-222. https://doi.org/10.1109/dcc.2001.917152
[20] H. Toivonen, F. Zhou, A. Hartikainen and A. Hinkka, Compression of weighted graphs, In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, (2011), 965-973. https://doi.org/10.1145/2020408.2020566
NSGA-II for biological graph compression 7
[21] X. Yan, M. R. Mehan, Y. Huang, M. S. Waterman, P. S. Yu and X. J. Zhou, A graph-based approach to systematically reconstruct human transcriptional regulatory modules, Bioinformatics, 23 (2007), i577-i586. https://doi.org/10.1093/bioinformatics/btm227
[22] Y. Zhao, M. Hayashida and T. Akutsu, Integer programming-based method for grammar-based tree compression and its application to pat-tern extraction of glycan tree structures, BMC Bioinformatics, 11 (2010). https://doi.org/10.1186/1471-2105-11-s11-s4
Received: November 30, 2016; Published: December 15, 2016
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
FH Campus Wien;
, Óbuda University
Project managers managing agile developed IT projects often find themselves in difficult situations. Their frameworks, like PMBOK project management framework of PMI Organizations, demand a deep level of planning, control and active management. On the other side, agile development frameworks like SCRUM demand self-management, flexibility and appreciate change. This article proposes solutions for five PMBOK processes that have been identified as critical in SCRUM development environments in the previous publication Suitability of PMBOK 6th edition for agile-developed IT Projects, by Rosenberger and Tick. The process of “Manage project execution” is adapted by introducing Strike Events; “Work Breakdown Structure Plan creation” and “Scheduling” processes are changed by dividing large backlogs into phases and break down individual phases into Macro and Micro level planning; “Cost Estimation” processes uses velocity of development teams as planning reference; “Developing and Managing Teams” is adapted by introducing the project manager as SCRUM master and if needed apply again the Strike System in case of serious problems. These proposed solutions adapt the classical PMBOK project framework to cope with SCRUM developed project to an “Agile IT Project Management Framework”. These process specific solution results are based on literature research. The actual applicability in agile developed projects and adaptations will researched and applied in a following step of this research topic towards the way of creating an optimized, tailored agile IT project management framework.
SCRUM, IT-Project Management, Agile, PMBOK
M15 (IT-Management)
Published in 2001 the agile manifesto (Agile Manifesto, 2001) provided the basis for SCUM framework of agile development in IT projects. The goal was to make development processes more flexible and to achieve early results for customer feedback. But the SCRUM framework as defined in the SCRUM Guide (SCRUM Guide, 2017, Schwaber K.& Sutherland) describes only an
1Second Editions are previously published papers that have continued relevance in today’s project management world, or which were originally published in conference proceedings or in a language other than English. Original publication acknowledged. This paper was originally presented at the 8th Scientific Conference on Project Management in the Baltic States at the University of Latvia in April 2019. It is republished here with the permission of the authors and conference organizers
2 How to cite this paper: Rosenberger, P. and Tick, J. (2019); Adaption of Selected PMBOK Processes To Fit SCRUM Developments; presented at the 8th Scientific Conference on Project Management in the Baltic States, University of Latvia, April 2019; republished in the PM World Journal, Vol. VIII, Issue VI, July.
© 2019 University of Latvia www.pmworldlibrary.net 1 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
agile process of software development. It was not meant to be seen as a project management approach.
But in reality, SCRUM is often used as a “agile project management” framework. By adopting agile tools and methods, or sometimes even just terminologies used in SCRUM organizations pretend to use agile project management approaches, without even deeply understanding the real nature of agile project management. However, these organizations are not being blamed. There is no real finalised “agile IT project management” framework existing at the moment. There are classical project management frameworks like PRINCE2 (Prince 2 Handbook, 2017, Axelos Global Best Practice) or PMBOK (PMBOK-Guide) – Sixth version, 2017, Project Management Institute, Pennsylvania, USA) of PMI organization. And then there are agile development models like SCRUM, which are used in classic project environments.
So when agile IT project management is defined as classical project management, including an agile development approach, problems can develop due to the fact that these two frameworks focus sometimes on completely different values. This cultural inaptitude, often results in decreased overall project success, problems in communication and understanding of project participants.
This article sets up the basis for an adapted PMBOK project framework specially focussed on agile, with SCRUM, developed IT projects. PMI organisation already took a first step in this direction by adding an “agile guideline” document to its newest sixth version of the PMBOK framework. But this guideline is only an introduction in agility and agile methods and tools. It does not change the processes defined in PMBOK as such.
To now completely redefine the PMBOK processes and make them suitable for SCRUM developed IT projects two steps need to be taken:
1) Critical areas of the PMBOK processes have to be defined.
2) Solutions regarding these areas have to be investigated, analysed and evaluated
The first step of identifying critical processes has already happened. In the IEEE publication “Suitability of PMBOK 6th edition for agile-developed IT Projects” (Rosenberger P. & Tick J ,2018) five processes have been identified to cause problems:
• Manage project execution
• Develop project structure plan
• Develop project schedule
• Estimate and define costs based on requirements
• Develop and manage team
This article now uses these identified critical areas as starting point and proposes approaches to be integrated into the existing PMBOK framework. These proposed solutions are based on existing tools and methods identified by literature research and followed by an assessment of applicability using a KPI evaluation. Please note, that the last step of proofing the applicability of the proposed solutions via a large scale online survey is yet not finalised and therefore not part of this article.
© 2019 University of Latvia www.pmworldlibrary.net 2 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
Solution proposal for PMBOK process “Manage project execution”
In traditional project management according PMI, a project manager is responsible for managing the project team and its execution (PMBOK-Guide) – Sixth version, 2017, Project Management Institute, Pennsylvania, USA). SCRUM, as a contrast, demands strict self-management of the development team (SCRUM Guide, 2017, Schwaber K.& Sutherland). Only the team itself takes care about delivering quality results. Often these self-managing development teams are even protected from any disturbance or influence by a SCRUM Master. This difference shows the gap between the two frameworks. The agile project manager has to take overall responsibility of the project and the product, but is not allowed to actively manage the execution in regards of programming done by the agile development team.
Three different solution approaches have been investigated and will later be evaluated using suitable KPIs:
1) Strike System
Lewthwaite (Lewthwaite, J., 2006) defines a “Strike” as a proactive intervention of a project manager overruling the self-management of SCRUM development teams. This overruling once started lasts the rest of the ongoing sprint. Trigger for such shifts in responsibilities need to be substantial because strikes completely undermine agile culture of self-management and trust. Trigger of such strike events need to be defined in detail to create a common understanding and avoid negative personal feelings as much as possible. Strikes could for example be triggered by:
• SCRUM Master intervention
• Danger of non-deliverable increments at the end of a sprint
• Extreme delay visualized in burndown charts
• Extreme bottle necks visualized on KANBAN boards,
• Great changes in effort estimations of user stories during a sprint in comparison to estimations in sprint planning meetings.
The strike system is therefore a kind of “Management by Exception” methodology.
2) Indirect Management by Backlog
Lewthwaite (Lewthwaite, J., 2006) mentioned regarding large scale IT projects that roll like a product manager or project manager acting outside a SCRUM development team can manage project execution indirectly by influencing the product backlog. These roles can change the completeness of user stories and priorities of user stories. With these tools, they can indirectly decide what will be developed next.
3) Traditional project management of chosen SCRUM artefacts
Pichler (Pichler, R., 2007.) suggests allowing switching specific SCRUM artefacts or epics from an agile towards a traditional project approach like waterfall. By doing so, a project manager can actively and directly manage the execution of this artefact. It needs to be mentioned, that keeping a well-functioning agile culture alive could get much harder by such interferences. Additional agile tools and methods like daily stand-ups and retrospectives can and should be kept alive, even in these “traditional managed islands”.
© 2019 University of Latvia www.pmworldlibrary.net 3 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
After presenting three possible solutions based on literature research a most suitable solution needs to be chosen by application of a pointing system measuring PMBOK in regards of several related success criteria. Points will be assigned based on the applicability of success criteria in regards to PMBOK integration of solution approaches in such way:
• Easy to be integrated into PMBOK processes: 2 points
• Possible but not easily to be integrated into PMBOK processes: 1 point
• Hardly to be integrated into PMBOK processes: 0 points
Note, that this approach of solution selection will be used on all 5 processes
Table1
Success Criteria Strike System Indirect Mgmt. Traditional Artefacts
Direct PM influence of project execution enabled? 2 0 2
PM influence of project execution
measurable? 2 0 2
Usable for all project scales? 2 1 (only
large scale) 2
Extensive communication required? 2 1 1
SUM
Conclusion
After comparison of three potential solutions enabling IT project managers in agile developed projects to manage project execution without disturbing SCRUM processes or culture, it shows that the Strike system is potentially the best candidate to be integrated in this particular PMBOK process. It acts without disturbing SCRUM development at all in most of the time. Only when high involvement of a project manager is needed in projects, it is used to solve issues that the self-managing development team was not able to solve on their own. This management by exception approach combines traditional project management methods and agile frameworks in the least conflicting way.
Solution proposal for PMBOK process “Develop Project Structure Plan” and “ Scheduling”
Note: Due to the strong relation between PMBOK Processes Structure Plan Development and Project Scheduling”, these two processes are analysed together
Traditional project management structures and schedules the whole project in the initial planning phase. This regards all work packages. Even work packages that are still far away in the future and very uncertain. There is no difference in the level of planning between certain and uncertain work packages accepting that uncertain packages may change in the future causing the project schedule to be adapted. SCRUM totally avoids this restructuring and re¬planning by just focusing on the next sprint. This gap in the two approaches can result in major conflicts between agile developments and traditional project managers. Three different solution approaches are now proposed and will later be evaluated using suitable KPIs:
© 2019 University of Latvia www.pmworldlibrary.net 4 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
Hybrid Macro and Micro Planning of Project Schedule and Structure
A hybrid approach could differentiate between a macro and micro structuring level - also separating the two cultures. The project manager keeps the overall scope and focus by structuring the whole project like usually with a project structure plan, but only on a macro level. Accepting, not knowing definite responsibilities and durations. But for example only T-Shirt size estimations on an epic instead of a user story level.
But during actual development, in development sprints, micro planning in form of planning poker story point estimations can be used in sprint planning meetings to get into details.
After several sprints, a factor between actual effort and rough T-Shirt size can be postulated. So with experience in project delivery, a project manager could even get quite detailed effort and structural estimations enabling him to even develop an understanding about longer term planning (Wendt R.., 2016).
1) Project Phase Specific Backlogs
This approach does not change the structural planning in the initial project phases at all. A project manager, will create a work breakdown structure and will define project phases and major milestones based on a basic specification in a traditional way. All these major project phases are then seen as “mini-agile-projects” within a traditional project. Each phase has its own specific backlog, SCRUM team and goal. With such an approach, the two cultures can easily coexist. On a big level, managed by a project manager in a traditional way, on the small level in a purely agile SCRUM based approach with minimal project management interference. (The Project Group, TPG Phase Method, 2019).
2) Extrapolation and continuous adjustment
When an agile development framework like SCRUM should not be changed, adapted or disturbed at all, a project manager could accept not planning and structuring before the start at all. Just starting the development, when the first user stories are ready for development in the product backlog. Based on a comparison of story points associated before the sprints to user stories and the actual needed time and cost consumption extrapolation about the open efforts and timelines with the currently existing product backlog can be made, fulfilling the PMBOK need to plan, structure and schedule. These extrapolations will get more and more accurate and refined, when more and more sprints have been finalised and learnings from these sprints are available for adjustment of extrapolation. Due to a product backlog that is typically constantly changing in agile projects, the planning and work breakdown structures will also be affected by these changes and the project manager has to constantly keep them up to date. In such a role, the project manager is essentially just “documenting change” and not really managing change, to apply to PMBOK process requirements.
Table2
Success Criteria Macro/ Micro Phase
specific
backlogs Extrapolation
Effort of planning and work
breakdown structure creation and
maintenance 2 2 0
Accuracy of planning 1 2 1
Agility and flexibility of planning and structure 2 1 2
SUM
© 2019 University of Latvia www.pmworldlibrary.net 5 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
Conclusion
Planning project work structures and based on those, planning the timing of projects is challenging in agile developed projects, because planning is based on fixed assumptions and agility is based on flexibility. However, the analysis of potential solutions and their applicability shows, that two approaches are especially usable to close this gap between traditional project management and agile development. Phase specific backlogs divide large projects into phases enabling maximal flexibility within the phases themselves. In these phases accepting only macro level planning in the beginning and micro level planning only on sprint level can help traditional project managers to fulfil PMBOK requirements.
Solution proposal for PMBOK process “Estimate and defin
requirements”
Cost estimation in traditional projects normally consists about manpower related costs and material related costs. IT projects, basically sharing these concepts with all other projects, often develop most of their costs manpower related. Often the actual time and effort invested by people is much more significant then investments in hardware or other material. Based on this understanding, the cost estimation can also be split in two parts:
• Material related costs:
This part of costs are untouched by agile development frameworks
• People related costs:
These costs are difficult to estimate and define, because complete and traditional requirements are missing in SCRUM developed IT projects, due to constant backlog changes.
So, focussing on people related costs, the following two approaches could be integrated into PMBOK processes:
1) Cost estimation based on Development Velocity
Velocity is a key performance indicator of agile development teams, describing the amount of Story Points being developed in each sprint in average. It’s the speed, the SCRUM teams are developing with. Often this measurement is also used in portfolio management of agile developed project portfolios (Rouse, M., 2013). Knowing and tracking the velocity of development teams can enable an agile project manager to estimate project costs. Knowing the developers involved and their internal and external hourly rates, the project manager can summarize the cost of one story point, or one average user story based on the amount of user stories developed by the team in one sprint. So relying on the planning of work breakdown structure and project scheduling and knowing the development teams and their costs and velocity the project manager can simply multiply planned development effort with velocity related cost factors and therefor develop a cost planning in the same way as a scheduling. It is important to mention, that the velocity can change and therefor the basis of the cost factor can change. The project manager needs to keep constant track of this factor.
2) Fixed Minimal Viable Product and unplanned ongoing feature development costs
A minimal viable product (MVP) is often used as basic concept of so called “hybrid” IT projects. This MVP is the smallest, fastest and most simple set of features providing desired functionality, without taking care of usability, design, safety, reliability and all other necessary
© 2019 University of Latvia www.pmworldlibrary.net 6 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
factors of a quality system. In hybrid IT projects the development of such MVPs is planned and executed in a classical waterfall approach, which is easily manageable with PMBOK processes due to the high level of planning activities and rigid structure. After finalization of the MVP increment, features and “quality” is added to the system in a strictly agile way. This hybrid approach of splitting MVPs and agile feature integration can also be used to solve cost estimation gaps in agile projects. Classical cost estimations are used to define MVP costs, and no cost estimation is used at all for agile feature integration later on. This allows strict separation of agile and classical frameworks, avoiding problems (Sharma, S, 2017).
Success Criteria Velocity based MVP based
Applicability for different project
categories and sizes 2 1
Accuracy of cost estimation 1 2
Effort of cost estimation 2 1
SUM
Conclusion
Velocity based cost estimation present itself as a usable solution for agile developed projects by building up on planning regarding schedule and project structure and just applying people-related costs. In addition to this solution classical cost estimation methods can be used, when the definition of a Minimal Viable Product makes sense or is defined in a hybrid project management environment. In this case, only the agile part of the hybrid project should use velocity based cost estimations. It is important to mention, that MVPs minimize the agile culture and advantages to a great extent, and the decision to use them should be taken carefully and with intensive communication effort to all project stakeholders.
Solution proposal for PMBOK process “Develop and Manage Team”
According PMBOK(PMBOK-Guide) – Sixth version, 2017, Project Management Institute,
Pennsylvania, USA), the project manager is responsible for organizing and managing resources, including human resources, for the project. The organizational part is not as critical. A project manager can and will set up a project team and including development teams in the initial project phase. As soon as the development team is set up, it demands self-management. Meaning, that there should not be active management and controlling from outside. This characteristic is a very strong one in SCRUM. The teams share work and task internally and are even “protected by a SCRUM master from outside disturbances. So as soon as a development team is set-up by a project manager the management tasks are taken away from the project manager and are transferred to the team itself. This shift in responsibility can cause trouble in a project and challenge a traditional PMI project manager who needs to take all-over project responsibility.
Two solutions approaches have been identified to close this gap:
1) Adaptation of Strike System for team management
As described in the first process, Lewthwaite (Lewthwaite, J., 2006) mentions a strike system as a potential compromise to share responsibility between self-managing SCRUM teams and outside project managers. This approach can not only be used in project execution, but also in processes of team-management. Potential trigger of Strike-Situations, in which the project manager will pause self-management of the team and take over, could be retrospective meetings, in which problems within the development team are discussed. It is important to
© 2019 University of Latvia www.pmworldlibrary.net 7 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
define clear situations within that retrospective to start a strike-action. Otherwise, development teams will always hesitate to solve problems within the retrospective meeting in fear of a potential loss of self-management.
2) Project Manager takes role as SCRUM Master
If a project manager is comfortable in only being “inside” of a self-managing development team but accepting their demands of self-organization, he or she could take the roll as SCRUM master. Within this role the project manager can actively trigger team-problem-solving in retrospective events or even on a daily-stand-up basis. This realization of critical situation and the start of a problem solving process is often more than enough to keep projects and team structures productive, even without acting as authority and directly manage and decide changes.
Success Criteria Strike System PM as SCRUM
Master
Applicability for different project
categories and sizes 2 2
Interference with agile culture and
methods 1 2
Amount of influence by PM if
necessary 2 1
SUM
Conclusion
Comparing both solutions proposes with each other and assessing their applicability with SCRUM methods as well as their ability to be integrated into PMBPOK processes, the KPIs show no favourite solution. So a potential approach combines both solutions. A project manager acting as SCRUM Master in non-critical project phases, triggering team communication and problem solving when necessary, but at the same time, being able to take corrective measures, using authority by applying the strike system.
This article searches for solutions regarding PMBOK processes not suitable for SCRUM developed IT projects. In the article “Suitability of PMBOK 6th edition for agile-developed IT Projects” (Rosenberger P. & Tick J ,2018) critical PMBOK processes have been identified. Based on literature research and a KPI based evaluation of possible solutions, the following methods can be used for PMBOK integration:
•
Strike Systematic enables the project manager to take full responsibility, via management by exception only when necessary, so the agile culture of self-management and personal responsibility is not disturbed at all as long as not necessary.
•
Phase specific backlogs divide large projects into phases which can be easily planned by an experienced project manager. Within these phases, only macro level planning at the
phase beginning and micro level planning and structuring on a sprint level is necessary.
© 2019 University of Latvia www.pmworldlibrary.net 8 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
•
Cost estimation based on agile structure planning and scheduling using velocity estimations of development teams and their manpower related costs can be used by project managers to handle work-related costs. No changes are necessary in estimating and defining material related costs.
•
Project Manager acts as SCRUM Master passively influencing team related problem solving and using temporary Strike System to stop classical SCRUM rolls and approaches and using management authority to manage projects as long as necessary.
Please note that these results are only based on detailed literature research and KPI assessments, but not proven to be effective in agile environment. In a next step of research, the effectiveness of these solutions will be measured by optimizing the PMBOK processes including the new solutions.
Agile Manifesto, 2001, [Online] Available at: https://agilemanifesto.org/ [Accessed 11.03.2019]
Lewthwaite, J., 2006. Managing People for the First Time: Gaining Commitment and Improving Performance. Thorogood Publishing
Project Management Institute, A Guide To The Project Management Body Of Knowledge (PMBOK-Guide) – Sixth version, 2017, Project Management Institute, Pennsylvania, USA
Pichler, R., 2007. Scrum - Agiles Projektmanagement erfolgreich einsetzen, 1 edition. ed. dpunkt.verlag, Heidelberg
Prince 2 Handbook, 2017, Axelos Global Best Practice, TSO The Stationary Office
Rosenberger P. & Tick J., Suitability of PMBOK 6th edition for agile-developed IT Projects, 2018, CINTI 2018 IEEE 18th International Symposium on Computational Intelligence and Informatics, Budapest Hungary
Rouse, M., 2013, Agile velocity [Online]. Available at http://whatis.techtarget.com/definition/Agile-velocity . [Accessed 11.03.2019]
SCRUM Guide, 2017, Schwaber K.& Sutherland J., [Online] Available at:
https://www.scrumguides.org/index.html [Accessed 11.03.2019]
Sharma, S, 2017, The DevOps Adoption Playbook. A Guide to Adopting DevOps in a Multi-Speed IT Enterprise. Wiley, s.l.
The Project Group, TPG Phase Method, [Online] Available at:
https://www.theprojectgroup.com/en/services/project-management-consulting/ms-project-implementation.html [Accessed 11.03.2019]
Wendt R.., 2016, Hybrides Projektmanagement für agile Projekte in mittelständischen und großen Unternehmen, [Online] Available at:
http://www.masventa.eu/fileadmin/user upload/media/pdf de/PMI/PMI-SG-Live 2016-12 1 .pdf, [Accessed 11.03.2019]
© 2019 University of Latvia www.pmworldlibrary.net 9 10
© 2019 Professional Association of Project Managers
PM World Journal Adaption of Selected PMBOK Processes To Fit SCRUM
Vol. VIII, Issue VI – July 2019 by Philipp Rosenberger, József Tick
www.pmworldjournal.com Second Edition – Conference Paper
About the Authors
is a professor at FH Campus Vienna at master program technical management focusing on IT project management in an agile development context. After many years in consulting focussing on technical aftersales and business as well as project management and especially IT project management development in Europe and China, he got into the financial sector, managing the implementation of a current account financial product implementation project at ING DiBa Online bank in Vienna and in parallel starting his own small consulting company ROSCON.at. In his current PhD program Philipp is focussing on improving agile IT project management models. Prof Rosenberger can be contacted at Philipp.rosenberger@fh-campuswien.ac.at
Budapest, Hungary
is an associate professor in the Institute of Software Design and Development, at the John von Neumann Faculty of Informatics of Óbuda University, Budapest, Hungary. He received his BSc in Computer Science in 1977, his Master Degree in Meseure- and Controlsystems in 1986, and his PhD degree in Computer Science in 2007 from the University of Veszprém. His research areas are Simulation of controlsystems, Object-oriented Software Development, Software Reuse, User Interface Design and Embedded System-control. He did a one year research in the field of Software Engineering at the Research Centre for Informatics in Karlsruhe Germany. Since 2000 he is the Vice rector of Óbuda University. He is an author and co¬author of numerous conference papers; he has given several presentations on national and international conferences. He has acted as a Program and Technical Committee Member on several international conferences. József can be contacted at tick@uni-obuda.hu
© 2019 University of Latvia www.pmworldlibrary.net 10 10
© 2019 Professional Association of Project Managers
www.elsevier.com/locate/jnca
D-SCIDS: Distributed soft computing intrusion detection system
Ajith Abrahama,~, Ravi Jainb, Johnson Thomasc,
Sang Yong Hana
aSchool of Computer Science and Engineering, Chung-Ang University, Korea
bUniversity of South Australia, Adelaide, Australia
cComputer Science Department, Oklahoma State University, OK 74106, USA
Received 28 June 2005; accepted 28 June 2005
Abstract
An Intrusion Detection System (IDS) is a program that analyzes what happens or has happened during an execution and tries to find indications that the computer has been misused. A Distributed IDS (DIDS) consists of several IDS over a large network (s), all of which communicate with each other, or with a central server that facilitates advanced network monitoring. In a distributed environment, DIDS are implemented using co-operative intelligent agents distributed across the network(s). This paper evaluates three fuzzy rule-based classifiers to detect intrusions in a network. Results are then compared with other machine learning techniques like decision trees, support vector machines and linear genetic programming. Further, we modeled Distributed Soft Computing-based IDS (D-SCIDS) as a combination of different classifiers to model lightweight and more accurate (heavy weight) IDS. Empirical results clearly show that soft computing approach could play a major role for intrusion detection.
© 2005 Elsevier Ltd. All rights reserved.
'Corresponding author.
E-mail addresses: ajith.abraham@ieee.org, , hansy@cau.ac.kr (A. Abraham), ravi.jain@unisa.edu.au
(R. Jain), jpt@okstate.edu (J. Thomas).
1084-8045/$ -see front matter © 2005 Elsevier Ltd. All rights reserved.
doi:10.1016/j.jnca.2005.06.001
82 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 1. Introduction
An intrusion is defined as any set of actions that attempt to compromise the integrity, confidentiality or availability of a resource. Intrusion detection is classified into two types: misuse intrusion detection and anomaly intrusion detection (Mukkamala et al., 2005). Misuse intrusion detection uses well-defined patterns of the attack that exploit weaknesses in system and application software to identify the intrusions. These patterns are encoded in advance and used to match against the user behavior to detect intrusion. Anomaly intrusion detection uses the normal usage behavior patterns to identify the intrusion. The normal usage patterns are constructed from the statistical measures of the system features. The behavior of the user is observed and any deviation from the constructed normal behavior is detected as an intrusion (Denning, 1987; Summers, 1997). In Distributed Intrusion Detection System (DIDS) conventional intrusion detection system are embedded inside intelligent agents and are deployed over a large network. In a distributed environment, IDS agents communicate with each other, or with a central server. By having these co-operative agents distributed across a network, incident analysts, network operations, and security personnel are able to get a broader view of what is occurring on their network as a whole. Distributed monitoring allows early detection of planned and coordinated attacks, thereby allowing network administrators to take preventive measures. DIDS also helps to control the spreading of worms, improves network monitoring and incident analysis, attack tracing and so on. It also helps to detect new threats from unauthorized users, back-door attackers and hackers to the network across multiple locations, which are geographically separated (Abraham and Thomas, 2005). In a DIDS it is important to ensure that the individual IDS are lightweight and accurate.
Data mining approaches for intrusion detection were first implemented in mining audit data for automated models for intrusion detection (Barbara et al., 2001; Cohen, 1996; Lee et al., 1999). Several data mining algorithms are applied to audit data to compute models that accurately capture the actual behavior of intrusions as well as normal activities. Audit data analysis and mining combine the association rules and classification algorithm to discover attacks in audit data. Soft Computing (SC) is an innovative approach to construct computationally intelligent systems consisting of artificial neural networks, fuzzy inference systems, approximate reasoning and derivative free optimization methods such as evolutionary computa¬tion, etc. (Zadeh, 1998). This paper introduces three fuzzy rule-based classifiers (Abraham et al., 2004) and compares its performance with Linear Genetic Programming (LGP) (Abraham, 2004), Support Vector Machines (SVM) (Vapnik, 1995) and Decision Trees (DT) (Brieman et al., 1984; Peddabachigari et al., 2004). Further, we modeled Soft Computing (SC)-based IDS (SCIDS) (Abraham et al., 2004) as a combination of different classifiers to model lightweight and more accurate (heavy weight) IDS. The rest of the paper is organized as follows. Section 2 provides a brief overview of the research on distributed intrusion detection systems. Soft computing for intrusion detection is introduced in Section 3 followed by the importance of attribute reduction (important feature selection) in Section 4.
A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 83
Experimental results are also presented in Section 4 followed by conclusions in Section 5.
2. Distributed intrusion detection system (DIDS)
A number of IDSs have been proposed for a networked or distributed environment. Early systems included ASAX (Mouinji et al., 1995), DIDS (Snapp et al., 1999) and NSTAT (Kemmerer, 1997). These systems require the audit data collected from different places to be sent to a central location for analysis. NetSTAT (Vigna and Kemmerer, 1999) is another example of such a system. In NetSTAT attack scenarios are modeled as hypergraphs and places are probed for network activities. Although NetSTAT also collects information in a distributed manner, it analyses them in a central place. The scalability of such systems is limited due to their centralized nature. To improve scalability later systems such as EMERALD (Porras and Neumann, 1997), GriDS (Staniford–Chen et al., 1996) and AAFID (Spafford and Zamboni, 2000), deployed instruction detection systems at different locations and organized them into a hierarchy such that low-level IDSs send designated information to higher level IDSs. EMERALD uses both misuse detection and statistical anomaly detection techniques by having a recursive framework, which allows generic components to be deployed in a distributed manner. To detect intruders, GriDS aggregates computer and network information into activity graphs which reveal the causal structure of network activity. AAFID consists of agents, filters transceivers and monitors organized in a tree structure. The hierarchical approaches employed by theses schemes scale better than the previous centralized approach. However, the main problem with such an approach is that if two or more IDSs that are far apart in the hierarchy detect a common intruder, the two detection cannot be correlated until the messages from the different IDSs reach a common high-level IDS. This will require the messages to traverse multiple IDSs resulting in communication overheads. The Common Intrusion Detection Framework (CIDF) (Staniford-Chen et al., 1998) goes one step further as it aims to enable different intrusion detection and response components to interoperate and share information and resources in a distributed environment. The intrusion detection inter-component adaptive negotiation protocol helps cooperating CIDF components to reach an agreement on each other’s needs and capabilities (Feiertag et al., 2000). MADAM ID uses CIDF to automatically get audit data, build models, and distribute signatures for novel attacks so that the gap between the discovery and detection of new attacks can be reduced (Lee et al., 2000). The coordinated and response system (CARDS) (Ning et al., 2002) aims at detecting distributed attacks that cannot be detected using data collected from any single location. CARDS decomposes global representations of distributed attacks into smaller units that correspond to the distributed events indicating the attacks. It then executes and coordinates the decomposed smaller units in the places where the corresponding units are observed. The message transmission between component IDSs is not determined by a centralized or hierarchical scheme, Instead, in CARDS, one component IDS sends a
84 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98
message to another only when the message is required by the later IDS to detect certain attacks. The communication cost is therefore reduced. Although JiNao (Jou et al., 2000) has been proposed as a distributed IDS for detecting intrusions network routing protocols, no specific mechanisms have been provided for doing so. JiNao focuses on the Open Shortest Path First (OSPF) protocol.
Software agents have been proposed as a technology for intrusion detection applications. Rationale for considering agents in an IDS ranges from increased adaptability for new threats to reduced communication costs. Since agents are independently executing entities, there is the potential that new detection capabilities can be added without completely halting, rebuilding, and restarting the IDS. Other potential advantages are described in Jansen et al. (1999) and Kruegel and Toth (2001). Kruegel and Toth (2001) also identify downside tradeoffs including increased design and operational complexity. The Autonomous Agents for Intrusion Detection (AAFID) framework (Spafford and Zamboni, 2000) employs autonomous agents for data collection and analysis. AAFID utilizes agents hosted on network nodes, filters to extract pertinent data, transceivers to oversee agent operation, and monitors to receive reports from transceivers. These entities are organized into a hierarchical architecture with centralized control. Cooperating Security Managers (CSMs) (White et al., 1996) enable individual distributed intrusion detection packages to cooperate in performing network intrusion detection without relying on centralized control. Each individual CSM detects malicious activity on the local host. When suspicious activity is detected, each CSM will report any noteworthy activity to the CSM on the host from which the connection originated. The local CSM will not notify all networked systems, but rather only the system immediately before it in the connection chain. Other agent-based hierarchical architectures include the Intelligent Agents for Intrusion Detection project (Helmer et al., 1998) with a centralized data warehouse at the root, data cleaners at the leaves, and classifier agents in between. Bernardes and dos Santos Moreira (2000) have proposed a hybrid framework with partially distributed decision making under the control of a centralized agent manager. Agents are deployed to observe behavior of the system and users. Agents communicate via messages to advise peers when an action is considered suspect. When an agent considers an activity to be suspect, an agent with a higher level of specialization for the suspected intrusion is activated. Agents then report their findings to a centralized manager. The main drawbacks with these systems, is that the use of one or more centralized repositories leave at least some portion of the network exposed to malicious attacks including tampering and denial of service attacks. Even if an autonomous mobile decision-making agent was to detect a problem, interlocking mechanisms would be necessary to preclude any accidental or malicious removal, delay, or spoofing the agent. The Tethered Agent and Collective Hive (TACH) architecture includes a centralized Hive to keep track of agents and collected data and an Agent Registry (AR) to track fingerprints of agents (Lu, 2000). An Aglet-based framework for TACH incorporates mobile agents for virus detection and misuse detection (Kapoor, 2000). Limitations of TACH include the use of a centralized entity for agent control and a period communication protocol between agents with time-out detection used to detect status changes in the
A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 85
agents. If the centralized entity is disabled then the entire TACH system has been compromised. The DIDS (Mukherjee et al., 1994) used a combination of host and LAN monitors to observe system and network activity. A centralized director obtained information from the monitors to detect intrusions. The CIDF nomenclature mentioned above includes reconnaissance agents for data gathering, analysis agents, and decision-response agents (Staniford-Chen et al., 1998). The Computer Immunology Project at the University of New Mexico (Forrest et al., 1997) explored designs of IDSs based on ideas gleaned by examining animal immune systems. Small, individual agents would roam a distributed system, identify intrusions, and resolve the intrusions. One portion of the project developed a sense of self for security-related computer programs by observing the normal sets of system calls executed by the programs. This sense of self can be used to detect intrusions by discovering when a program executes an unusual set of system calls. The JAM Project at Columbia University (Stolfo et al., 1997) uses intelligent, distributed Java agents and data mining to learn models of fraud and intrusive behavior that can be shared between organizations. Helmar et al. propose lightweight agents for intrusion detection (Helmer et al., 2003). Their multi-agent system includes agents that travel between monitored systems in a network of distributed systems, obtain information from data cleaning agents, classify and correlate information, and report the information to a user interface and database via mediators. Agent systems with lightweight agent support allow runtime addition of new capabilities to agents. DeMara and Rocke (2004) propose an IDS based on mobile agents for detecting malicious activity by people with legitimate access to resources and services. These include attacks such as spoofing, termination, sidetracking, alteration of internal data, and selective deception. Their system employs techniques such as encapsulation, redundancy, scrambling, and mandatory obsolescence.
DIDS are simply a superset of the conventional IDS implemented in a distributed environment. Due to the distributed nature the implementation poses several challenges. IDS could be embedded inside agents and placed in the network to be monitored. The individual IDS may be configured to detect a single attack, or they may detect several types of attacks. Each network component may host one or many IDS. Since there will be a large number of flag generators (detection of an attack, event, etc.), these must be abstracted, analyzed, and condensed by a suitable architecture before arriving at a final conclusion. Typically there would be a centralized analyzing and control facility. The most popular architecture is of the master–slave type which may be suitable for small networks. In a hierarchical architecture analysis and control are done at different layers because of the geographical distribution or due to the size of the network. Attacks/event detection information is passed to analyzer/controller nodes that aggregate information from multiple IDS agents. It is to be noted that the event information, which is detected by the IDS agents will follow a bottom-up approach for analysis and the various command and control flow will follow a top-down approach. The physical location of IDS agents will be fixed since they monitor fixed network segments. In the case of hierarchical architecture, the analyzer/controller nodes may exist at many locations
86 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98
Fig. 1. Hierarchical architecture with free communication between layers.
in the network since they receive their input and give their output via network connections. Depending on the network environment the communication between the different layers could be implemented as depicted in Fig. 1 (Abraham and Thomas, 2005).
In the hierarchical architecture, the Central Analyzer and Controller (CAC) is the heart and soul of the DIDS. The CAC usually consists of a database and Web server, which allows interactive querying by the network administrators for attack information/analysis and initiate precautionary measures. CAC also performs attack aggregation, building statistics, identify attack patterns and perform rudimentary incident analysis. The co-operative intelligent agent network is one of the most important components of the DIDS. Ideally these agents will be located on separate network segments, and very often geographically separated. Communica¬tion among the agents is done utilizing TCP/IP sockets.
Agent modules running on the host machines are capable of data analysis and to formulate adequate response actions and are very often implemented as read only and fragile. In the event of tampering or modification the agent reports to the server agent and automatically ends its life. Agents residing in the individual analyzer/ controllers consist of modules responsible for agent regeneration, dispatch, updating and maintaining intrusion signatures and so on. These agents control the individual IDS agents for monitoring the network, manage all the communication and life cycle of the IDS agents and also updates the IDS agents with detection algorithms, response and trace mechanisms.
3. Soft computing
Soft computing was first proposed by Zadeh (1998), to construct new generation computationally intelligent hybrid systems consisting of neural networks, fuzzy inference system, approximate reasoning and derivative free optimization
A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 87
techniques. It is well known that intelligent systems, which can provide human like expertise such as domain knowledge, uncertain reasoning, and adaptation to a noisy and time-varying environment, are important in tackling practical computing problems. In contrast with conventional Artificial Intelligence (AI) techniques which only deal with precision, certainty and rigor the guiding principle of hybrid systems is to exploit the tolerance for imprecision, uncertainty, low solution cost, robustness, partial truth to achieve tractability, and better rapport with reality.
3.1. Fuzzy rule-based systems
Fuzzy logic has proved to be a powerful tool for decision making to handle and manipulate imprecise and noisy data. The notion central to fuzzy systems is that truth values (in fuzzy logic) or membership values (in fuzzy sets) are indicated by a value in the range [0.0, 1.0], with 0.0 representing absolute falseness and 1.0 representing absolute truth. A fuzzy system is characterized by a set of linguistic statements based on expert knowledge. The expert knowledge is usually in the form of if-then rules.
Definition 1. Let X be some set of objects, with elements noted as x. Thus, X = {x].
Definition 2. A fuzzy set A in X is characterized by a membership function which are easily implemented by fuzzy conditional statements. In the case of fuzzy statement if the antecedent is true to some degree of membership then the consequent is also true to that same degree.
A simple rule structure: If antecedent then consequent.
A simple rule: If variable1 is low and variable2 is high then output is benign else output is malignant.
In a fuzzy classification system, a case or an object can be classified by applying a set of fuzzy rules based on the linguistic values of its attributes. Every rule has a weight, which is a number between 0 and 1 and this is applied to the number given by the antecedent. It involves 2 distinct parts. First the antecedent is evaluated, which involves fuzzifying the input and applying any necessary fuzzy operators and second applying that result to the consequent known as inference. To build a fuzzy classification system, the most difficult task is to find a set of fuzzy rules pertaining to the specific classification problem. We explored three fuzzy rule generation methods for intrusion detection systems. Let us assume that we have a n-dimensional c-class pattern classification problem whose pattern space is an n-dimensional unit cube [0, 1]n. We also assume that m patterns xp = (xpl, ... , xpn), p = 1, 2,. .. , m, are given for generating fuzzy if-then rules where xp E [0, 1] for p = 1, 2, .. . , m.
3.1.1. Rule generation based on the histogram of attribute values (FR1)
In this method, use of the histogram itself is an antecedent membership function. Each attribute is partitioned into 20 membership functions fh(•), h = 1, 2, ... , 20. The smoothed histogram mki (xi)of class k patterns for the ith attribute is calculated using
88 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 the 20 membership functions fh(•) as follows:
Emki(xi) = mk 1
xpEclass k
forbh-1pxipbh, h = 1, 2, ..., 20, (1)
where mk is the number of class k patterns, bh-1,bh
[ ] is the hth crisp interval
corresponding to the 0.5-level set of the membership function fh(•):
b1 = 0, b20 = 1, (2)
I I
1 1
bh = 20 - 1 h - for h = 1, 2, ..., 19. (3)
2
The smoothed histogram in (1) is normalized so that its maximum value is 1. A single fuzzy if-then rule is generated for each class. The fuzzy if-then rule for the kth class can be written as
If x1 is Ak1and ... and xn then class k, (4)
where Akiis an antecedent fuzzy set for the ith attribute. The membership function of Aki is specified as
Aki (xi) = exp - (xi - mki )2, (5)
2(sk i )2
where mki is the mean of the ith attribute values xpi of class k patterns, and ski is the standard deviation. Fuzzy if-then rules for the two-dimensional two-class pattern classification problem are written as follows:
If x3 is A13 and x4 is A14 then class 2, (6)
If x3 is A23 and x4 is A2 4 then class 3. (7)
Membership function of each antecedent fuzzy set is specified by the mean and the standard deviation of attribute values. For a new pattern xp = (xp3, xp4), the winner rule is determined as follows:
{
A 3(xp3).A 2(xp4) = max Ak 1(xp3).Ak 2(xp4)Ik = 1,21. (8)
3.1.2. Rule generation based on partition of overlapping areas (FR2)
Fig. 2 demonstrates a simple fuzzy partition, where the two-dimensional pattern space is partitioned into 25 fuzzy subspaces by five fuzzy sets for each attribute (S: small, MS: medium small, M: medium, ML: medium large, L: large). A single fuzzy if-then rule is generated for each fuzzy subspace. One disadvantage of this approach is that the number of possible fuzzy if-then rules exponentially increases with the dimensionality of the pattern space. Because the specification of each membership function does not depend on any information about training patterns, this approach uses fuzzy if-then rules with certainty grades. The local information about training patterns in the
A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 89
Fig. 2. An example of fuzzy partition.
corresponding fuzzy subspace is used for determining the consequent class and the grade of certainty. In this approach, fuzzy if-then rules of the following type are used:
If x1 is Aj1 and ... and xn then class Cj
with CF = CFj, j = 1, 2,. .. , N, (9)
where j indexes the number of rules, N is the total number of rules, Aji the antecedent fuzzy set of the ith rule for the ith attribute, Cj; is the consequent class, and CFj is the grade of certainty. The consequent class and the grade of certainty of each rule are determined by the following simple heuristic procedure:
Step 1: Calculate the compatibility of each training pattern xp = (xp1, xp2, ... , xpn) with the jth fuzzy if-then rule by the following product operation:
pj(xp) = Aj1(xp1) x ... x Ajn(xpn), p = 1,2,. .., m. (10)
Step 2: For each class, calculate the sum of the compatibility grades of the training patterns with the jth fuzzy if-then rule Rj:
n
~bclass k(Rj) = p xp
( ), k = 1,2, ..., c, (11)
xpEclass k
where bclass k(Rj) the sum of the compatibility grades of the training patterns in class
k with the jth fuzzy if-then rule Rj.
Step 3: Find class A'j that has the maximum valuebclass k(Rj):
bclass kj' = Maxfbclass 1(Rj), . . . , bclass c(Rj)}. (12)
If two or more classes take the maximum value or no training pattern compatible with the jth fuzzy if-then rule (i.e., if bclass k(Rj) = 0 for k = 1, 2,. . . , c), the consequent class Ci can not be determined uniquely. In this case, let Ci be f.
Step 4: If the consequent class Ci is 0, let the grade of certainty CFj be CFj = 0. Otherwise the grade of certainty CFj is determined as follows:
E ( )
bclass k Rj
k=1
90 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98
Fig. 3. Fuzzy partition of each attribute: (a) simple fuzzy grid approach; (b) modified fuzzy grid approach.
where
b¯ =
k=1
kak'
j bclass k(Rj)
(c - 1) ,
The above approach could be modified by partitioning only the overlapping areas as illustrated in Fig. 3.
This approach generates fuzzy if-then rules in the same manner as the simple fuzzy grid approach except for the specification of each membership function. Because this approach utilizes the information about training patterns for specifying each membership function as mentioned in Section 3.1, the performance of generated fuzzy if-then rules is good even when we do not use the certainty grade of each rule in the classification phase. In this approach, the effect of introducing the certainty grade to each rule is not so important when compared to conventional grid partitioning.
3.2. Neural learning of fuzzy rules (FR3)
In a fused neuro-fuzzy architecture, neural network learning algorithms are used to determine the parameters of fuzzy inference system (membership functions and number of rules). An Evolving Fuzzy Neural Network implements a Mamdani-type FIS and all nodes are created during learning. Each input variable is represented here by a group of spatially arranged neurons to represent a fuzzy quantization of this variable. New neurons can evolve in this layer if, for a given input vector, the corresponding variable value does not belong to any of the existing MF to a degree greater than a membership threshold. Technical details of the learning algorithm are given in Kasabov (1998).
4. Experimental setup and results
Complex relationships exist between features, which are difficult for humans to discover. The IDS must therefore reduce the amount of data to be processed. This is very important if real-time detection is desired. The easiest way to do this is by doing an intelligent input feature selection. Certain features may contain false correlations,
A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 91
which hinder the process of detecting intrusions. Further, some features may be redundant since the information they add is contained in other features. Extra features can increase computation time, and can impact the accuracy of IDS. Feature selection improves classification by searching for the subset of features, which best classifies the training data (Chebrolu et al., 2005).
Feature selection is done based on the contribution the input variables made to the construction of the decision tree. Feature importance is determined by the role of each input variable either as a main splitter or as a surrogate. Surrogate splitters are defined as back-up rules that closely mimic the action of primary splitting rules. Suppose that, in a given model, the algorithm splits data according to variable ‘protocol_type’ and if a value for ‘protocol_type’ is not available, the algorithm might substitute ‘flag’ as a good surrogate. Variable importance, for a particular variable is the sum across all nodes in the tree of the improvement scores that the predictor has when it acts as a primary or surrogate (but not competitor) splitter. Example, for node i, if the predictor appears as the primary splitter then its contribution towards importance could be given as iimportance. But if the variable appears as the nth surrogate instead of the primary variable, then the importance becomes iimportance 1/4 (pn)*iimprovement in which p is the ‘surrogate improvement weight’ which is a user controlled parameter set between (0–1) (Shah et al., 2004).
The data for our experiments was prepared by the 1998 DARPA intrusion detection evaluation program by MIT Lincoln Labs MIT. The LAN was operated in a real environment, but was subjected to multiple attacks. For each TCP/IP connection, 41 various quantitative and qualitative features were extracted. The data set has 41 attributes for each connection record plus one class label as given in Table 1. The data set contains 24 attack types that could be classified into four main categories.
DoS: Denial of service
Denial of service (DoS) is a class of attack where an attacker makes a computing or memory resource too busy or too full to handle legitimate requests, thus denying legitimate users access to a machine.
R2L: unauthorized access from a remote machine
A remote to user (R2L) attack is a class of attack where an attacker sends packets to a machine over a network, then exploits the machine’s vulnerability to illegally gain local access as a user.
U2Su: unauthorized access to local super user (root)
User to root (U2Su) exploits are a class of attacks where an attacker starts out with access to a normal user account on the system and is able to exploit vulnerability to gain root access to the system.
Probing: surveillance and other probing
Probing is a class of attack where an attacker scans a network to gather information or find known vulnerabilities. An attacker with a map of machines and services that are available on a network can use the information to look for exploits.
Our experiments have three phases namely input feature reduction, training phase and testing phase. In the data reduction phase, important variables for real-time intrusion detection are selected by feature selection.
92 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98
Table 1
Variables for intrusion detection data set
Variable no.
Variable name Variable type Variable label
1 duration Continuous A
2 protocol_type Discrete B
3 service Discrete C
4 flag Discrete D
5 src_bytes Continuous E
6 dst_bytes Continuous F
7 land Discrete G
8 wrong_fragment Continuous H
9 urgent Continuous I
10 hot Continuous J
11 num_failed_logins Continuous K
12 logged_in Discrete L
13 num_compromised Continuous M
14 root_shell Continuous N
15 su_attempted Continuous O
16 num_root Continuous P
17 num_file_creations Continuous Q
18 num_shells Continuous R
19 num_access_files Continuous S
20 num_outbound_cmds Continuous T
21 is_host_login Discrete U
22 is_guest_login Discrete V
23 count Continuous W
24 srv_count Continuous X
25 serror_rate Continuous Y
26 srv_serror_rate Continuous X
27 rerror_rate Continuous AA
28 srv_rerror_rate Continuous AB
29 same_srv_rate Continuous AC
30 diff_srv_rate Continuous AD
31 srv_diff_host_rate Continuous AE
32 dst_host_count Continuous AF
33 dst_host_srv_count Continuous AG
34 dst_host_same_srv_rate Continuous AH
35 dst_host_diff_srv_rate Continuous AI
36 dst_host_same_src_port_rate Continuous AJ
37 dst_host_srv_diff_host_rate Continuous AK
38 dst_host_serror_rate Continuous AL
39 dst_host_srv_serror_rate Continuous AM
40 dst_host_rerror_rate Continuous AN
41 dst_host_srv_rerror_rate Continuous AO
In the training phase, the different soft computing models were constructed using the training data to give maximum generalization accuracy on the unseen data. The test data is then passed through the saved trained model to detect intrusions in the testing phase. The 41 features are labeled as shown in Table 1 and the class label is named as AP.
A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 93
This data set has five different classes namely Normal, DoS, R2L, U2R and Probes. The training and test comprises of 5092 and 6890 records, respectively (KDD Cup, 1999). Using all 41 variables could result in a big IDS model, which would result in substantial overhead for online detection. All the training data were scaled to (0–1). The decision tree approach described before resulted in to reducing the number of variables to 12 significant variables or features. The list of reduced variables is shown in Table 2.
Using the original and reduced data sets, we performed a 5-class classification. The normal data belongs to class 1, probe belongs to class 2, denial of service belongs to class 3, user to super user belongs to class 4, remote to local belongs to class 5. All the IDS models are trained and tested with the same set of data.
We examined the performance of all three fuzzy rule-based approaches (FR1, FR2 and FR3) mentioned in Section 3. When an attack is correctly classified the grade of certainty is increased and when an attack is misclassified the grade of certainty is decreased. A learning procedure is used to determine the grade of certainty. Triangular membership functions were used for all the fuzzy rule based classifiers. For FR1 and FR2 two triangular membership functions were assigned and 2 and 212 rules were learned respectively for the reduced data set. For FR3, 4 triangular membership functions were used for each input variable. A sensitivity threshold Sthr = 0.95 and error threshold Errthr = 0.05 was used for all the classes and 89 rule nodes were developed during the one pass learning. For comparison purposes various other empirical results were adapted from Mukkamala et al. (2005), Peddabachigari et al. (2004), Shah et al. (2004), Chebrolu et al. (2005) and Abraham (2004).
The settings of various linear genetic programming system parameters are of utmost importance for successful performance of the system. The population size was set at 120,000 and a tournament size of 8 is used for all the 5 classes. Crossover and mutation probability is set at 65–75% and 75–86%, respectively for the different classes (Abraham, 2004). Our trial experiments with SVM revealed that the polynomial kernel option often performs well on most of the datasets. We also constructed decision trees using the training data and then testing data was passed through the constructed classifier to classify the attacks (Mukkamala et al., 2005).
A number of observations and conclusions are drawn from the results illustrated in Tables 3 and 4. Using 41 attributes, the FR2 method gave 100% accuracy for all the 5 classes, showing the importance of fuzzy inference systems. For the full data set, LGP outperformed decision trees and support vector machines in terms of detection accuracies (except for U2R class).
Using 12 attributes most of the classifiers performed very well except the fuzzy classifiers (FR1, FR2). For detecting U2R attacks FR2 gave the best accuracy. However, due to the tremendous reduction in the number of attributes (about 70%
Table 2
Reduced variable set
C, E, F, L, W, X, Y, AB, AE, AF, AG, AI
94 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98
Table 3
Performance comparison using full data set
Attack type
Classification accuracy on test data set (%)
FR1 FR2 FR3 DT SVM LGP
Normal 40.44 100.00 98.26 99.64 99.64 99.73
Probe 53.06 100.00 99.21 99.86 98.57 99.89
DOS 60.99 100.00 98.18 96.83 99.92 99.95
U2R 66.75 100.00 61.58 68.00 40.00 64.00
R2L 61.10 100.00 95.46 84.19 33.92 99.47
Table 4
Performance comparison using reduced data set
Attack type
Classification accuracy on test data set (%)
FR1 FR2 FR3 DT SVM LGP
Normal 74.82 79.68 99.56 100.00 99.75 99.97
Probe 45.36 89.84 99.88 97.71 98.20 99.93
DOS 60.99 60.99 98.99 85.34 98.89 99.96
U2R 94.11 99.64 65.00 64.00 59.00 68.26
R2L 91.83 91.83 97.26 95.56 56.00 99.98
less), we are able to design a computational efficient (lightweight) IDS. Since a particular classifier could not provide accurate results for all the classes, we propose to use a combination of different classifiers to detect different attacks. The D-SCIDS architecture using 41 attributes (heavy weight) and 12 attributes (lightweight) are depicted in Fig. 4. The proposed heavy weight model could detect with 100% accuracy while the lightweight model could detect all the attacks with high accuracy (lowest accuracy being 94.11% for U2R class using FR1). It is to be noted that FR1 classifier is preferred for the lightweight D-SCIDS (due to fewer number of rules) even though it has slightly lower accuracy when compared to FR2 (Table 5).
In some classes the accuracy figures tend to be very small and may not be statistically significant, especially in view of the fact that the 5 classes of patterns differ in their sizes tremendously. For example only 27 data sets were available for training the U2R class. More definitive conclusions can only be made after analyzing more comprehensive sets of network traffic.
5. Conclusions
Effective intrusion detection and management systems are critical components of cyber infrastructure as they are in the forefront of the battle against cyber-terrorism. In this paper, we presented a framework for Distributed Intrusion Detection Systems (DIDS) using several soft computing paradigms. We also demonstrated the
A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 95
Fig. 4. Light/heavy-weight SCIDS architecture.
Table 5
Performance of the light weight DSCIDS
Attack type Classification accuracy on test data (%)
Normal 100.00
Probe 99.93
DOS 99.96
U2R 94.11
R2L 99.98
importance of feature reduction to model lightweight intrusion detection systems. Finally, we propose a hybrid architecture involving ensemble and base classifiers for intrusion detection.
For real time intrusion detection systems, LGP would be the ideal candidate as it can be manipulated at the machine code level. Overall, the fuzzy classifier (FR2) gave
96 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98
100% accuracy for all attack types using all the 41 attributes. The proposed hybrid combination of classifiers requires only 12 input variables. While the lightweight SCIDS would be useful for MANET/distributed systems, the heavy weight SCIDS would be ideal for conventional static networks, wireless base stations etc. More data mining techniques are to be investigated for attribute reduction and enhance the performance of other soft computing paradigms.
With the increasing incidents of cyber attacks, building an effective intrusion detection models with good accuracy and real-time performance are essential. This field is developing continuously. More data mining techniques should be investigated and their efficiency should be evaluated as intrusion detection models.
Acknowledgments
This research was supported by the International Joint Research Grant of the Institute of Information Technology Assessment (IITA) foreign professor invitation program of the Ministry of Information and Communication (MIC), Korea.
References
Abraham A. Evolutionary computation in intelligent web management, evolutionary computing in data mining. In: Ghosh A, Lakhmi J, editors. Studies in fuzziness and soft computing. Germany: Springer; 2004. p. 189–210 [Chapter 8].
Abraham A, Thomas J. Distributed intrusion detection systems: a computational intelligence approach. In: Abbass HA, Essam D, editors. Applications of information systems to homeland security and defense. USA: Idea Group Inc. Publishers; 2005. p. 105–35 [Chapter 5].
Abraham A, Jain R, Sanyal S, Han SY. SCIDS: a soft computing intrusion detection system. Sixth international workshop on distributed computing (IWDC 2004). Lecture Notes in Computer Science, vol. 3326. Germany: Springer; 2004. p. 252–7. ISBN: 3-540-24076-4.
Barbara D, Couto J, Jajodia S, Wu N. ADAM: a testbed for exploring the use of data mining in intrusion detection. SIGMOD Rec 2001;30(4):15–24.
Bernardes MC, dos Santos Moreira E. Implementation of an intrusion detection system based on mobile agents. In: Proceedings of the international symposium on software engineering for parallel and distributed systems, 2000. p. 158–64.
Brieman L, Friedman J, Olshen R, Stone C. Classification of regression trees. Wadsworth Inc.; 1984. Chebrolu S, Abraham A, Thomas J. Feature deduction and ensemble design of intrusion detection systems, computers and security, vol. 24/4. Amsterdam: Elsevier; 2005. p. 295–307.
Cohen W. Learning trees and rules with set-valued features. American Association for Artificial Intelligence (AAAI); 1996.
DeMara RF, Rocke AJ. Mitigation of network tampering using dynamic dispatch of mobile agents. Comput Security 2004;23:31–42.
Denning D. An intrusion-detection model. IEEE Trans Software Eng 1987;SE-13(2):222–32.
Feiertag R, Rho S, Benzingher L, Wu S, Redmond T, Zhang C, Levitt K, Peticolas D, Heckman M, Staniford S, McAlerney J. Intrusion detection inter-component adaptive negotiation. Comput Networks 2000;34:605–21.
Forrest S, Hofmeyr SA, Somayaji A. Computer immunology. CACM 1997;40(10):88–96.
Helmer G, Wong J, Honavar V, Miller L. Intelligent agents for intrusion detection. Available from http:// citeseer.nj.nec.com/helmer98intelligent.html, 1998.
A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98 97
Helmer G, Wong JSK, Honavar V, Miller L, Wang Y. Lightweight agents for intrusion detection. J Systems Software 2003;67:109–22.
Jansen W, Mell P, Karygiannis T, Marks D. Applying mobile agents to intrusion detection and response. National Institute of Standards and Technology, Computer Security Division, 1999. Available from: http://csrc.nist.gov/publications/nistir/ir6416.pdf.
Jou YF, Gong F, Sargor C, Wu X, Wu SF, Chang HC, Wang F. Design and implementation of a scalable intrusion detection system for the protection of network infrastructure. DARPA information survivability conference and exposition, 2000.
Kapoor B. Remote misuse detection system using mobile agents and relational database query techniques. Master’s thesis, University of Central Florida, 2000.
Kasabov N. Evolving fuzzy neural networks—algorithms, applications and biological motivation. In: Yamakawa T, Matsumoto G, editors. Methodologies for the conception, design and application of soft computing. Singapore: World Scientific; 1998. p. 271–4.
KDD Cup, 1999. Intrusion detection data set: <http://kdd.ics.uci.edu/databases/kddcup99/kddcup.da-ta_10_percent.gzS
Kemmerer RA. NSTAT: a model-based real-time network intrusion detection system. Technical Report TRCS97-18, Reliable Software Group, Department of Computer Science, University of California at Santa Barbara, 1997.
Kruegel C, Toth T. Applying mobile agent technology to intrusion detection. Proceedings of the ICSE workshop on software engineering and mobility, 2001. Available from: http://citeseer.nj.nec.com/ kr01applying.html.
Lee W, Stolfo S, Mok K. A Data Mining Framework for Building Intrusion Detection Models. In: Proceedings of the IEEE symposium on security and privacy, 1999.
Lee W, Nimbalker RA, Yee KK, Patil SB, Desai PH, Tran PP, Stolfo SJ. A data mining and CIDF based approach for detecting novel and distributed intrusions, Proceeings of the third international workshop on recent advances in intrusion detection, 2000.
Lu J. Mobile agent protocols for distributed detection of network intrusions. Master’s thesis, University of Central Florida, 2000.
MIT Lincoln Laboratory. <http://www.ll.mit.edu/IST/ideval/S
Mouinji A, Charlier BL, Zampunieris D, Habra N. Distributed audit trail analysis. Proceedings of the ISOC 95 symposium on network and distributed system security, 1995. p. 102–12.
Mukherjee B, Todd Heberlein L, Levitt KN. Network intrusion detection. IEEE Network 1994;8(3):26–41.
Mukkamala S, Sung A, Abraham A. Intrusion detection using ensemble of soft computing and hard computing paradigms. J Network Comput Appl 2005;28(2):167–82.
Ning P, Jajodia S, Wang XS. Design and implementation of a decentralized prototype system for detecting distributed attacks. Comput Commun 2002;25:1374–91.
Peddabachigari S, Abraham A, Thomas J. Intrusion detection systems using decision trees and support vector machines. Int J Appl Sci Comput USA 2004;11(3):118–34.
Porras PA, Neumann PG. EMERALD: event monitoring enabling response to anomalous live disturbances. Proceedings of the 20th national information security conference, NIST, 1997.
Shah K, Dave N, Chavan S, Mukherjee S, Abraham A, Sanyal S. Adaptive neuro-fuzzy intrusion detection system. IEEE international conference on information technology: coding and computing (ITCC’04), vol. 1, USA. Silver Spring, MD: IEEE Computer Society; 2004. p. 70–4.
Snapp SR, Bretano J, Diaz GV, Goan TL, Heberlain LT, Ho C, Levitt KN, Mukherjee B, Smaha SE, Grance T, Teal DM, Mansur D. DIDS (Distributed Intrusion Detection System)—motivation architecture and an early prototype. In: Proceedings of the 14th national computer security conference, Washington, DC, October, 1999. p. 167–76.
Spafford EH, Zamboni D. Intrusion detection using autonomous agents. Comput Networks 2000;34:547–70.
Staniford-Chen S, Cheung S, Crawford R, Dilger M, Frank J, Hoagland J, Levitt K, Wee C, Yipi R, Erkle DZ. GriDS—a large scale intrusion detection system for large networks. Proceedings of the 19th national information security conference 1996;1:361–70.
98 A. Abraham et al. / Journal of Network and Computer Applications 30 (2007) 81–98
Staniford-Chen S, Tung SB, Schnackenberg D. The common intrusion detection framework (CIDF). Proceedings of the information survivability workshop, Orlando FL, October 1998.
Stolfo SJ, Prodromidis AL, Tselepis S, Lee W, Fan D, Chan PK. JAM: Java agents for meta-learning over distributed databases. In: Proceedings of the third international conference on knowledge discovery and data mining, Newport Beach, CA, USA, 1997. p. 74–81.
Summers RC. Secure computing: threats and safeguards. New York: McGraw Hill; 1997.
Vapnik VN. The nature of statistical learning theory. Berlin: Springer; 1995.
Vigna G, Kemmerer RA. NetSTAT: a network-based intrusion detection system. J Comput Security 1999;7(1):37–71.
White GB, Fisch EA, Wpooch U. Cooperating security managers: a peer-based intrusion detection system. IEEE Networks 1996.
Zadeh LA. Roles of soft computing and fuzzy logic in the conception, design and deployment of Information/Intelligent Systems. In: Kaynak O, Zadeh LA, Turksen B, Rudas IJ, editors. Computational intelligence: soft computing and fuzzy-neuro integration with applications, 1998. p. 1–9.
Tidak ada komentar:
Posting Komentar