Sunday, August 19, 2007
SERVICE-ORIENTED BUSINESS: SOA = BUSINESS-ORIENTED ARCHITECTURE
Although many organizations are seriously considering SOA, some are doing it from a pure IT perspective while others are really looking at a new way of running their businesses. They are pursuing the concept of service-oriented business, or the idea of running all aspects of their organization from a "services" perspective. This has broad ramifications for an organization.
IT is a major budget item in many firms, especially for financial services organizations. IT affects all aspects of these organizations in profound ways. IT affects the cost of everyday business transactions, which over time equates to billions in efficiencies. IT allows interdisciplinary collaboration across a business enterprise, which means better cooperation, better sharing of information, and a more united capability to compete in your markets. IT improves many diverse business processes and increases productivity across the organization. IT amplifies the productivity and efficiency of all business processes, both internally as well as those that are exposed to trading partners and customers.
Business Impact of SOA
The potential business impact of SOA is significant to an organization, but only if the appropriate business modeling and business context are considered during the planning and implementation of SOA. The range of business benefits is broad, and includes both specific and tangible benefits as well as less tangible yet more compelling benefits. Of these, one of the most interesting concepts is the notion of the SOA Network Effect. This concept was first developed in a ComputerWorld column to describe the importance of the soft issues relating to achieving service-oriented architectures, such as cultural issues, behavioral issues, and organizational dynamics. [2] Are these as important as the technology of SOA? The answer from the field is that these soft issues are more important than the technology, and thus they are not all that "soft" as we would suppose. These soft issues are among the most difficult aspects of achieving SOA. Yet they are the key to an SOA's ultimate success or demise.
Characteristics of a Service-Oriented Business
A service-oriented business is an organization that has progressed with its SOA efforts such that its business really does operate using an SOA. SOA as an IT architectural strategy actually uses services as the "operating system" for the business and its business processes.
What if your CEO could acquire another firm and integrate its information and operations into the existing business and IT architecture without integration challenges? What if the acquired firm's business processes could integrate seamlessly with yours with zero latency? What if there was no integration effort required at all? What if the IT systems were "preintegrated," meaning that they could exchange information with one another without any incremental integration expenses and effort? What if business processes could be more quickly combined into new composite processes that represent the combined business entity better than those of the two individual business processes?
Imagine how business would be without two factors that have become ingrained in our expectations of IT today: the IT integration hurdles that attend every business initiative and the inevitable time lag between the need for the business initiative and the ability of the IT organization to deliver it on time and, more important, with zero time-to-market latency. This is the zero integration business enterprise.
How would this enhance the business case for a mergers-and-acquisition strategy? How would this improve the return on investment equation for any IT or business initiative? What is the value of faster time to market for any information-based initiative? That's the business case for a service-oriented business.
Exhibit 1.9 is a vision of how SOA becomes the foundation for a service-oriented business. It depicts how SOA can enable significant strategic business benefits through many tangible contributions. These include IT benefits, such as reuse and asset optimization that are often initial target benefits of most SOA efforts. However, as we progress from the IT level to business processes, organization and structure, and ultimately the strategic level where business and financial goals are critical, SOA has value to offer there as well.
Exhibit 1.9: Service-Oriented Business
CHARACTERISTICS OF A ZERO-INTEGRATION ENTERPRISE
Implementing SOA will allow tremendous reduction in integration expense and maintenance, such that we go so far as to call it the "zero-integration enterprise." A zero-integration enterprise is an organization whose business and IT teams are committed to the precepts of SOA. These organizations see the business value and strategic advantages of SOA, and are migrating their organization, processes, applications, and skills to support the concept of services, specifically Web services. This organization has eliminated traditional integration models in favor of SOA and standards-based integration. Some characteristics of a zero-integration enterprise follow.
This organization can launch new business initiatives faster than its competitors because less IT integration is required to support various business projects. This time-to-market benefit allows faster response to business conditions, customer requirements, competitive threats, and increased innovation.
This organization has a higher return on assets and greater IT productivity than its peer companies due to the asset-related benefits of SOA, such as software component reuse, services reuse, and extending the capabilities of existing IT systems and infrastructure.
This organization launches new software applications 35% faster than before with reduced quality assurance and testing effort due to component and services reuse. Building on proven software and services capabilities, the organization spends less time developing new code and more time focusing on business process issues.
This organization uses the 30% of its IT budget previously used for integration projects to solve strategic business problems, such as improving customer satisfaction, reducing time to market for new products, and increasing sales through various IT initiatives.
This organization implements concepts of agility and flexibility through its SOA initiatives. Agility is reality, and is measured by clear unambiguous metrics.
This organization has an agile business model that can quickly respond to business challenges, competitive threats, and customer needs.
This organization has greater customer focus deriving from reduced effort spent on internal integration and more effort spent on customer satisfaction, partner communication, and efficient business processes.
This organization has a business-focused IT organization that no longer must concern itself with assuring interoperability issues but rather can focus on forward-looking strategic issues.
This organization has a flexible IT architecture, based on SOA and services, that facilitates superior business performance, enables world-class business processes, and is highly efficient with all corporate resources.
This organization integrates without integrating, both internally and externally with customers and partners. This organization does not integrate, it service-enables instead.
WHAT ARE THE CHALLENGES OF SOA?
Now, with all of these benefits, where's the catch? Simple. SOA is difficult to implement, manage, and control. Not because of the technology, mind you, but due to the organizational, cultural, and behavioral aspects of SOA that contribute to success. And technically, although there has been great progress with regard to the standards, supporting tools, and development and run-time platforms, there are still issues to be resolved. These issues include support for long-running transactions, security concerns, and many others. However, the organizational, cultural, and governance issues far outweigh the technical aspects of implementing SOA. That's not a reason to avoid trying to achieve SOA, but it certainly is a reason to pay attention to many of the softer aspects of technology initiatives to ensure you can reap the rewards. A discussion of major SOA challenges organizations will face as they migrate to SOA follows.
Enterprise Architecture Model May Need Tuning for SOA
As many organizations consider their SOA approach, they will realize that the organization, processes, and disciplines of their enterprise architecture organization may require tuning to suit the requirements of SOA. Often the process of enterprise architecture is somewhat flawed, which helps explain the current state of IT architectures today: rigid IT architectures characterized by heavy carryover legacy systems, inflexible "digital concrete" of enterprise applications, and a portfolio of applications that demand integration software to make them work together.
Perhaps architects have been too focused on building things and getting them to work as opposed to building things that are flexible, reusable, and support the business: the things that are now most important for businesses today as they grapple with change and global forces. These are the new requirements of SOA.
SOA will fail unless the process of architecture is changed from one of static advice, creation of presentations, application blueprints, and architecture road maps to one of actively shaping and implementing flexible and reusable IT assets that support business processes. In other words, SOAs. A model for tuning the enterprise architecture process is presented in Chapter 8.
SOA Is Spatially and Temporally Distributed
One of the challenges of SOA is that it is not implemented all at once. Rather, it is achieved through many discrete projects across both space and time. This temporal and spatial distribution of SOA projects makes governance all the more critical to SOA success. SOA governance and enforceable policies are the keys to managing conformance to the SOA across geographic and time horizons.
SOA Is Organizationally Complex and Behaviorally Challenging
SOA is a complex goal to achieve. It is organizationally, behaviorally, and culturally challenging for most organizations. We describe an SOA behavioral model in Chapter 7 to help you anticipate and create the behavioral pattern your SOA will require.
SOA Requires Governance to Achieve and Manage
SOA requires a robust SOA governance model, clear and enforceable policies, and a way to implement SOA governance across all the life-cycle processes of an organization: enterprise architecture, services design, publishing, discovery, and run-time. SOA governance is essential to realize the ultimate business value of SOA.
SOA Is All About Services
Implementing SOA requires new approaches to identifying, modeling, and implementing reusable, interoperable services. Repeatable processes for determining appropriate granularity, version management, and enforcing design-time policies for services are essential. In fact, even the definition of services is important for many organizations.
The fundamental architectural unit of an SOA is a "service." Services, per our definition of SOA, are units of business capabilities, processes, or functions that are delivered in a repeatable way to consumers of those services. Consumers of services in an SOA can be developers, architects, and analysts, or they can be external customers, business partners, and internal business customers.
Without services, there is no SOA, and an SOA with services is useless unless there is actual consumption of the services that are available. Therefore, an organization can deliver or realize no SOA value unless the services in an SOA have real consumers using them for business reasons.
Services Identification, Modeling, and Design Challenges
Services are critical to an SOA. However, the process of identifying the right services for your organization is challenging. And what makes these services the "right" ones? This discussion is further complicated by questions about granularity, reuse, and other related services modeling and design issues. Chapters 3, 4, and 5 address these concerns using some innovative services modeling concepts. These chapters will help you achieve the "SOA Three Rights": Identify the "right" services; build those services the "right" way; and run them on the "right" enabling technology stack.
Where Are We Headed?
SOA is an iterative business approach. There is no single correct path to achieving SOA. Instead, there are multiple routes to the SOA goal. It is important to recognize the fundamentally iterative approach that SOA requires to achieve its stated goals objectives and business results over time. This approach, which we call an SOA business iteration model, is depicted in Exhibit 1.10.
Exhibit 1.10: SOA Business Iteration Model
This model builds explicit business context into the SOA strategy and planning process, recognizing that SOA must be aligned to business and IT objectives as well as to the current urgencies of the organization at that particular moment in time.
Remember: SOA is a lifestyle change. It is a long-term commitment to achieving specific business objectives. That is why we wrote this book, and that presumably is why you are reading it: to learn how to plan, design, and implement SOA via reusable services to achieve clear business results.
IT is a major budget item in many firms, especially for financial services organizations. IT affects all aspects of these organizations in profound ways. IT affects the cost of everyday business transactions, which over time equates to billions in efficiencies. IT allows interdisciplinary collaboration across a business enterprise, which means better cooperation, better sharing of information, and a more united capability to compete in your markets. IT improves many diverse business processes and increases productivity across the organization. IT amplifies the productivity and efficiency of all business processes, both internally as well as those that are exposed to trading partners and customers.
Business Impact of SOA
The potential business impact of SOA is significant to an organization, but only if the appropriate business modeling and business context are considered during the planning and implementation of SOA. The range of business benefits is broad, and includes both specific and tangible benefits as well as less tangible yet more compelling benefits. Of these, one of the most interesting concepts is the notion of the SOA Network Effect. This concept was first developed in a ComputerWorld column to describe the importance of the soft issues relating to achieving service-oriented architectures, such as cultural issues, behavioral issues, and organizational dynamics. [2] Are these as important as the technology of SOA? The answer from the field is that these soft issues are more important than the technology, and thus they are not all that "soft" as we would suppose. These soft issues are among the most difficult aspects of achieving SOA. Yet they are the key to an SOA's ultimate success or demise.
Characteristics of a Service-Oriented Business
A service-oriented business is an organization that has progressed with its SOA efforts such that its business really does operate using an SOA. SOA as an IT architectural strategy actually uses services as the "operating system" for the business and its business processes.
What if your CEO could acquire another firm and integrate its information and operations into the existing business and IT architecture without integration challenges? What if the acquired firm's business processes could integrate seamlessly with yours with zero latency? What if there was no integration effort required at all? What if the IT systems were "preintegrated," meaning that they could exchange information with one another without any incremental integration expenses and effort? What if business processes could be more quickly combined into new composite processes that represent the combined business entity better than those of the two individual business processes?
Imagine how business would be without two factors that have become ingrained in our expectations of IT today: the IT integration hurdles that attend every business initiative and the inevitable time lag between the need for the business initiative and the ability of the IT organization to deliver it on time and, more important, with zero time-to-market latency. This is the zero integration business enterprise.
How would this enhance the business case for a mergers-and-acquisition strategy? How would this improve the return on investment equation for any IT or business initiative? What is the value of faster time to market for any information-based initiative? That's the business case for a service-oriented business.
Exhibit 1.9 is a vision of how SOA becomes the foundation for a service-oriented business. It depicts how SOA can enable significant strategic business benefits through many tangible contributions. These include IT benefits, such as reuse and asset optimization that are often initial target benefits of most SOA efforts. However, as we progress from the IT level to business processes, organization and structure, and ultimately the strategic level where business and financial goals are critical, SOA has value to offer there as well.
Exhibit 1.9: Service-Oriented Business
CHARACTERISTICS OF A ZERO-INTEGRATION ENTERPRISE
Implementing SOA will allow tremendous reduction in integration expense and maintenance, such that we go so far as to call it the "zero-integration enterprise." A zero-integration enterprise is an organization whose business and IT teams are committed to the precepts of SOA. These organizations see the business value and strategic advantages of SOA, and are migrating their organization, processes, applications, and skills to support the concept of services, specifically Web services. This organization has eliminated traditional integration models in favor of SOA and standards-based integration. Some characteristics of a zero-integration enterprise follow.
This organization can launch new business initiatives faster than its competitors because less IT integration is required to support various business projects. This time-to-market benefit allows faster response to business conditions, customer requirements, competitive threats, and increased innovation.
This organization has a higher return on assets and greater IT productivity than its peer companies due to the asset-related benefits of SOA, such as software component reuse, services reuse, and extending the capabilities of existing IT systems and infrastructure.
This organization launches new software applications 35% faster than before with reduced quality assurance and testing effort due to component and services reuse. Building on proven software and services capabilities, the organization spends less time developing new code and more time focusing on business process issues.
This organization uses the 30% of its IT budget previously used for integration projects to solve strategic business problems, such as improving customer satisfaction, reducing time to market for new products, and increasing sales through various IT initiatives.
This organization implements concepts of agility and flexibility through its SOA initiatives. Agility is reality, and is measured by clear unambiguous metrics.
This organization has an agile business model that can quickly respond to business challenges, competitive threats, and customer needs.
This organization has greater customer focus deriving from reduced effort spent on internal integration and more effort spent on customer satisfaction, partner communication, and efficient business processes.
This organization has a business-focused IT organization that no longer must concern itself with assuring interoperability issues but rather can focus on forward-looking strategic issues.
This organization has a flexible IT architecture, based on SOA and services, that facilitates superior business performance, enables world-class business processes, and is highly efficient with all corporate resources.
This organization integrates without integrating, both internally and externally with customers and partners. This organization does not integrate, it service-enables instead.
WHAT ARE THE CHALLENGES OF SOA?
Now, with all of these benefits, where's the catch? Simple. SOA is difficult to implement, manage, and control. Not because of the technology, mind you, but due to the organizational, cultural, and behavioral aspects of SOA that contribute to success. And technically, although there has been great progress with regard to the standards, supporting tools, and development and run-time platforms, there are still issues to be resolved. These issues include support for long-running transactions, security concerns, and many others. However, the organizational, cultural, and governance issues far outweigh the technical aspects of implementing SOA. That's not a reason to avoid trying to achieve SOA, but it certainly is a reason to pay attention to many of the softer aspects of technology initiatives to ensure you can reap the rewards. A discussion of major SOA challenges organizations will face as they migrate to SOA follows.
Enterprise Architecture Model May Need Tuning for SOA
As many organizations consider their SOA approach, they will realize that the organization, processes, and disciplines of their enterprise architecture organization may require tuning to suit the requirements of SOA. Often the process of enterprise architecture is somewhat flawed, which helps explain the current state of IT architectures today: rigid IT architectures characterized by heavy carryover legacy systems, inflexible "digital concrete" of enterprise applications, and a portfolio of applications that demand integration software to make them work together.
Perhaps architects have been too focused on building things and getting them to work as opposed to building things that are flexible, reusable, and support the business: the things that are now most important for businesses today as they grapple with change and global forces. These are the new requirements of SOA.
SOA will fail unless the process of architecture is changed from one of static advice, creation of presentations, application blueprints, and architecture road maps to one of actively shaping and implementing flexible and reusable IT assets that support business processes. In other words, SOAs. A model for tuning the enterprise architecture process is presented in Chapter 8.
SOA Is Spatially and Temporally Distributed
One of the challenges of SOA is that it is not implemented all at once. Rather, it is achieved through many discrete projects across both space and time. This temporal and spatial distribution of SOA projects makes governance all the more critical to SOA success. SOA governance and enforceable policies are the keys to managing conformance to the SOA across geographic and time horizons.
SOA Is Organizationally Complex and Behaviorally Challenging
SOA is a complex goal to achieve. It is organizationally, behaviorally, and culturally challenging for most organizations. We describe an SOA behavioral model in Chapter 7 to help you anticipate and create the behavioral pattern your SOA will require.
SOA Requires Governance to Achieve and Manage
SOA requires a robust SOA governance model, clear and enforceable policies, and a way to implement SOA governance across all the life-cycle processes of an organization: enterprise architecture, services design, publishing, discovery, and run-time. SOA governance is essential to realize the ultimate business value of SOA.
SOA Is All About Services
Implementing SOA requires new approaches to identifying, modeling, and implementing reusable, interoperable services. Repeatable processes for determining appropriate granularity, version management, and enforcing design-time policies for services are essential. In fact, even the definition of services is important for many organizations.
The fundamental architectural unit of an SOA is a "service." Services, per our definition of SOA, are units of business capabilities, processes, or functions that are delivered in a repeatable way to consumers of those services. Consumers of services in an SOA can be developers, architects, and analysts, or they can be external customers, business partners, and internal business customers.
Without services, there is no SOA, and an SOA with services is useless unless there is actual consumption of the services that are available. Therefore, an organization can deliver or realize no SOA value unless the services in an SOA have real consumers using them for business reasons.
Services Identification, Modeling, and Design Challenges
Services are critical to an SOA. However, the process of identifying the right services for your organization is challenging. And what makes these services the "right" ones? This discussion is further complicated by questions about granularity, reuse, and other related services modeling and design issues. Chapters 3, 4, and 5 address these concerns using some innovative services modeling concepts. These chapters will help you achieve the "SOA Three Rights": Identify the "right" services; build those services the "right" way; and run them on the "right" enabling technology stack.
Where Are We Headed?
SOA is an iterative business approach. There is no single correct path to achieving SOA. Instead, there are multiple routes to the SOA goal. It is important to recognize the fundamentally iterative approach that SOA requires to achieve its stated goals objectives and business results over time. This approach, which we call an SOA business iteration model, is depicted in Exhibit 1.10.
Exhibit 1.10: SOA Business Iteration Model
This model builds explicit business context into the SOA strategy and planning process, recognizing that SOA must be aligned to business and IT objectives as well as to the current urgencies of the organization at that particular moment in time.
Remember: SOA is a lifestyle change. It is a long-term commitment to achieving specific business objectives. That is why we wrote this book, and that presumably is why you are reading it: to learn how to plan, design, and implement SOA via reusable services to achieve clear business results.
SOA MEANS "SERVICE-ORIENTED AGILITY"
SOA holds promise to finally make the word "agility" real for organizations. That's why the "A" in SOA, which stands for "architecture," could just as well stand for "agility." Service-oriented agility is one of the most oft-articulated goals of SOA. SOA enables business and IT agility along a number of dimensions. Although nearly every business and IT executive for the last 30 years has wistfully dreamed of achieving business agility, there has been little real progress toward that end save for a few exceptional firms. For most organizations, business agility is a vision without reality. Until now. SOA and services provide a means to achieving true business agility. Business agility can come from two broad forms: the ability to change business processes to meet changing market demands and customer requirements, and reduce costs; or the ability to execute business processes faster or launch new processes, products, and services faster. Agility and speed are both real and tangible benefits of migrating to SOA and reusable services.
SOA can help an organization unshackle its business processes and data from the IT systems that support or, in many cases, constrain them. Using a services approach and SOA, enterprise software systems will be decoupled from business processes through the use of business services. Business services will be defined as abstracted entities separate from the business logic that is locked within enterprise applications such as SAP, Oracle, Siebel, and other monolithic enterprise applications. When application logic is exposed as a business service, it becomes a shareable and reusable software asset, or service, that can be coupled with services from other applications to create new sources of business process value, completely new business processes, and even more efficient versions of existing processes using business process management (BPM) tools.
Service-Oriented Agility: Speed
One aspect of agility is speed. The ability of an organization to hasten its response to market changes or competitive threats, or to quickly preempt competitive moves from the competition, is clearly an advantage. Speed consists of two dimensions: the total elapsed time of a business action or response, and the speed of the IT component of the business action or response. Enhancing speed could require installing a new system, developing a new system, running a new report, or whatever the specific business requirement is to support of the business.
If the software development cycle of an organization is too slow for the business to respond to market changes or competitive threats, then the business does not have agility, and clearly IT doesn't have it either. SOA can create agility through speed via faster application development, which in turn will contribute to speedier business responsiveness to market conditions, competitive threats, and customer requests.
Service-Oriented Agility: Flexibility and Range of Response
Service-oriented agility can also be expressed as flexibility, or by allowing a greater range of options for a competitive response and making that response easier. An IT architecture can by its very nature limit the range of options an organization has to respond to market opportunities and customer requests. However, an SOA may offer a greater range of options by reducing the fundamental unit of IT to a business service.
Business agility and IT flexibility are always mentioned in corporate documents, in annual reports, and by business executives when they talk to analysts and customers. Agility and flexibility are among the most discussed and yet least achieved goals in corporate history. Part of the problem comes from a failure to operationalize the terms so they can be implemented and to put metrics in place to help realize them.
SOA: Agility Focal Point
SOA represents an opportunity to regain the agility and flexibility that many organizations lost in the 1990s with enterprise software applications and point-to-point home-grown and commercial integration models. SOA allows the creation of a services layer that resides between the business architecture of an organization and the IT architecture of an organization. We call this layer the agility focal point. This services layer is the decoupling abstraction layer that insulates business processes from IT changes and allows IT to change technology without changing business processes. Exhibit 1.8 depicts the concept of the agility focal point via an SOA.
Exhibit 1.8: SOA and the Agility Focal Point
An SOA implements the agility focal point concept by facilitating the flexing of the business and IT architectures—the SOA in particular—in response to business changes. SOA enables this strategic business capability, which allows an organization to compete based on agility, or service-oriented agility.
Competing on Service-Oriented Agility
With more IT flexibility and business agility, all organizations should be seeking faster time to market for new products and services; faster responses to business, competitive, and environmental change; and an overall better ability to quickly adapt both business processes and IT systems in support of change. In their seminal book Competing Against Time, Stalk and Hout discuss the notion of time-based competition and how those who are faster to market are more profitable than those who are not. [1] SOA enables many of these time-based capabilities by eliminating many of the traditional IT and business barriers to change (e.g., inflexible business processes, business processes locked up within rigid IT systems, inflexible IT architectures with rearview-mirror commitments to legacy systems, etc.). SOA and services provide a business solution to the problem of adapting to change, and this business solution is based on both the business and IT being able to adapt and respond quickly.
Regaining IT Flexibility: Breaking the "Rearview Mirror" Paradox
Much as agility is the Holy Grail for business executives, flexibility is equally sought after and is equally elusive for IT executives. For IT executives, their continual challenge is supporting the years of accumulated legacy systems and infrastructure in the face of shrinking budgets. SOA and Services provide a pathway toward breaking free of the rearview-mirror budgeting paradox. First, in order to inject flexibility into your IT architecture, you do not have to rewrite or refactor every legacy application or enterprise application in your portfolio. You merely have to begin exposing portions of business functionality as services that match to business process requirements of the organization. Second, most of the services you will target initially in your SOA initiatives are contained within existing applications. The challenge is to expose the business functions as reusable services that can in time be combined with other services into composite applications, orchestrated processes, and BPM solutions. Third, SOA is an incremental architecture, meaning that it is not implemented or attained in a big bang model. SOA is achieved over time by defining and adhering to a body of architectural goals, standards, and design guidelines so that all services will interoperate over time within and, when necessary, outside of your enterprise. SOA is the "anti-enterprise application" in that it encourages freeing services from inflexible application architectures imposed by others, namely software vendors, and begins to define your vision of services and business processes to better match the way you want to conduct business. Finally, the concept of business services can allow the IT organization to insulate itself from the constraints imposed by both its legacy systems and its more modern applications. SOAs future-proof your IT architecture. SOAs are built to accommodate change.
Why You Must Begin SOA Initiatives Now
With the analyst and media buzz about SOA, you may be asking yourself why you should believe it. What's so special about SOA that you need to invest your time and resources in this concept at this time? Why now, when you survived the past technology paradigms that were nascent attempts at SOA, namely CORBA, COM/DCOM, and others?
This time, the stars and moon are aligned in SOA's favor. We've covered many of these already, such as the unanimous agreement on the core standards of SOA, the cross-platform capabilities of SOA using Web services and these core standards, and the fact that IT maturity is now more able to assimilate the concept of SOA to drive a business result. There is so much discomfort with the current state of IT that something has to give, and in this case, it's the entire process of IT architecture and services delivery that has to be reconstructed.
SOA presents an opportunity to change the rules of the game. SOA will allow firms to compete using their SOA efforts along a number of business and IT dimensions. These firms will be applying SOA to their businesses to create service-oriented business.
These advantages will characterize SOA first movers:
Competitive advantage. If you beat your competitors to the SOA punch, you will have achieved competitive advantage on a number of fronts, including business speed and agility, IT cost and delivery, and customer satisfaction.
SOA cycles of learning. SOA first movers will gain the experience required to fend off IT vendors partners; you may as well preempt your vendors by getting in front of the SOA wave. If you're going to end up with services anyway, you may as well be prepared and ramp up your ability to consume and provide services.
Break the rearview-mirror budget crisis. There are two ways out of this situation: (1) fix your architecture process to suit an agile, changing services world and (2) stop integrating and service-enable instead.
SOA first movers will be in a better position to compete in a world of services, where vendors, customers, and business partners will all eventually transact via SOA and services. This is a world of service-oriented business, or business-oriented architecture.
SOA can help an organization unshackle its business processes and data from the IT systems that support or, in many cases, constrain them. Using a services approach and SOA, enterprise software systems will be decoupled from business processes through the use of business services. Business services will be defined as abstracted entities separate from the business logic that is locked within enterprise applications such as SAP, Oracle, Siebel, and other monolithic enterprise applications. When application logic is exposed as a business service, it becomes a shareable and reusable software asset, or service, that can be coupled with services from other applications to create new sources of business process value, completely new business processes, and even more efficient versions of existing processes using business process management (BPM) tools.
Service-Oriented Agility: Speed
One aspect of agility is speed. The ability of an organization to hasten its response to market changes or competitive threats, or to quickly preempt competitive moves from the competition, is clearly an advantage. Speed consists of two dimensions: the total elapsed time of a business action or response, and the speed of the IT component of the business action or response. Enhancing speed could require installing a new system, developing a new system, running a new report, or whatever the specific business requirement is to support of the business.
If the software development cycle of an organization is too slow for the business to respond to market changes or competitive threats, then the business does not have agility, and clearly IT doesn't have it either. SOA can create agility through speed via faster application development, which in turn will contribute to speedier business responsiveness to market conditions, competitive threats, and customer requests.
Service-Oriented Agility: Flexibility and Range of Response
Service-oriented agility can also be expressed as flexibility, or by allowing a greater range of options for a competitive response and making that response easier. An IT architecture can by its very nature limit the range of options an organization has to respond to market opportunities and customer requests. However, an SOA may offer a greater range of options by reducing the fundamental unit of IT to a business service.
Business agility and IT flexibility are always mentioned in corporate documents, in annual reports, and by business executives when they talk to analysts and customers. Agility and flexibility are among the most discussed and yet least achieved goals in corporate history. Part of the problem comes from a failure to operationalize the terms so they can be implemented and to put metrics in place to help realize them.
SOA: Agility Focal Point
SOA represents an opportunity to regain the agility and flexibility that many organizations lost in the 1990s with enterprise software applications and point-to-point home-grown and commercial integration models. SOA allows the creation of a services layer that resides between the business architecture of an organization and the IT architecture of an organization. We call this layer the agility focal point. This services layer is the decoupling abstraction layer that insulates business processes from IT changes and allows IT to change technology without changing business processes. Exhibit 1.8 depicts the concept of the agility focal point via an SOA.
Exhibit 1.8: SOA and the Agility Focal Point
An SOA implements the agility focal point concept by facilitating the flexing of the business and IT architectures—the SOA in particular—in response to business changes. SOA enables this strategic business capability, which allows an organization to compete based on agility, or service-oriented agility.
Competing on Service-Oriented Agility
With more IT flexibility and business agility, all organizations should be seeking faster time to market for new products and services; faster responses to business, competitive, and environmental change; and an overall better ability to quickly adapt both business processes and IT systems in support of change. In their seminal book Competing Against Time, Stalk and Hout discuss the notion of time-based competition and how those who are faster to market are more profitable than those who are not. [1] SOA enables many of these time-based capabilities by eliminating many of the traditional IT and business barriers to change (e.g., inflexible business processes, business processes locked up within rigid IT systems, inflexible IT architectures with rearview-mirror commitments to legacy systems, etc.). SOA and services provide a business solution to the problem of adapting to change, and this business solution is based on both the business and IT being able to adapt and respond quickly.
Regaining IT Flexibility: Breaking the "Rearview Mirror" Paradox
Much as agility is the Holy Grail for business executives, flexibility is equally sought after and is equally elusive for IT executives. For IT executives, their continual challenge is supporting the years of accumulated legacy systems and infrastructure in the face of shrinking budgets. SOA and Services provide a pathway toward breaking free of the rearview-mirror budgeting paradox. First, in order to inject flexibility into your IT architecture, you do not have to rewrite or refactor every legacy application or enterprise application in your portfolio. You merely have to begin exposing portions of business functionality as services that match to business process requirements of the organization. Second, most of the services you will target initially in your SOA initiatives are contained within existing applications. The challenge is to expose the business functions as reusable services that can in time be combined with other services into composite applications, orchestrated processes, and BPM solutions. Third, SOA is an incremental architecture, meaning that it is not implemented or attained in a big bang model. SOA is achieved over time by defining and adhering to a body of architectural goals, standards, and design guidelines so that all services will interoperate over time within and, when necessary, outside of your enterprise. SOA is the "anti-enterprise application" in that it encourages freeing services from inflexible application architectures imposed by others, namely software vendors, and begins to define your vision of services and business processes to better match the way you want to conduct business. Finally, the concept of business services can allow the IT organization to insulate itself from the constraints imposed by both its legacy systems and its more modern applications. SOAs future-proof your IT architecture. SOAs are built to accommodate change.
Why You Must Begin SOA Initiatives Now
With the analyst and media buzz about SOA, you may be asking yourself why you should believe it. What's so special about SOA that you need to invest your time and resources in this concept at this time? Why now, when you survived the past technology paradigms that were nascent attempts at SOA, namely CORBA, COM/DCOM, and others?
This time, the stars and moon are aligned in SOA's favor. We've covered many of these already, such as the unanimous agreement on the core standards of SOA, the cross-platform capabilities of SOA using Web services and these core standards, and the fact that IT maturity is now more able to assimilate the concept of SOA to drive a business result. There is so much discomfort with the current state of IT that something has to give, and in this case, it's the entire process of IT architecture and services delivery that has to be reconstructed.
SOA presents an opportunity to change the rules of the game. SOA will allow firms to compete using their SOA efforts along a number of business and IT dimensions. These firms will be applying SOA to their businesses to create service-oriented business.
These advantages will characterize SOA first movers:
Competitive advantage. If you beat your competitors to the SOA punch, you will have achieved competitive advantage on a number of fronts, including business speed and agility, IT cost and delivery, and customer satisfaction.
SOA cycles of learning. SOA first movers will gain the experience required to fend off IT vendors partners; you may as well preempt your vendors by getting in front of the SOA wave. If you're going to end up with services anyway, you may as well be prepared and ramp up your ability to consume and provide services.
Break the rearview-mirror budget crisis. There are two ways out of this situation: (1) fix your architecture process to suit an agile, changing services world and (2) stop integrating and service-enable instead.
SOA first movers will be in a better position to compete in a world of services, where vendors, customers, and business partners will all eventually transact via SOA and services. This is a world of service-oriented business, or business-oriented architecture.
Exploring an SOA Business Scenario
Below depicts a hypothetical approach to SOA as an alternative to conventional integration. This exhibit shows a fictional insurance company with three business units—Group, Voluntary, and Individual—at the top. Corresponding to the three business units are their own collections of systems and duplicate IT capabilities, accumulated over the years, and purportedly so unique that the business units must have their own systems to accomplish the very same business processes.
Typical IT System Silos
Each business unit has duplicate systems for sales and contact management, enrollment, claims, settlement, policy administration, and management reporting and controls. On top of these, all the business units have portal, intranet, and extranet capabilities to allow self-service for customers, agents, and brokers.
Identify SOA Business Opportunities
Exhibit 1.3 examines this hypothetical business from an SOA perspective, exploring the potential business value that SOA may bring to this enterprise. What if the duplicate IT systems and business processes could be integrated and united as shared services by all three business units? What if the business processes across the three business units could be simplified to take advantage of shared IT services in an SOA model?
Exhibit 1.3: Identifying SOA Business Opportunities
As shown in the exhibit, SOA offers several potential business benefits to this organization, such as product and process simplification, integrated systems, integrated business services, better customer satisfaction, cost reductions, increased revenue and margin, and better agent and/or broker productivity. Of course, you cannot realize these potential benefits without performing proper analysis.
Examining the SOA Information Technology Potential
Continuing the example, Exhibit 1.4 shows specific SOA opportunities that may apply to this insurance company. These opportunities include integration, process orchestration, better interoperability, services reuse, improved IT productivity, and achieving a real-time, event-driven enterprise.
Exhibit 1.4: Identifying SOA Opportunities
Again, proper analysis and SOA planning of the business and SOA opportunities will determine the unique SOA value that will apply to any given organization.
Seeking Reuse Value from an SOA Initiative
Let us assume that services reuse is a key SOA driver for this insurance company. Our SOA strategy, planning, and analysis will identify significant services reuse benefits in many areas of this company, across the multiple business units, and across the IT systems that support them. Exhibit 1.5 focuses this example on services reuse. In this 1insurance company, it was determined that reuse was essential to reducing costs, increasing IT productivity, and improving responsiveness to business demands.
Exhibit 1.5: Services Reuse in an SOA
This example shows how analysis might show that reuse of specific IT capabilities may offer tremendous business value. In this scenario, we focus on the enrollment process, which is supported by an excellent system but which can be improved by exposing its capabilities as shared reusable services in an SOA. Doing this allows reuse of a single IT service, enrollment, across three business units. This reuse eliminates IT complexity, increases IT productivity, and leads to simplified business processes in this organization. SOA applies as much to business processes and enabling better business functionality as it applies to IT systems—eliminating redundant systems, duplicate support infrastructure, and fragile and expensive integration strategies.
Service and Process Orchestration in an SOA
Furthermore, this same organization is able to leverage its reusable services by orchestrating them into business processes. Service and process orchestration extends reuse benefits by leveraging services as reusable composite applications and orchestrated process work flows. Exhibit 1.6 shows the enrollment process as a simplified series of services orchestrated into the business process work flow that the company decides is most reflective of how it wants to operate.
Exhibit 1.6: Orchestrating Processes in an SOA
SOA allows an organization to stop integrating and instead recast its IT capabilities as shared reusable services. Once there are enough services to reuse in an SOA, further value can be harvested by orchestrating business processes based on these reusable services. In addition, the integration challenges that have historically plagued IT organizations can be avoided. Implementing the SOA scenario just described enables an organization to realize the many business benefits of SOA.
SOA: Competitive Advantage via Services
SOA is a concept that has direct business and competitive advantage implications for all organizations. For the business, SOA means increased customer satisfaction, real business agility, faster time to market, ease of partnering, and lower business costs. Imagine that you can launch new products and services 30% faster than your competitors because you have eliminated friction within your enterprise, allowing better collaboration between your suppliers and your design engineers as well as better collaboration with your channel partners.
Your SOA has allowed you to speed up IT delivery to your business consumers. The time to implement needed system changes to support these new products has been cut by 25%, and you are able to make these changes using fewer resources: less development resources, less quality assurance resources, and less overall IT resources. Exhibit 1.7 depicts typical business benefits of SOA.
Exhibit 1.7: Business Benefits of SOA
For IT organizations, SOA means greater productivity, faster time to market, greater asset reuse, agility, and lower IT costs. What if you could deliver IT services with less budget and few resources, while providing faster, higher-quality application support? SOA benefits an IT organization through faster application development, lower overall costs, greater software asset and services reuse, more repeatable software development processes, higher-quality applications via pretested components and validated Web services interfaces, and overall faster response to business customer requests for system enhancements and modifications.
SOA benefits the business through greater flexibility, faster time to market for business initiatives, faster response to business changes, and closer mapping of IT services to business needs. Many analysts see SOA as a mechanism to help finally achieve alignment of business goals and objectives with IT services and capabilities. Better alignment is partly attributable to the speed with which Web services can be developed and deployed in an SOA as well as the flexibility from leveraging proven, tested software components and Web services.
SOA also benefits an organization by abstracting business services (or Web services) from the specific technologies they were originally developed with and the platforms they were meant to run on. This twofold abstraction has two key benefits: (1) an organization can modify the technology architecture without mandating changes to the services available, and (2) the business community can change business processes without causing the ripple effect of changes to the services and underlying IT systems. Doing this leads to greater business agility and IT flexibility.
Typical IT System Silos
Each business unit has duplicate systems for sales and contact management, enrollment, claims, settlement, policy administration, and management reporting and controls. On top of these, all the business units have portal, intranet, and extranet capabilities to allow self-service for customers, agents, and brokers.
Identify SOA Business Opportunities
Exhibit 1.3 examines this hypothetical business from an SOA perspective, exploring the potential business value that SOA may bring to this enterprise. What if the duplicate IT systems and business processes could be integrated and united as shared services by all three business units? What if the business processes across the three business units could be simplified to take advantage of shared IT services in an SOA model?
Exhibit 1.3: Identifying SOA Business Opportunities
As shown in the exhibit, SOA offers several potential business benefits to this organization, such as product and process simplification, integrated systems, integrated business services, better customer satisfaction, cost reductions, increased revenue and margin, and better agent and/or broker productivity. Of course, you cannot realize these potential benefits without performing proper analysis.
Examining the SOA Information Technology Potential
Continuing the example, Exhibit 1.4 shows specific SOA opportunities that may apply to this insurance company. These opportunities include integration, process orchestration, better interoperability, services reuse, improved IT productivity, and achieving a real-time, event-driven enterprise.
Exhibit 1.4: Identifying SOA Opportunities
Again, proper analysis and SOA planning of the business and SOA opportunities will determine the unique SOA value that will apply to any given organization.
Seeking Reuse Value from an SOA Initiative
Let us assume that services reuse is a key SOA driver for this insurance company. Our SOA strategy, planning, and analysis will identify significant services reuse benefits in many areas of this company, across the multiple business units, and across the IT systems that support them. Exhibit 1.5 focuses this example on services reuse. In this 1insurance company, it was determined that reuse was essential to reducing costs, increasing IT productivity, and improving responsiveness to business demands.
Exhibit 1.5: Services Reuse in an SOA
This example shows how analysis might show that reuse of specific IT capabilities may offer tremendous business value. In this scenario, we focus on the enrollment process, which is supported by an excellent system but which can be improved by exposing its capabilities as shared reusable services in an SOA. Doing this allows reuse of a single IT service, enrollment, across three business units. This reuse eliminates IT complexity, increases IT productivity, and leads to simplified business processes in this organization. SOA applies as much to business processes and enabling better business functionality as it applies to IT systems—eliminating redundant systems, duplicate support infrastructure, and fragile and expensive integration strategies.
Service and Process Orchestration in an SOA
Furthermore, this same organization is able to leverage its reusable services by orchestrating them into business processes. Service and process orchestration extends reuse benefits by leveraging services as reusable composite applications and orchestrated process work flows. Exhibit 1.6 shows the enrollment process as a simplified series of services orchestrated into the business process work flow that the company decides is most reflective of how it wants to operate.
Exhibit 1.6: Orchestrating Processes in an SOA
SOA allows an organization to stop integrating and instead recast its IT capabilities as shared reusable services. Once there are enough services to reuse in an SOA, further value can be harvested by orchestrating business processes based on these reusable services. In addition, the integration challenges that have historically plagued IT organizations can be avoided. Implementing the SOA scenario just described enables an organization to realize the many business benefits of SOA.
SOA: Competitive Advantage via Services
SOA is a concept that has direct business and competitive advantage implications for all organizations. For the business, SOA means increased customer satisfaction, real business agility, faster time to market, ease of partnering, and lower business costs. Imagine that you can launch new products and services 30% faster than your competitors because you have eliminated friction within your enterprise, allowing better collaboration between your suppliers and your design engineers as well as better collaboration with your channel partners.
Your SOA has allowed you to speed up IT delivery to your business consumers. The time to implement needed system changes to support these new products has been cut by 25%, and you are able to make these changes using fewer resources: less development resources, less quality assurance resources, and less overall IT resources. Exhibit 1.7 depicts typical business benefits of SOA.
Exhibit 1.7: Business Benefits of SOA
For IT organizations, SOA means greater productivity, faster time to market, greater asset reuse, agility, and lower IT costs. What if you could deliver IT services with less budget and few resources, while providing faster, higher-quality application support? SOA benefits an IT organization through faster application development, lower overall costs, greater software asset and services reuse, more repeatable software development processes, higher-quality applications via pretested components and validated Web services interfaces, and overall faster response to business customer requests for system enhancements and modifications.
SOA benefits the business through greater flexibility, faster time to market for business initiatives, faster response to business changes, and closer mapping of IT services to business needs. Many analysts see SOA as a mechanism to help finally achieve alignment of business goals and objectives with IT services and capabilities. Better alignment is partly attributable to the speed with which Web services can be developed and deployed in an SOA as well as the flexibility from leveraging proven, tested software components and Web services.
SOA also benefits an organization by abstracting business services (or Web services) from the specific technologies they were originally developed with and the platforms they were meant to run on. This twofold abstraction has two key benefits: (1) an organization can modify the technology architecture without mandating changes to the services available, and (2) the business community can change business processes without causing the ripple effect of changes to the services and underlying IT systems. Doing this leads to greater business agility and IT flexibility.
SOA
SOA is not a product. SOA is not a solution. SOA is not a technology. SOA cannot be reduced to vendors' software products, much as they would like you to believe. SOA is not a quick fix for the IT complexity that has accumulated over 30-plus years. And finally, SOA does not address every IT challenge facing business and IT executives today.
SOA is a conceptual business architecture where business functionality, or application logic, is made available to SOA users, or consumers, as shared, reusable services on an IT network. "Services" in an SOA are modules of business or application functionality with exposed interfaces, and are invoked by messages.
ELEMENTS OF AN SOA
The following list represents the essential ingredients of a successful SOA.
Conceptual SOA vision
Services
Enabling technology
SOA governance and policies
SOA metrics
Organizational and behavioral model
Conceptual SOA Vision
An SOA is a business concept, an idea or approach, of how IT functionality can be planned, designed, and delivered as modular business services to achieve specific business benefits. The conceptual SOA vision includes clearly defined business, IT and architectural goals, and a governance model and policies to help enforce standards and technical requirements of the SOA over time. This is the definition of an SOA target state, the goal to be achieved over time
Services
Yes, an SOA needs services, which as we said, means all possible services in the organization. Along with services comes a services design model to assure reusability, interoperability, and integration across all business processes and technology platforms. Services are the central artifact of an SOA. Services are the primary architectural asset of an SOA. As such, they merit significant attention throughout this book and throughout an organization's migration toward SOA through many projects and initiatives, each of which will most likely contribute services to the SOA over time.
Enabling Technology
While the technology of Web services and SOA generates lots of press, it is probably the easiest area to implement despite the vendor flux and standards volatility for various categories of technology solutions. The technology is essential to support realization of your SOA vision. However, the enabling technology is not your SOA. The enabling technology must be implemented to accomplish two objectives: (1) It must allow your services to operate reliably and securely in your enterprise in support of your stated business objectives; and (2) it must enable you to carry forward your existing IT architecture as well as enable your legacy systems to be leveraged to support your SOA goals. In many organizations, legacy mainframe systems and other applications are major contributors of services to an SOA.
SOA Governance and Policies
An SOA conceptual architecture cannot be realized unless it is communicated to the constituents of the SOA—business users, developers, architects, business and IT executives, and business analysts. In addition, communicating your SOA conceptual architecture to close trading partners is also advised. However, telling your SOA constituents what your conceptual architecture, vision, and goals are is one thing. Enforcing conformance to your SOA conceptual architecture, vision, and goals is another matter. SOA is not a big bang implementation model that we expect from large, packaged software applications. SOA is achieved incrementally through time at the project level by continuously defining and enforcing the standards that it will be based on. These standards are the policies that in the aggregate define your SOA conceptual architecture and, when implemented, help your organization achieve its SOA vision and business goals. An SOA governance model defines the various governance processes, organizational roles and responsibilities, standards and policies that must be adhered to in your SOA conceptual architecture.
Metrics
SOAs require a battery of metrics in order to measure the results you are achieving. These metrics include fine-grained metrics, such as service-level agreements (SLAs) for individual services, as well as usage metrics, policy conformance metrics, developer metrics, business and return on investment (ROI) metrics, and process metrics. Plan your metrics early, and don't forget them when you go live with services. You'll want the data, count on it.
Organizational and Behavioral Model
Your current IT architecture is the result of years of organizational behaviors, business decisions, and architectural choices. In order to achieve SOA, behavioral and organizational considerations must be understood and changed first; then over time will come gradual migration toward your SOA vision and goals. New organizational models and behavioral models will be essential to your SOA success.
SOA: BEHAVIOR AND CULTURE
SOAs contain a substantial amount of behavioral content because these initiatives are process-driven and span organizational boundaries. The "soft issues" of an SOA strategy must address the organizational issues and challenges that may help or inhibit SOA adoption, such as services ownership, the business and IT relationship, budgeting practices, and more. Organizational, cultural, and process issues thread through several facets of an SOA initiative. How do you organize your enterprise architecture functions and roles to support an SOA? How do you organize your developer resources to help ensure the realization of the goals and performance of your SOA initiative? What is the optimal IT structure for an SOA? Is centralized IT better? Or is a centralized enterprise architecture team optimal, supported by distributed developers embedded within specific business units? What are the skills, roles, and competencies of your architecture organization that will facilitate migration to and attainment of your SOA?
NEW SOA CONCEPTUAL, ARCHITECTURAL, AND ORGANIZATIONAL MODELS
SOA initiatives will stress and in most cases break current operational and architectural models of IT organizations. SOA will require new ways of modeling and implementing various IT processes we have become accustomed to, such as services design models, integration models, reuse models, architecture processes, and enterprise architecture models. These other models, of course, augment the required SOA governance model.
Over the years, your IT architecture has accumulated layer upon layer of complexity. When client-server architectures dominated the IT industry, client-server applications were layered over your mainframe platforms. When the Internet era rose to ascendance, Webcentric platforms were added on top of client-server solutions. And as these evolved into n-tiered architectures, the layers of IT complexity built up, building more modern complexity on top of legacy complexity.
Although IT architecture through the years is accretive, and nothing seems to ever go away, there are ways to "architect" your way out of this conundrum. One approach is to rip out all the legacy applications and replace them with modern ones. This rip-and-replace model is too expensive for most organizations. In addition, often it is not worth the risk and effort of replacing these still-working legacy systems with new software
Another approach is to rewrite or refactor your legacy systems for modern application server platforms, such as J2EE or .NET. Even though rewriting systems is also very expensive, at least you know they will match your business processes when you are through. Like the rip-and-replace model, this approach usually is avoided, however, not because it is not the right thing to do, but because it is expensive and difficult to cost justify to the business.
But there is a way out of this mess that avoids big bang system rewrites and expensive enterprise software projects. The way is service-oriented architecture. SOA is a simple concept, one that has the potential to alleviate many long-standing IT challenges and enable many coveted business goals that have until now been very elusive. SOA, by introducing a services layer into an existing IT architecture, can provide opportunities to isolate areas or elements of the architecture that are problematic, failure prone, or cost prohibitive. The services layering approach can enable the isolation, replacement, and/or potential consolidation of these architecture challenges while enabling the flexibility of reusable services. How often have business executives pronounced a desire to become more agile? When has time-to-market not been a mission-critical business requirement? Yet more often than not, these lofty business goals are constrained by outdated IT systems and incapable business processes that are subservient to tradition as well as to the digital concrete of today's enterprise software applications.
Why SOA Now?
SOA has also achieved trend status because of the degree of dissatisfaction that IT and business executives share with the current state of IT within their enterprises. Chief executives (CEOs) are fed up with hearing why they cannot expand into a new geographical location because the IT systems are not ready yet. They don't want to hear why the enterprise resource planning (ERP) system that cost $30 million will not support the new business process targeted to launch in six months. Chief financial officers (CFOs) are tired of waiting for regulatory compliance issues to be resolved, and they certainly are not pleased with an overbudget IT organization. Chief operating officers (COOs) resent being told they cannot get a report because data are spread across three different systems, all on different computing platforms. Chief technology officers (CTOs) are fed up with vendors pushing more new technology when the old technology is still underutilized and operating as islands of functionality. They are tired of the endless need to keep integrating systems when the integration models themselves become part of the problem—more legacy silos to maintain. And chief information officers (CIOs)? They are tired of explaining the same problems time and time again. They're tired of having their budgets cut. They wish they had more funding for strategic projects instead of being hamstrung by having 80 to 90% of their budget committed to maintaining legacy systems. CIOs could do much more for the business if they could shed their legacy systems and focus on forward-looking strategic solutions for the business. There has to be a way out of this quandary. SOA could well be an answer. SOA is not new, but it's here to stay. SOA is finally achievable thanks to three major factors:
Standards consensus. Microsoft and IBM agreed, and the rest fell in line.
SOA enabling technology. Finally, implementing standards-based services is possible and affordable.
Integration fatigue. There has to be a better way to achieve application and business integration.
Standards Consensus
For the first time in the history of IT, there is widespread agreement on major SOA and Web services standards by all IT vendors. This nearly unanimous agreement means that whether you move now or later, you most certainly are going to be using services in your organization. Your software and platform vendors are going to take you there whether you want to go or not. Our advice? Preempt your vendors with your own SOA strategy and roadmap. Rapidly accumulate SOA experience. And be prepared to fend off any proprietary platform-specific approaches to services. Implement industry standards in your SOA governance model and in specific policies that will govern services identification, design, and implementation. You may have to dedicate some internal resources to tracking relevant standards, but the ROI on standards will be well worth it.
SOA Tools and Infrastructure
With the advent of new tools and infrastructure solutions that enable SOA and services in a cross-platform, reusable, and interoperable fashion, SOA is real. This is perhaps the most significant departure from previous SOA implementations, such as CORBA (Common Object Request Broker Architecture), COM/DCOM (Common Object Model/Distributed Common Object Model), DCE (Distributed Computing Environment), and other proprietary schemes for reusable services. Interoperability for services is largely due to the standards for Web services, primarily SOAP for messaging and WSDL (Web services description language) for service descriptions. The variety of tools for legacy systems enablement, services development and exposure, Web services management, and multiple run-time environments for services have made the SOA industry very interesting to watch. There are as many ways to enable services and SOA as there are legacy systems and platforms in your architecture. Of course, bear in mind that we refer to the general case of "services" in this book. Some of your services will be Web services based on XML (eXtensible Markup Language), SOAP, and WSDL, as well as the extended WS-* standards. However, do not limit your SOA total value by examining and considering Web services only. Think services first, and then specific implementation models later.
Integration Fatigue: "There Has to Be a Better Way"
IT "business as usual" is over. Business and IT executives are frustrated with the lack of integration of their internal systems, with their business processes, with their trading partners, and with their customers. We call this integration fatigue. The business is demanding change, and IT executives know there has to be a better way. The frustration with IT as we know it is at an all-time high. IT budgets continue to be stressed, and they rarely increase. The majority of IT budgets are focused on maintaining current systems and keeping the lights on; very little IT budget is focused on strategic initiatives that may pay future dividends. It is a do-more-with-less environment.
Business continues to change while IT is saddled with maintaining the systems and architectures of the past. IT doesn't have the luxury of eliminating its legacy or underperforming assets. Those very IT assets contain business logic that is most likely running a mission-critical portion of the organization's business. Yet while the business logic is mission critical, nonetheless the logic and data are locked up within individual silos of systems and technology. You cannot afford to rewrite the application, and your integration strategy has proved to be supremely costly to implement and maintain. There has to be a better way, and there is. It's called service-oriented architecture.
SOA: EVOLVED ENTERPRISE INTEGRATION
A major impetus for SOA initiatives is solving the age-old problem of integration. For many executives, SOA holds the potential to eliminate, via industry standards and modern tools, the proprietary integration model they've become accustomed to. According to many analyst estimates, up to 30% of a typical IT budget is allocated to integration activities. What would business be like if there was less integration, or, rather, if the integration that was performed was directly related to process integration, enterprise integration, and mergers and acquisitions (M&A) integration? In other words, value-added integration. How would spending less money on integration change an organization's competitive advantage? Could that budget be shifted to more strategic projects?
Origins of the Integration Problem
Where did all this IT complexity come from? Why is 80 to 90% of your IT budget focused backward on maintaining the past rather than looking ahead to supporting the future? This "rearview mirror budgeting" problem is legendary among CIOs and is partly responsible for the lack of strategic IT investment by CIOs today. How backward committed is your IT budget? What percent of the IT budget is allocated to maintaining your legacy investments rather than focused on forward-facing initiatives that may move the organization ahead? This is a real challenge for both business and IT executives today. If you feel as if you're managing your IT budgets using a rearview mirror, you're not alone.
The demand for IT and process integration is driven by business requirements, such as:
Increased M&A activity
Corporate reorganization or restructuring
Application and/or system consolidation
Data integration and data warehousing initiatives
New business strategies leveraging current systems for new processes
Achieving regulatory compliance (e.g., Sarbanes-Oxley, or HIPAA, the Health Insurance Portability and Accountability Act of 1996)
Streamlining of business processes to improve productivity
Addressing the business drivers for integration is a great impetus for SOA and Web services. At what point does IT complexity become an obstacle to business goals and an impediment to achieving IT's goals? We believe that complexity becomes intolerable when organizations are considering or taking these actions:
Hiring a chief architect
Creating a central architecture team
Acquiring or developing your own enterprise application integration (EAI) software
Creating an internal integration team, or a middleware organization, to help solve the integration challenges for your organization
Now, this does not mean that hiring a chief architect is a bad thing. More than likely it is a good thing. Chief architects can help address the architectural complexity and pain your organization is facing. The other actions just listed also can help address these areas. The problems are symptoms of the IT challenges that SOA could address. If you have considered or taken one or more of these actions, your organization is at the point where the integration burden is consuming IT resources, compounding the existing complexity problem, and inhibiting business and IT effectiveness. You most likely have a rearview-mirror IT budget, and you are ready to try a new approach.
Stop Integrating; Service-Enable Instead
Stop integrating now. What we mean is stop integrating the way you have been using proprietary middleware solutions, homegrown point-to-point integration techniques, and tactical integration approaches that are doomed from the outset. These techniques are almost always going to break and require a significant ongoing maintenance burden from the organization.
An organization should integrate without integrating. Eliminate all point-to-point integration projects in your enterprise and rearchitect these initiatives from an SOA point of view. Inventory the integration solutions in your organization. Identify the IT budget allocated to these solutions and projects, including support staff, maintenance, and infrastructure. Determine how many of these integration efforts can be eliminated using reusable services in an SOA.
SOA is a conceptual business architecture where business functionality, or application logic, is made available to SOA users, or consumers, as shared, reusable services on an IT network. "Services" in an SOA are modules of business or application functionality with exposed interfaces, and are invoked by messages.
ELEMENTS OF AN SOA
The following list represents the essential ingredients of a successful SOA.
Conceptual SOA vision
Services
Enabling technology
SOA governance and policies
SOA metrics
Organizational and behavioral model
Conceptual SOA Vision
An SOA is a business concept, an idea or approach, of how IT functionality can be planned, designed, and delivered as modular business services to achieve specific business benefits. The conceptual SOA vision includes clearly defined business, IT and architectural goals, and a governance model and policies to help enforce standards and technical requirements of the SOA over time. This is the definition of an SOA target state, the goal to be achieved over time
Services
Yes, an SOA needs services, which as we said, means all possible services in the organization. Along with services comes a services design model to assure reusability, interoperability, and integration across all business processes and technology platforms. Services are the central artifact of an SOA. Services are the primary architectural asset of an SOA. As such, they merit significant attention throughout this book and throughout an organization's migration toward SOA through many projects and initiatives, each of which will most likely contribute services to the SOA over time.
Enabling Technology
While the technology of Web services and SOA generates lots of press, it is probably the easiest area to implement despite the vendor flux and standards volatility for various categories of technology solutions. The technology is essential to support realization of your SOA vision. However, the enabling technology is not your SOA. The enabling technology must be implemented to accomplish two objectives: (1) It must allow your services to operate reliably and securely in your enterprise in support of your stated business objectives; and (2) it must enable you to carry forward your existing IT architecture as well as enable your legacy systems to be leveraged to support your SOA goals. In many organizations, legacy mainframe systems and other applications are major contributors of services to an SOA.
SOA Governance and Policies
An SOA conceptual architecture cannot be realized unless it is communicated to the constituents of the SOA—business users, developers, architects, business and IT executives, and business analysts. In addition, communicating your SOA conceptual architecture to close trading partners is also advised. However, telling your SOA constituents what your conceptual architecture, vision, and goals are is one thing. Enforcing conformance to your SOA conceptual architecture, vision, and goals is another matter. SOA is not a big bang implementation model that we expect from large, packaged software applications. SOA is achieved incrementally through time at the project level by continuously defining and enforcing the standards that it will be based on. These standards are the policies that in the aggregate define your SOA conceptual architecture and, when implemented, help your organization achieve its SOA vision and business goals. An SOA governance model defines the various governance processes, organizational roles and responsibilities, standards and policies that must be adhered to in your SOA conceptual architecture.
Metrics
SOAs require a battery of metrics in order to measure the results you are achieving. These metrics include fine-grained metrics, such as service-level agreements (SLAs) for individual services, as well as usage metrics, policy conformance metrics, developer metrics, business and return on investment (ROI) metrics, and process metrics. Plan your metrics early, and don't forget them when you go live with services. You'll want the data, count on it.
Organizational and Behavioral Model
Your current IT architecture is the result of years of organizational behaviors, business decisions, and architectural choices. In order to achieve SOA, behavioral and organizational considerations must be understood and changed first; then over time will come gradual migration toward your SOA vision and goals. New organizational models and behavioral models will be essential to your SOA success.
SOA: BEHAVIOR AND CULTURE
SOAs contain a substantial amount of behavioral content because these initiatives are process-driven and span organizational boundaries. The "soft issues" of an SOA strategy must address the organizational issues and challenges that may help or inhibit SOA adoption, such as services ownership, the business and IT relationship, budgeting practices, and more. Organizational, cultural, and process issues thread through several facets of an SOA initiative. How do you organize your enterprise architecture functions and roles to support an SOA? How do you organize your developer resources to help ensure the realization of the goals and performance of your SOA initiative? What is the optimal IT structure for an SOA? Is centralized IT better? Or is a centralized enterprise architecture team optimal, supported by distributed developers embedded within specific business units? What are the skills, roles, and competencies of your architecture organization that will facilitate migration to and attainment of your SOA?
NEW SOA CONCEPTUAL, ARCHITECTURAL, AND ORGANIZATIONAL MODELS
SOA initiatives will stress and in most cases break current operational and architectural models of IT organizations. SOA will require new ways of modeling and implementing various IT processes we have become accustomed to, such as services design models, integration models, reuse models, architecture processes, and enterprise architecture models. These other models, of course, augment the required SOA governance model.
Over the years, your IT architecture has accumulated layer upon layer of complexity. When client-server architectures dominated the IT industry, client-server applications were layered over your mainframe platforms. When the Internet era rose to ascendance, Webcentric platforms were added on top of client-server solutions. And as these evolved into n-tiered architectures, the layers of IT complexity built up, building more modern complexity on top of legacy complexity.
Although IT architecture through the years is accretive, and nothing seems to ever go away, there are ways to "architect" your way out of this conundrum. One approach is to rip out all the legacy applications and replace them with modern ones. This rip-and-replace model is too expensive for most organizations. In addition, often it is not worth the risk and effort of replacing these still-working legacy systems with new software
Another approach is to rewrite or refactor your legacy systems for modern application server platforms, such as J2EE or .NET. Even though rewriting systems is also very expensive, at least you know they will match your business processes when you are through. Like the rip-and-replace model, this approach usually is avoided, however, not because it is not the right thing to do, but because it is expensive and difficult to cost justify to the business.
But there is a way out of this mess that avoids big bang system rewrites and expensive enterprise software projects. The way is service-oriented architecture. SOA is a simple concept, one that has the potential to alleviate many long-standing IT challenges and enable many coveted business goals that have until now been very elusive. SOA, by introducing a services layer into an existing IT architecture, can provide opportunities to isolate areas or elements of the architecture that are problematic, failure prone, or cost prohibitive. The services layering approach can enable the isolation, replacement, and/or potential consolidation of these architecture challenges while enabling the flexibility of reusable services. How often have business executives pronounced a desire to become more agile? When has time-to-market not been a mission-critical business requirement? Yet more often than not, these lofty business goals are constrained by outdated IT systems and incapable business processes that are subservient to tradition as well as to the digital concrete of today's enterprise software applications.
Why SOA Now?
SOA has also achieved trend status because of the degree of dissatisfaction that IT and business executives share with the current state of IT within their enterprises. Chief executives (CEOs) are fed up with hearing why they cannot expand into a new geographical location because the IT systems are not ready yet. They don't want to hear why the enterprise resource planning (ERP) system that cost $30 million will not support the new business process targeted to launch in six months. Chief financial officers (CFOs) are tired of waiting for regulatory compliance issues to be resolved, and they certainly are not pleased with an overbudget IT organization. Chief operating officers (COOs) resent being told they cannot get a report because data are spread across three different systems, all on different computing platforms. Chief technology officers (CTOs) are fed up with vendors pushing more new technology when the old technology is still underutilized and operating as islands of functionality. They are tired of the endless need to keep integrating systems when the integration models themselves become part of the problem—more legacy silos to maintain. And chief information officers (CIOs)? They are tired of explaining the same problems time and time again. They're tired of having their budgets cut. They wish they had more funding for strategic projects instead of being hamstrung by having 80 to 90% of their budget committed to maintaining legacy systems. CIOs could do much more for the business if they could shed their legacy systems and focus on forward-looking strategic solutions for the business. There has to be a way out of this quandary. SOA could well be an answer. SOA is not new, but it's here to stay. SOA is finally achievable thanks to three major factors:
Standards consensus. Microsoft and IBM agreed, and the rest fell in line.
SOA enabling technology. Finally, implementing standards-based services is possible and affordable.
Integration fatigue. There has to be a better way to achieve application and business integration.
Standards Consensus
For the first time in the history of IT, there is widespread agreement on major SOA and Web services standards by all IT vendors. This nearly unanimous agreement means that whether you move now or later, you most certainly are going to be using services in your organization. Your software and platform vendors are going to take you there whether you want to go or not. Our advice? Preempt your vendors with your own SOA strategy and roadmap. Rapidly accumulate SOA experience. And be prepared to fend off any proprietary platform-specific approaches to services. Implement industry standards in your SOA governance model and in specific policies that will govern services identification, design, and implementation. You may have to dedicate some internal resources to tracking relevant standards, but the ROI on standards will be well worth it.
SOA Tools and Infrastructure
With the advent of new tools and infrastructure solutions that enable SOA and services in a cross-platform, reusable, and interoperable fashion, SOA is real. This is perhaps the most significant departure from previous SOA implementations, such as CORBA (Common Object Request Broker Architecture), COM/DCOM (Common Object Model/Distributed Common Object Model), DCE (Distributed Computing Environment), and other proprietary schemes for reusable services. Interoperability for services is largely due to the standards for Web services, primarily SOAP for messaging and WSDL (Web services description language) for service descriptions. The variety of tools for legacy systems enablement, services development and exposure, Web services management, and multiple run-time environments for services have made the SOA industry very interesting to watch. There are as many ways to enable services and SOA as there are legacy systems and platforms in your architecture. Of course, bear in mind that we refer to the general case of "services" in this book. Some of your services will be Web services based on XML (eXtensible Markup Language), SOAP, and WSDL, as well as the extended WS-* standards. However, do not limit your SOA total value by examining and considering Web services only. Think services first, and then specific implementation models later.
Integration Fatigue: "There Has to Be a Better Way"
IT "business as usual" is over. Business and IT executives are frustrated with the lack of integration of their internal systems, with their business processes, with their trading partners, and with their customers. We call this integration fatigue. The business is demanding change, and IT executives know there has to be a better way. The frustration with IT as we know it is at an all-time high. IT budgets continue to be stressed, and they rarely increase. The majority of IT budgets are focused on maintaining current systems and keeping the lights on; very little IT budget is focused on strategic initiatives that may pay future dividends. It is a do-more-with-less environment.
Business continues to change while IT is saddled with maintaining the systems and architectures of the past. IT doesn't have the luxury of eliminating its legacy or underperforming assets. Those very IT assets contain business logic that is most likely running a mission-critical portion of the organization's business. Yet while the business logic is mission critical, nonetheless the logic and data are locked up within individual silos of systems and technology. You cannot afford to rewrite the application, and your integration strategy has proved to be supremely costly to implement and maintain. There has to be a better way, and there is. It's called service-oriented architecture.
SOA: EVOLVED ENTERPRISE INTEGRATION
A major impetus for SOA initiatives is solving the age-old problem of integration. For many executives, SOA holds the potential to eliminate, via industry standards and modern tools, the proprietary integration model they've become accustomed to. According to many analyst estimates, up to 30% of a typical IT budget is allocated to integration activities. What would business be like if there was less integration, or, rather, if the integration that was performed was directly related to process integration, enterprise integration, and mergers and acquisitions (M&A) integration? In other words, value-added integration. How would spending less money on integration change an organization's competitive advantage? Could that budget be shifted to more strategic projects?
Origins of the Integration Problem
Where did all this IT complexity come from? Why is 80 to 90% of your IT budget focused backward on maintaining the past rather than looking ahead to supporting the future? This "rearview mirror budgeting" problem is legendary among CIOs and is partly responsible for the lack of strategic IT investment by CIOs today. How backward committed is your IT budget? What percent of the IT budget is allocated to maintaining your legacy investments rather than focused on forward-facing initiatives that may move the organization ahead? This is a real challenge for both business and IT executives today. If you feel as if you're managing your IT budgets using a rearview mirror, you're not alone.
The demand for IT and process integration is driven by business requirements, such as:
Increased M&A activity
Corporate reorganization or restructuring
Application and/or system consolidation
Data integration and data warehousing initiatives
New business strategies leveraging current systems for new processes
Achieving regulatory compliance (e.g., Sarbanes-Oxley, or HIPAA, the Health Insurance Portability and Accountability Act of 1996)
Streamlining of business processes to improve productivity
Addressing the business drivers for integration is a great impetus for SOA and Web services. At what point does IT complexity become an obstacle to business goals and an impediment to achieving IT's goals? We believe that complexity becomes intolerable when organizations are considering or taking these actions:
Hiring a chief architect
Creating a central architecture team
Acquiring or developing your own enterprise application integration (EAI) software
Creating an internal integration team, or a middleware organization, to help solve the integration challenges for your organization
Now, this does not mean that hiring a chief architect is a bad thing. More than likely it is a good thing. Chief architects can help address the architectural complexity and pain your organization is facing. The other actions just listed also can help address these areas. The problems are symptoms of the IT challenges that SOA could address. If you have considered or taken one or more of these actions, your organization is at the point where the integration burden is consuming IT resources, compounding the existing complexity problem, and inhibiting business and IT effectiveness. You most likely have a rearview-mirror IT budget, and you are ready to try a new approach.
Stop Integrating; Service-Enable Instead
Stop integrating now. What we mean is stop integrating the way you have been using proprietary middleware solutions, homegrown point-to-point integration techniques, and tactical integration approaches that are doomed from the outset. These techniques are almost always going to break and require a significant ongoing maintenance burden from the organization.
An organization should integrate without integrating. Eliminate all point-to-point integration projects in your enterprise and rearchitect these initiatives from an SOA point of view. Inventory the integration solutions in your organization. Identify the IT budget allocated to these solutions and projects, including support staff, maintenance, and infrastructure. Determine how many of these integration efforts can be eliminated using reusable services in an SOA.
Wednesday, August 15, 2007
Complexity and Modularity
Overview
Complexity is one of the primary problems that we attempt to address with software development tools and methods. Complexity, if not managed, can cause a product to be low quality or delivered late, over budget, or the project could be cancelled completely. There are no easy solutions to removing complexity and improving productivity and quality. The best we can do is reduce or manage complexity. Modularity is a fundamental principle for achieving simplicity of design and development effort and thus reducing and managing complexity
Complexity is one of the primary problems that we attempt to address with software development tools and methods. Complexity, if not managed, can cause a product to be low quality or delivered late, over budget, or the project could be cancelled completely. There are no easy solutions to removing complexity and improving productivity and quality. The best we can do is reduce or manage complexity. Modularity is a fundamental principle for achieving simplicity of design and development effort and thus reducing and managing complexity
Introduction to Software Design
Overview
Problems in Software Architectural Design
Function, Form, and Fabrication: The Vitruvian Triad
The Scope of Design
The Psychology and Philosophy of Design
General Methodology of Design
Summary
Overview
Higher-quality design, not just higher-quality development processes, is necessary in order to achieve a high-quality product. But what characterizes a higher-quality design, and how do we achieve it? And, more fundamentally, what characterizes an activity as design, and what is the product of design? This presents the fundamental methods of the software design process and establishes the scope of the architectural level of design. The same fundamental design principles and methods apply whether a software architect is analyzing the problem domain, specifying the function of an application to address the problem domain, or designing the structure or form of the software. They also apply when a software engineer is creating a detailed design (including coding) or testing an implementation or designing a development environment in which to create the code. A generalized method of design is presented.
The obstacles to achieving high-quality architectural design in software development are:
A lack of awareness of the importance of architectural design to software development
A lack of understanding of the role of the software architect
A widespread view that designing is an art form, not a technical activity
A lack of understanding of the design process
A lack of design experience in a development organization
Insufficient software architecture design methods and tools
A lack of understanding of how to evaluate designs
Poor communication among stakeholders
Most of these obstacles stem from ignorance around what software architecture and design are and why they are critical to the success of an application or other system development project. This ignorance results in an inability to effectively communicate what problems an application is really addressing and how a technical solution solves those problems.
The solution to the problem of inadequate design must address the obstacles to achieving adequate design. The solution involves the following:
Evangelizing the importance of software architecture
Improving software architecture education
Using architecture methods and tools
General Methodology of Design
In this section, we look at the general elements of a design method based on Pahl and Beitz (Pahl, 1996). Not all design methods include all of these elements, but these elements are present in all design methods:
Purposeful thinking
Analysis
Abstraction
Synthesis
General heuristics
Purposeful Thinking
Design requires systematic thinking, which separates design from routine tasks. Systematic, or discursive, techniques are not the opposite of creative techniques nor are they the opposite of subconscious, or intuitive, thought. Relying on intuition alone, however, is not a good design practice so we need to be more deliberate in our approach to design. A purely intuitive approach to design has several disadvantages: The right solution rarely comes at the right time, the results depend on the architect's skills and talents, and the solutions may be negatively influenced by preconceived ideas (also called fixation). Intuition can lead to good solutions but requires conscious and deliberate involvement with the problem. Discursive thought can stimulate intuitive thinking. Programmers experience this frequently. A programmer may spend hours or days struggling with some problem only to have the solution come to her while she is away from the computer, not even consciously thinking about the problem.
Design should be primarily discursive; that is, it should proceed deliberately in a stepwise fashion. Unconscious procedures can be transformed into deliberate and systematic methods through systematic rules, clear task formulation, and structured design procedures.
Errors are unavoidable so our processes must allow for this. Therefore the architect should analyze the design for errors or weak points in the early stages of development. Clearly defined requirements and problem statements help reduce the risk of design errors by minimizing the amount of guesswork that goes into the design process as well as minimizing the amount of rework that would otherwise be necessary. A discursive design approach helps reduce errors by enforcing systematic testing of design ideas and assumptions early in the development process. Fixed design ideas (assumptions) should be avoided and existing methods and tools should be adapted as necessary (don't let a given methodology artificially constrain you). The deliberate use of specific methods for failure or error identification should be used.
Creativity is inhibited or encouraged by different influences. Some techniques for encouraging creativity are:
Interrupt the activity to create incubation periods (but be careful; too many interruptions can be disruptive).
Apply different solution-finding methods.
Move from abstract to concrete ideas.
Find and collect information from design catalogues (such as design patterns).
Divide work among architecture team members.
Make realistic plans: Realistic planning is found to encourage motivation and creativity, while unrealistic planning is inhibiting.
Analysis
Analysis is the decomposition of complex systems into elements and their interrelationships, identifying essential distinctions, and discarding accidental distinctions. The activities of analysis include identification, definition, and structuring with the purpose of acquiring information about a subject that can be transformed into knowledge. Analytic methods are useful in all stages of software design, from requirements gathering to evaluating designs and technologies to implementation.
Formulating problems clearly and unambiguously, via analysis of an application domain, helps avoid many expensive design and implementation errors. Careful analysis and formulation of problems are two of the most important and effective tools a software architect has. All design methodologies have some analysis aspect. However, I find that many modern design methodologies tend to blur the distinction between analysis and design. Many object-oriented methodologies promote a seamless development concept where the problem or application domain is continuously transformed into a solution, for example, Object Modeling Technique (OMT). In practice, the analysis is rarely done or is not done adequately.
Structured analyses in software, such as the techniques of Gane and Sarson, Yourdon, DeMarco, Ward and Mellor, and the Structured Systems Analysis and Design Methodology (SSADM), are methods for formulating and structuring the problem. Structured analysis is a process where the problem is formulated in terms of data flows, as represented in data flow diagrams (DFDs) and in system state transitions as represented in state transition diagrams. These models emphasize a hierarchical structuring of the functions of a system.
Similarly, object-oriented analysis, such as the techniques of Jacobson, Booch, Rumbaugh (OMT), and Yourdon and Coad, are techniques for structuring the problem domain in terms of objects, their interrelationships, and their class hierarchies. Whereas structured analysis emphasizes a functional hierarchy, object orientation emphasizes a data hierarchy.
Abstraction
Through abstraction, we can infer more general and comprehensive relationships among the elements of our problem. Abstraction reduces the complexity of a problem while emphasizing the essential characteristics of it and aids in the discovery of solutions. Abstractions are not always invented; sometimes they are discovered by using existing abstraction models and attempting to fit the problem into them.
This is an example of predication: A statement is created about the problem abstraction and an existing model such as "problem A (new) is a form of problem B (existing)." An otherwise seemingly novel problem may turn out to be a variation of an existing problem to which some known solutions already exist. By formulating problem A in terms of problem B, the architect then only needs to find suitable abstractions for the remaining part of problem A. A problem can have many abstractions, and, using a discursive approach, the architect can discover several suitable abstractions and apply many existing problem models. Metamodels or reference models are examples of such reusable abstractions.
Analysis aids in the discovery and evaluation of abstractions. Not all analysis results in abstractions, but most abstractions are discovered via analysis.
Synthesis
Synthesis is the combining of individual elements or parts to produce a new effect. It is the integration of solutions to subproblems and the evaluation of the resulting system. When we decompose a problem via analysis into sub-problems and discover solutions to those subproblems, there is still the risk that synthesizing those solutions will not solve the larger problem entirely or, worse, the solutions will be incompatible. Existing abstraction models may also be synthesized to form larger, more specialized abstractions. Synthesis is not only applied to the models of software architecture, it is also applied to the design methods
General Heuristics
In this section, we look at some heuristic design methods. Heuristics are techniques that are characterized as the searching for suitable solutions. These methods apply to many design tasks:
Persistent questions
Negation
Forward steps
Backward steps
Factorization
Systematic variation
Division of labor and collaboration
The Method of Persistent Questions
This method is useful when applying any systematic procedure such as requirements engineering (systems analysis), architectural design, and architectural evaluation. Requirements are almost never complete or at the right level of abstraction and the architect must help the acquirer to discover the essential requirements of the application they are envisioning during the analysis of the requirements. Persistent questioning stimulates ideas and intuition and helps to eliminate assumptions and preconceived notions. The method of persistent questions is a tool that the architect may use when analyzing the problem to discover useful abstractions, as well as a tool for evaluating solutions. An architect or architecture team may even keep a database of standard sets of questions for various purposes. Such a database of questions helps extend the memory of the architect and transfer useful design methods and knowledge to others. Asking questions is considered one of the most important methodological tools, and the technique is found in many existing product development methodologies
The Method of Persistent Questions
This method is useful when applying any systematic procedure such as requirements engineering (systems analysis), architectural design, and architectural evaluation. Requirements are almost never complete or at the right level of abstraction and the architect must help the acquirer to discover the essential requirements of the application they are envisioning during the analysis of the requirements. Persistent questioning stimulates ideas and intuition and helps to eliminate assumptions and preconceived notions. The method of persistent questions is a tool that the architect may use when analyzing the problem to discover useful abstractions, as well as a tool for evaluating solutions. An architect or architecture team may even keep a database of standard sets of questions for various purposes. Such a database of questions helps extend the memory of the architect and transfer useful design methods and knowledge to others. Asking questions is considered one of the most important methodological tools, and the technique is found in many existing product development methodologies
The Method of Forward Steps
The method of forward steps is also known as the method of divergent thought. It starts with a first solution attempt and proceeds to follow as many solution paths as possible, yielding other solutions. The method of forward steps is not necessarily systematic and usually starts with an unsystematic divergence of ideas. For example, the first solution attempt at an application problem may be to apply a client/server architectural style. By applying forward steps, the architect follows several paths such as browser-based client, Web client with browser plug-ins, fat client with embedded browser, or fat client that speaks HTTP with the server.
This method can be applied to any model at any level of abstraction and is typically followed recursively as far down a path as possible. For example, starting with the fat client that speaks HTTP solution, additional application of the method of forward steps could yield solutions like fat client with local storage, fat client with server-side storage only, and fat client with heterogeneous local and server-side storage. The process can follow any path. This is a good way to brainstorm over an object model of the application domain as well as the choice of technologies in the solution domain. This method is similar to the method of systematic variation.
The Method of Backward Steps
The method of backward steps is also known as the method of convergent thought. In this approach, the architect starts with a goal in mind rather than the initial problem. This technique is most useful for setting up an engineering design process for a product and a product development plan after the architecture has been determined. Starting with the final objective of the development effort, all (or a reasonable subset of) possible paths that could have led up to the objective are retraced. This method is useful not only in preparing an engineering process but also for organizing an engineering department around the architecture of a product (known as Conway's law).
The Method of Factorization
This method is the basis of the refactoring technique, popular among software engineers when reworking existing source code to make it more readable, manageable, maintainable, and reusable. The method involves breaking a complex system into less complex elements or factors. Factorization is a type of analysis method. In architecting, this technique is used to find the essential problems being solved (for example, factoring the objectives into subobjectives). Just as in mathematics, the factorization may not be obvious and requires some insight, skill, and lots of practice. In algebra, you may factor an equation by introducing zero as a factor. Similarly, finding that one of the factors of a publishing system is a file-versioning repository may not be obvious (although the experienced architect may see the factorization immediately). Another example is taking what may appear to be an indivisible component and finding a way to divide it into two components and an interface.
The Method of Systematic Variation
The method of systematic variation of solutions may have originated with Leonardo DaVinci. DaVinci kept meticulous notes on his ideas and inventions and applied this technique to derive multiple variations of ideas, each varying only by a single characteristic from the former. The method starts with a generalized classification structure, such as a class hierarchy, that represents the various problem characteristics and possible solutions. By systematically varying a single characteristic, the architect may discover more optimized solutions. The approach yields a solution field. Evaluation techniques can be applied to each solution to determine its suitability.
Division of Labor and Collaboration
The study of human factors has found that implementing large and complex tasks requires a division of labor. The more specialized the task the more important the division of labor. Software development tasks are commonly based not only on functional areas of the system but also on the technology used. We typically have user interface designers, user interface engineers, middle-tier engineers, back-end or database engineers, test engineers, configuration management engineers, and so on. A complex application or system also benefits from a division of labor with respect to the architecture. For example, it seems natural to separate the tasks of requirements engineering, functional specification, interaction design, and structural design, all of which are in the scope of architecture. However, whenever there is a division of labor there is an information exchange problem. Systematic methods and the creation of models can help overcome this problem.
Problems in Software Architectural Design
Function, Form, and Fabrication: The Vitruvian Triad
The Scope of Design
The Psychology and Philosophy of Design
General Methodology of Design
Summary
Overview
Higher-quality design, not just higher-quality development processes, is necessary in order to achieve a high-quality product. But what characterizes a higher-quality design, and how do we achieve it? And, more fundamentally, what characterizes an activity as design, and what is the product of design? This presents the fundamental methods of the software design process and establishes the scope of the architectural level of design. The same fundamental design principles and methods apply whether a software architect is analyzing the problem domain, specifying the function of an application to address the problem domain, or designing the structure or form of the software. They also apply when a software engineer is creating a detailed design (including coding) or testing an implementation or designing a development environment in which to create the code. A generalized method of design is presented.
The obstacles to achieving high-quality architectural design in software development are:
A lack of awareness of the importance of architectural design to software development
A lack of understanding of the role of the software architect
A widespread view that designing is an art form, not a technical activity
A lack of understanding of the design process
A lack of design experience in a development organization
Insufficient software architecture design methods and tools
A lack of understanding of how to evaluate designs
Poor communication among stakeholders
Most of these obstacles stem from ignorance around what software architecture and design are and why they are critical to the success of an application or other system development project. This ignorance results in an inability to effectively communicate what problems an application is really addressing and how a technical solution solves those problems.
The solution to the problem of inadequate design must address the obstacles to achieving adequate design. The solution involves the following:
Evangelizing the importance of software architecture
Improving software architecture education
Using architecture methods and tools
General Methodology of Design
In this section, we look at the general elements of a design method based on Pahl and Beitz (Pahl, 1996). Not all design methods include all of these elements, but these elements are present in all design methods:
Purposeful thinking
Analysis
Abstraction
Synthesis
General heuristics
Purposeful Thinking
Design requires systematic thinking, which separates design from routine tasks. Systematic, or discursive, techniques are not the opposite of creative techniques nor are they the opposite of subconscious, or intuitive, thought. Relying on intuition alone, however, is not a good design practice so we need to be more deliberate in our approach to design. A purely intuitive approach to design has several disadvantages: The right solution rarely comes at the right time, the results depend on the architect's skills and talents, and the solutions may be negatively influenced by preconceived ideas (also called fixation). Intuition can lead to good solutions but requires conscious and deliberate involvement with the problem. Discursive thought can stimulate intuitive thinking. Programmers experience this frequently. A programmer may spend hours or days struggling with some problem only to have the solution come to her while she is away from the computer, not even consciously thinking about the problem.
Design should be primarily discursive; that is, it should proceed deliberately in a stepwise fashion. Unconscious procedures can be transformed into deliberate and systematic methods through systematic rules, clear task formulation, and structured design procedures.
Errors are unavoidable so our processes must allow for this. Therefore the architect should analyze the design for errors or weak points in the early stages of development. Clearly defined requirements and problem statements help reduce the risk of design errors by minimizing the amount of guesswork that goes into the design process as well as minimizing the amount of rework that would otherwise be necessary. A discursive design approach helps reduce errors by enforcing systematic testing of design ideas and assumptions early in the development process. Fixed design ideas (assumptions) should be avoided and existing methods and tools should be adapted as necessary (don't let a given methodology artificially constrain you). The deliberate use of specific methods for failure or error identification should be used.
Creativity is inhibited or encouraged by different influences. Some techniques for encouraging creativity are:
Interrupt the activity to create incubation periods (but be careful; too many interruptions can be disruptive).
Apply different solution-finding methods.
Move from abstract to concrete ideas.
Find and collect information from design catalogues (such as design patterns).
Divide work among architecture team members.
Make realistic plans: Realistic planning is found to encourage motivation and creativity, while unrealistic planning is inhibiting.
Analysis
Analysis is the decomposition of complex systems into elements and their interrelationships, identifying essential distinctions, and discarding accidental distinctions. The activities of analysis include identification, definition, and structuring with the purpose of acquiring information about a subject that can be transformed into knowledge. Analytic methods are useful in all stages of software design, from requirements gathering to evaluating designs and technologies to implementation.
Formulating problems clearly and unambiguously, via analysis of an application domain, helps avoid many expensive design and implementation errors. Careful analysis and formulation of problems are two of the most important and effective tools a software architect has. All design methodologies have some analysis aspect. However, I find that many modern design methodologies tend to blur the distinction between analysis and design. Many object-oriented methodologies promote a seamless development concept where the problem or application domain is continuously transformed into a solution, for example, Object Modeling Technique (OMT). In practice, the analysis is rarely done or is not done adequately.
Structured analyses in software, such as the techniques of Gane and Sarson, Yourdon, DeMarco, Ward and Mellor, and the Structured Systems Analysis and Design Methodology (SSADM), are methods for formulating and structuring the problem. Structured analysis is a process where the problem is formulated in terms of data flows, as represented in data flow diagrams (DFDs) and in system state transitions as represented in state transition diagrams. These models emphasize a hierarchical structuring of the functions of a system.
Similarly, object-oriented analysis, such as the techniques of Jacobson, Booch, Rumbaugh (OMT), and Yourdon and Coad, are techniques for structuring the problem domain in terms of objects, their interrelationships, and their class hierarchies. Whereas structured analysis emphasizes a functional hierarchy, object orientation emphasizes a data hierarchy.
Abstraction
Through abstraction, we can infer more general and comprehensive relationships among the elements of our problem. Abstraction reduces the complexity of a problem while emphasizing the essential characteristics of it and aids in the discovery of solutions. Abstractions are not always invented; sometimes they are discovered by using existing abstraction models and attempting to fit the problem into them.
This is an example of predication: A statement is created about the problem abstraction and an existing model such as "problem A (new) is a form of problem B (existing)." An otherwise seemingly novel problem may turn out to be a variation of an existing problem to which some known solutions already exist. By formulating problem A in terms of problem B, the architect then only needs to find suitable abstractions for the remaining part of problem A. A problem can have many abstractions, and, using a discursive approach, the architect can discover several suitable abstractions and apply many existing problem models. Metamodels or reference models are examples of such reusable abstractions.
Analysis aids in the discovery and evaluation of abstractions. Not all analysis results in abstractions, but most abstractions are discovered via analysis.
Synthesis
Synthesis is the combining of individual elements or parts to produce a new effect. It is the integration of solutions to subproblems and the evaluation of the resulting system. When we decompose a problem via analysis into sub-problems and discover solutions to those subproblems, there is still the risk that synthesizing those solutions will not solve the larger problem entirely or, worse, the solutions will be incompatible. Existing abstraction models may also be synthesized to form larger, more specialized abstractions. Synthesis is not only applied to the models of software architecture, it is also applied to the design methods
General Heuristics
In this section, we look at some heuristic design methods. Heuristics are techniques that are characterized as the searching for suitable solutions. These methods apply to many design tasks:
Persistent questions
Negation
Forward steps
Backward steps
Factorization
Systematic variation
Division of labor and collaboration
The Method of Persistent Questions
This method is useful when applying any systematic procedure such as requirements engineering (systems analysis), architectural design, and architectural evaluation. Requirements are almost never complete or at the right level of abstraction and the architect must help the acquirer to discover the essential requirements of the application they are envisioning during the analysis of the requirements. Persistent questioning stimulates ideas and intuition and helps to eliminate assumptions and preconceived notions. The method of persistent questions is a tool that the architect may use when analyzing the problem to discover useful abstractions, as well as a tool for evaluating solutions. An architect or architecture team may even keep a database of standard sets of questions for various purposes. Such a database of questions helps extend the memory of the architect and transfer useful design methods and knowledge to others. Asking questions is considered one of the most important methodological tools, and the technique is found in many existing product development methodologies
The Method of Persistent Questions
This method is useful when applying any systematic procedure such as requirements engineering (systems analysis), architectural design, and architectural evaluation. Requirements are almost never complete or at the right level of abstraction and the architect must help the acquirer to discover the essential requirements of the application they are envisioning during the analysis of the requirements. Persistent questioning stimulates ideas and intuition and helps to eliminate assumptions and preconceived notions. The method of persistent questions is a tool that the architect may use when analyzing the problem to discover useful abstractions, as well as a tool for evaluating solutions. An architect or architecture team may even keep a database of standard sets of questions for various purposes. Such a database of questions helps extend the memory of the architect and transfer useful design methods and knowledge to others. Asking questions is considered one of the most important methodological tools, and the technique is found in many existing product development methodologies
The Method of Forward Steps
The method of forward steps is also known as the method of divergent thought. It starts with a first solution attempt and proceeds to follow as many solution paths as possible, yielding other solutions. The method of forward steps is not necessarily systematic and usually starts with an unsystematic divergence of ideas. For example, the first solution attempt at an application problem may be to apply a client/server architectural style. By applying forward steps, the architect follows several paths such as browser-based client, Web client with browser plug-ins, fat client with embedded browser, or fat client that speaks HTTP with the server.
This method can be applied to any model at any level of abstraction and is typically followed recursively as far down a path as possible. For example, starting with the fat client that speaks HTTP solution, additional application of the method of forward steps could yield solutions like fat client with local storage, fat client with server-side storage only, and fat client with heterogeneous local and server-side storage. The process can follow any path. This is a good way to brainstorm over an object model of the application domain as well as the choice of technologies in the solution domain. This method is similar to the method of systematic variation.
The Method of Backward Steps
The method of backward steps is also known as the method of convergent thought. In this approach, the architect starts with a goal in mind rather than the initial problem. This technique is most useful for setting up an engineering design process for a product and a product development plan after the architecture has been determined. Starting with the final objective of the development effort, all (or a reasonable subset of) possible paths that could have led up to the objective are retraced. This method is useful not only in preparing an engineering process but also for organizing an engineering department around the architecture of a product (known as Conway's law).
The Method of Factorization
This method is the basis of the refactoring technique, popular among software engineers when reworking existing source code to make it more readable, manageable, maintainable, and reusable. The method involves breaking a complex system into less complex elements or factors. Factorization is a type of analysis method. In architecting, this technique is used to find the essential problems being solved (for example, factoring the objectives into subobjectives). Just as in mathematics, the factorization may not be obvious and requires some insight, skill, and lots of practice. In algebra, you may factor an equation by introducing zero as a factor. Similarly, finding that one of the factors of a publishing system is a file-versioning repository may not be obvious (although the experienced architect may see the factorization immediately). Another example is taking what may appear to be an indivisible component and finding a way to divide it into two components and an interface.
The Method of Systematic Variation
The method of systematic variation of solutions may have originated with Leonardo DaVinci. DaVinci kept meticulous notes on his ideas and inventions and applied this technique to derive multiple variations of ideas, each varying only by a single characteristic from the former. The method starts with a generalized classification structure, such as a class hierarchy, that represents the various problem characteristics and possible solutions. By systematically varying a single characteristic, the architect may discover more optimized solutions. The approach yields a solution field. Evaluation techniques can be applied to each solution to determine its suitability.
Division of Labor and Collaboration
The study of human factors has found that implementing large and complex tasks requires a division of labor. The more specialized the task the more important the division of labor. Software development tasks are commonly based not only on functional areas of the system but also on the technology used. We typically have user interface designers, user interface engineers, middle-tier engineers, back-end or database engineers, test engineers, configuration management engineers, and so on. A complex application or system also benefits from a division of labor with respect to the architecture. For example, it seems natural to separate the tasks of requirements engineering, functional specification, interaction design, and structural design, all of which are in the scope of architecture. However, whenever there is a division of labor there is an information exchange problem. Systematic methods and the creation of models can help overcome this problem.
The Architecture Design Process
the architectural view of software development is composed of a predesign phase, a domain analysis phase, a schematic design phase, and a design development phase. The focus of each phase is on a different, but possibly overlapping, set of problems. In the predesign phase, the architect is concerned mostly with forming and understanding the enterprise context in which an application will exist. This is represented by a model that depicts the system as an entity within a community of other entities, such as other software systems and human users. During the domain analysis phase, the application requirements are analyzed and structured. In particular, the requirements are those around the problem domain but not specific to the domain of software itself, such as user interface or HCI-related requirements. During the schematic design phase, the architect begins to develop the solution models such as identifying the modules of the system and the design rules that establish the boundaries between modules. Modules are discrete units of design work that rely on shared information and also have hidden information. Finally, during the design development phase, the architecture is refined and possibly multiple variations are created in order to explore the best fi
The basic architecture design process is composed of these steps:
Understand the problem.
Identify design elements and their relationships.
Evaluate the architecture design.
Transform the architecture design.
The first step is arguably the most crucial because it affects the quality of the design that follows. Without a clear understanding of the problem, it is not possible to create an effective solution.
In the second step, we identify design elements and their interdependencies. In the early phases of the design project, we perform a naive functional decomposition of the application, which establishes a baseline for future design tasks and design transformation. An example of this method can be seen in Jan Bosch's Design and Use of Software Architectures, which focuses on product-line architectures for real-time and embedded software systems. In Bosch's model, this step is called functionality-based architectural design (Bosch, 2000).
The third step involves assessing the architecture for conformance to architectural quality attribute requirements. The functional behavior of the application cannot be ideally tested from an architectural decomposition. However, many other quality attributes can be assessed by inspecting the design or by implementing prototypes of the architecturally significant component interactions.
The fourth step involves the application of design operations to transform the architecture design into a new design that addresses the quality attribute requirements better than the previous design. The phase may be repeated multiple times and even performed recursively
Identifying Design Elements and Their Relationships
In this step, we establish a baseline decomposition that initially splits the system based on functional requirements. The decomposition can be modeled using a design structure matrix (DSM), which represents the dependencies between design elements without prescribing the granularity of the elements. A first draft of this model could be created during the predesign phase, and then revised during subsequent steps.
Evaluating the Architecture
The third step involves evaluating the architectural design to determine if it satisfies the architectural requirements. The design is evaluated in order to gather qualitative measures or quantitative data, both of which are metrics
Transforming the Architecture
This step is performed after an evaluation or informal assessment of the architectural design. If the architectural design does not fully satisfy its quality attribute requirements, then it must be changed until it does satisfy them. However, instead of starting from scratch, we can take the existing design and apply design operators to it in order to transform it. The new version of the design is then assessed and the process continues until a satisfactory design is achieved. Sometimes you may need to reverse (undo) operations that were already applied and go down a different design path. You may also be creating several candidate designs at the same time. Starting with an initial root design (perhaps a single monolithic module), you may create a directed graph of designs. Each node represents a design decision path. Some paths may merge, in which case the application of design operations are commutative. In this case, you may transform some of the designs and eliminate ones that are obviously not going to transform well based on the requirements.
A design is transformed by applying design operators, styles, or patterns. There are two types of design operators: those that affect the modular architecture and those that affect the component architecture. There are six modular operators (Baldwin, 2000):
Splitting a design into two or more modules
Substituting one design module for another
Augmenting the system by adding a new module
Excluding a module from the system
Inverting a module to create new interfaces (design rules)
Porting a module to another system
Similar to the modular operators are design operators. The basic design operators identified by Bass (Bass, 1998) are:
Decomposition of a system into components.
Replication of components to improve reliability.
Compression of two or more components into a single component to improve performance.
Abstraction of a component to improve modifiability and adaptability.
Resource sharing, or the sharing of a single component instance with other components to improve integrability (the ability to integrate applications and systems), portability, and modifiability.
Modular and design operators are very similar. The modular operators are concerned with the module view of the system: those units of individual design and development that primarily affect the nonoperational qualities. The design operators are concerned with the runtime component view of the system, which are those units of execution that primarily affect the operational qualities
The basic architecture design process is composed of these steps:
Understand the problem.
Identify design elements and their relationships.
Evaluate the architecture design.
Transform the architecture design.
The first step is arguably the most crucial because it affects the quality of the design that follows. Without a clear understanding of the problem, it is not possible to create an effective solution.
In the second step, we identify design elements and their interdependencies. In the early phases of the design project, we perform a naive functional decomposition of the application, which establishes a baseline for future design tasks and design transformation. An example of this method can be seen in Jan Bosch's Design and Use of Software Architectures, which focuses on product-line architectures for real-time and embedded software systems. In Bosch's model, this step is called functionality-based architectural design (Bosch, 2000).
The third step involves assessing the architecture for conformance to architectural quality attribute requirements. The functional behavior of the application cannot be ideally tested from an architectural decomposition. However, many other quality attributes can be assessed by inspecting the design or by implementing prototypes of the architecturally significant component interactions.
The fourth step involves the application of design operations to transform the architecture design into a new design that addresses the quality attribute requirements better than the previous design. The phase may be repeated multiple times and even performed recursively
Identifying Design Elements and Their Relationships
In this step, we establish a baseline decomposition that initially splits the system based on functional requirements. The decomposition can be modeled using a design structure matrix (DSM), which represents the dependencies between design elements without prescribing the granularity of the elements. A first draft of this model could be created during the predesign phase, and then revised during subsequent steps.
Evaluating the Architecture
The third step involves evaluating the architectural design to determine if it satisfies the architectural requirements. The design is evaluated in order to gather qualitative measures or quantitative data, both of which are metrics
Transforming the Architecture
This step is performed after an evaluation or informal assessment of the architectural design. If the architectural design does not fully satisfy its quality attribute requirements, then it must be changed until it does satisfy them. However, instead of starting from scratch, we can take the existing design and apply design operators to it in order to transform it. The new version of the design is then assessed and the process continues until a satisfactory design is achieved. Sometimes you may need to reverse (undo) operations that were already applied and go down a different design path. You may also be creating several candidate designs at the same time. Starting with an initial root design (perhaps a single monolithic module), you may create a directed graph of designs. Each node represents a design decision path. Some paths may merge, in which case the application of design operations are commutative. In this case, you may transform some of the designs and eliminate ones that are obviously not going to transform well based on the requirements.
A design is transformed by applying design operators, styles, or patterns. There are two types of design operators: those that affect the modular architecture and those that affect the component architecture. There are six modular operators (Baldwin, 2000):
Splitting a design into two or more modules
Substituting one design module for another
Augmenting the system by adding a new module
Excluding a module from the system
Inverting a module to create new interfaces (design rules)
Porting a module to another system
Similar to the modular operators are design operators. The basic design operators identified by Bass (Bass, 1998) are:
Decomposition of a system into components.
Replication of components to improve reliability.
Compression of two or more components into a single component to improve performance.
Abstraction of a component to improve modifiability and adaptability.
Resource sharing, or the sharing of a single component instance with other components to improve integrability (the ability to integrate applications and systems), portability, and modifiability.
Modular and design operators are very similar. The modular operators are concerned with the module view of the system: those units of individual design and development that primarily affect the nonoperational qualities. The design operators are concerned with the runtime component view of the system, which are those units of execution that primarily affect the operational qualities
Software architecting involves
Software architecting involves the design of a system from multiple viewpoints. The common viewpoints used in software engineering are the technology stack (or physical) view, the object (or data) model, and the use case (or behavioral) view. These viewpoints are useful and necessary because they capture many types of design decisions and represent many system qualities such as functionality, information, and physical construction. They do not represent many other important system quality attributes such as modifiability, buildability, security, reliability, and performance, nor do they represent non-operational or business-oriented qualities such as the ability to reduce development and maintenance costs.
The problem with representing an architecture with this single technology-focused view is that we only see a vertical slice through a multidimensional system. Many architectural decisions cannot be represented in this view. If this is the only view we create, then we will probably neglect the other views to the detriment of the system itself.
There are a couple of philosophies concerning how to improve the software crisis. One approach is to improve the quality of the software development process. In this school of thought, quality can be improved by using iterative development techniques, rapid application development (RAD) tools, frequent integration and testing, and keeping careful records so that an organization can build up historical data that will aid in improving the process in future product cycles. It uses iterative/increment development processes like the Rational Unified Process and the Capability Maturity Model (CMM). Another approach for improving software quality is to stay away from the heavyweight planning-oriented processes and instead adopt agile processes and use of techniques such as RAD and eXtreme Programming (XP).
The problem with representing an architecture with this single technology-focused view is that we only see a vertical slice through a multidimensional system. Many architectural decisions cannot be represented in this view. If this is the only view we create, then we will probably neglect the other views to the detriment of the system itself.
There are a couple of philosophies concerning how to improve the software crisis. One approach is to improve the quality of the software development process. In this school of thought, quality can be improved by using iterative development techniques, rapid application development (RAD) tools, frequent integration and testing, and keeping careful records so that an organization can build up historical data that will aid in improving the process in future product cycles. It uses iterative/increment development processes like the Rational Unified Process and the Capability Maturity Model (CMM). Another approach for improving software quality is to stay away from the heavyweight planning-oriented processes and instead adopt agile processes and use of techniques such as RAD and eXtreme Programming (XP).
Monday, August 13, 2007
Lightweight Enterprise Architectures
Lightweight Enterprise Architecture (LEA) combines the successful elements of architecture implemented in organizations today, provides a simple framework that does not burden the organization, and offers quicker results. LEA is not a rigid architecture that requires a long learning cycle, but provides a solid framework to more successfully evolve the technical landscape of most organizations today. LEA is a simpler set of responsibilities and deliverables for the architect to facilitate better communication between the resources in an organization.
In addition, the success of integrating technology is not simply building a technical masterpiece, as there are many other factors of success. First, technology needs to offer the organization the needed functionality to deliver a product or service to the marketplace with minimal effort. This means that technology needs alignment with the direction of the corporate strategy and fit within the current infrastructure of the organization. Technology is not just building a better technical solution, but understanding the processes, people, and goals of the organization — all within the limits of striving for profitability. This is the core goal of successful enterprise architecture and the force behind the development of LEA.
Enterprise architecture as defined through LEA focuses on aligning all the systems in the organization to meet the needs of business from a holistic business perspective. It is the goal of enterprise architecture to deploy all the components of systems together as a whole, versus optimizing a particular aspect of technology. That is, it is more important to deliver a better total solution even through the sacrifice of one or more parts of the system. Specific architectures (e.g., application architecture, security architecture, etc.) are concerned with optimal design and evolution of their particular domain. These specific architectures in LEA play a support role and are critical to the successful execution of enterprise architecture but do not drive the enterprise architecture.
LEA bridges the gap of business vision and the technical community through a simple framework of architectural responsibilities. These responsibilities require a minimal set of deliverables other resources use (or consume) that progressively transforms the visions of the enterprise into structured, executable artifacts for technology. The simplicity of LEA is that in progressive stages of enterprise architecture, the artifacts support and build into the next set of artifacts. By keeping to a minimal set and building the artifacts onto each other, LEA is a simple framework to understand. This ultimately increases the chance of success as this design is more usable by diverse resources, simplifies communications, and reduces drifts of interpretation from vision to implementation
To better help the architect with these various levels of responsibility, LEA frames these into three separate and interrelated perspectives. LEA consists of three realms (or perspectives) of architecture that include Strategic Architecture, Conceptual Architecture, and Execution Architecture.
The Strategic Architecture focuses on building strategic principles and developing guidelines by working closely with the leadership of an organization. Taking input from the business vision and goals, the Strategic Architecture frames the measures and practices for the enterprise's technology. Essentially, the Strategic Architecture sets the direction for LEA to ensure effort on technology is consistent with the needs and visions of the enterprise.
Often in organizations, some of these activities are in the IT strategic plan. However, IT strategic plans often are vehicles promoting the budget requirements of a department versus an enterprise view. In LEA, the Strategic Architecture is a continuous, ongoing process to translate business strategy and provide the framework guiding technology stewardship as best interpreted from leadership.
Strategic Architecture
The Strategic Architecture goal is to capture the vision of leadership that translates readily for adoption in system design. Many strategy plans are well designed but are written by and mainly for the adoption of the business owners. Very few strategies offer the perspective of how to change systems, but rather focus on the intended outcome — either functional or budgetary needs. Some strategies provide some indication of how systems should change, but these are again mainly to reflect the business needs with minimal account for the technological impacts.
However, this does not mean the reformulation of the strategic plan to benefit technology, but rather the need for the translation of the strategic plan into tangible guides for technology to be successful
Business Strategy and LEA
In the Strategic Architecture, the architect needs good interaction with leadership and the business owners to best understand their stake in the future of the enterprise. This often entails participation in strategy meetings and identifying impacts on technology and the potential effect on various strategic scenarios. However, the key to a successful Strategic Architecture is the development of a core Strategic Architecture that remains consistent over time.
A good Strategic Architecture contains the following three fundamental components:
Enterprise Principles
Enterprise Patterns
Enterprise Guidelines
In addition, the success of integrating technology is not simply building a technical masterpiece, as there are many other factors of success. First, technology needs to offer the organization the needed functionality to deliver a product or service to the marketplace with minimal effort. This means that technology needs alignment with the direction of the corporate strategy and fit within the current infrastructure of the organization. Technology is not just building a better technical solution, but understanding the processes, people, and goals of the organization — all within the limits of striving for profitability. This is the core goal of successful enterprise architecture and the force behind the development of LEA.
Enterprise architecture as defined through LEA focuses on aligning all the systems in the organization to meet the needs of business from a holistic business perspective. It is the goal of enterprise architecture to deploy all the components of systems together as a whole, versus optimizing a particular aspect of technology. That is, it is more important to deliver a better total solution even through the sacrifice of one or more parts of the system. Specific architectures (e.g., application architecture, security architecture, etc.) are concerned with optimal design and evolution of their particular domain. These specific architectures in LEA play a support role and are critical to the successful execution of enterprise architecture but do not drive the enterprise architecture.
LEA bridges the gap of business vision and the technical community through a simple framework of architectural responsibilities. These responsibilities require a minimal set of deliverables other resources use (or consume) that progressively transforms the visions of the enterprise into structured, executable artifacts for technology. The simplicity of LEA is that in progressive stages of enterprise architecture, the artifacts support and build into the next set of artifacts. By keeping to a minimal set and building the artifacts onto each other, LEA is a simple framework to understand. This ultimately increases the chance of success as this design is more usable by diverse resources, simplifies communications, and reduces drifts of interpretation from vision to implementation
To better help the architect with these various levels of responsibility, LEA frames these into three separate and interrelated perspectives. LEA consists of three realms (or perspectives) of architecture that include Strategic Architecture, Conceptual Architecture, and Execution Architecture.
The Strategic Architecture focuses on building strategic principles and developing guidelines by working closely with the leadership of an organization. Taking input from the business vision and goals, the Strategic Architecture frames the measures and practices for the enterprise's technology. Essentially, the Strategic Architecture sets the direction for LEA to ensure effort on technology is consistent with the needs and visions of the enterprise.
Often in organizations, some of these activities are in the IT strategic plan. However, IT strategic plans often are vehicles promoting the budget requirements of a department versus an enterprise view. In LEA, the Strategic Architecture is a continuous, ongoing process to translate business strategy and provide the framework guiding technology stewardship as best interpreted from leadership.
Strategic Architecture
The Strategic Architecture goal is to capture the vision of leadership that translates readily for adoption in system design. Many strategy plans are well designed but are written by and mainly for the adoption of the business owners. Very few strategies offer the perspective of how to change systems, but rather focus on the intended outcome — either functional or budgetary needs. Some strategies provide some indication of how systems should change, but these are again mainly to reflect the business needs with minimal account for the technological impacts.
However, this does not mean the reformulation of the strategic plan to benefit technology, but rather the need for the translation of the strategic plan into tangible guides for technology to be successful
Business Strategy and LEA
In the Strategic Architecture, the architect needs good interaction with leadership and the business owners to best understand their stake in the future of the enterprise. This often entails participation in strategy meetings and identifying impacts on technology and the potential effect on various strategic scenarios. However, the key to a successful Strategic Architecture is the development of a core Strategic Architecture that remains consistent over time.
A good Strategic Architecture contains the following three fundamental components:
Enterprise Principles
Enterprise Patterns
Enterprise Guidelines
Agent Toolkits
Agent-based applications need a significant amount of enabling infrastructure before even one message is exchanged between agents. A large set of supporting services must be made available throughout an agent-based system before the application developer can move on to focus on the application domain, be it e-business, Grid computing, or Ambient Intelligence. Such services range from basic communication to discovery, coordination, security, and so on, and come together to provide an environment that can support agent-based computing. In this way, they can be considered as providing the operating system for agents
Agent Architectures : OTHER APPROACHES
AGENTO and PLACA
In the examples so far, the focus has been on agent architectures to support the design of agent-based systems. Another approach to building agents is to design a programming language with semantics based on some theory of rational or intentional agency, such as the BDI model, and to program the desired behavior of individual agents directly using mental attitudes. Such a technique is referred to as agent-oriented programming (AOP). An agent-oriented program comprises a set of transition rules that specify how an agent in a given mental state will respond to an input, which may be a set of messages from other agents, by defining its new mental state and any outputs.
The language we describe here is PLACA [44], which extends the expressive power of AGENTO [45] developed by Shoham, who introduced the concept of agent-oriented programs. We do not describe PLACA in detail, but include it simply as an example of an alternative approach, especially because of its formal nature, which provides a counterpoint to the earlier architectures in this chapter. This section thus only includes brief summaries, and the interested reader can explore further through the references.
When it was first introduced, AGENTO (and later its successor, PLACA) represented a new programming paradigm that supported the notion of a societal view of computation in which agents would interact with each other in order to achieve their individual goals. Shoham asserted that an approach that could enable a system of agents to be built in this way needs three particular elements, as follows:
It requires a formal logical language to describe the mental state of each agent. Such mental state consists of beliefs and intentions, as we have seen in other systems described in this chapter.
It also needs a programming language with which to specify the agents and their behavior.
Finally, it needs a method for adapting nonagent legacy systems so that they can be incorporated into an agent system, thereby allowing agents to communicate with them by attributing beliefs and intentions.
A program in PLACA is defined by an initial mental state and a set of mental-state rules specifying how the mental state changes in various scenarios. At run time, an agent's state consists of capabilities and its mental state, which comprises beliefs, intentions, and plans (where plans, and possibly intentions, are initially empty). At every step, an agent collects messages that have been received from others from the input buffer, clears this buffer, and updates its mental state according to its defining program.
At the beginning of each execution step, those transition rules that are satisfied in the current state are identified, and applied to the current mental state and messages collected from the input buffer. Once the mental state is updated, messages that need to be sent are placed in the output buffer, and actions that need to be performed are recorded and executed in the next step. If there is sufficient time before the next tick of the clock, the planner may construct and refine current plans for satisfying intentions. A basic representation of the PLACA interpreter is shown in Figure 2.7.
Figure 2.7: The PLACA interpreter. (After: [44].)
PLACA is based on a modal logic specification of the relationship between the mental-state components. Its designers claim that PLACA agents satisfy all the axioms provided, but the relationship between the logic and the programming language is not well-defined. This is a persistent problem in the agent field: how to determine the correspondence between computational systems on the one hand, and formal models, typically constructed in modal logic, on the other. One approach to this problem is to carefully design temporal logics so that agents can be described using logic formulas that are directly executable, as we will see next.
2.6.2 Concurrent METATEM
The same societal view of computation as in AOP is also present in Concurrent METATEM [46], a programming language in which each agent is defined as a concurrently executing process, communicating with other such processes via message-passing. The distinct characteristic of this work is that agents are specified using a temporal logic that is directly executable. To directly execute a specification written in a temporal logic formula, the interpreter attempts to construct a model structure in which the agent specification is satisfied. However, since the environment is dynamic, it may constantly be altering that model, so that responses must be made to what has just become true or untrue because of this dynamism.
The way in which the model is constructed in a dynamic environment is referred to as the execution strategy. When a Concurrent METATEM program is executed, it produces a sequence of temporally ordered states, each labeled with a model structure stating those propositions that are true. This trace provides Concurrent METATEM specifications with a concrete computational interpretation.
The basic form of a specification in Concurrent METATEM is as follows: "on the basis of what has happened in the past do something in the future." Thus, each agent is specified by a set of rules of the form:
and at each cycle, the precedent of each rule is matched against an internal history, firing if a match can be found. Once a rule fires, the agent is committed to the antecedent, which typically involves trying to make some predicate true.
As an example, consider the following Concurrent METATEM program, specifying the behavior of a controller agent solely responsible for supplying an infinitely renewable resource [47]. While the resource cannot be used by two agents at the same time, it is possible that the controller may be asked by two different agents for the resource simultaneously. The predicates ask(x) and give(x) mean that agent x has been asked for the resource, and that agent x has been given the resource, respectively.
In temporal logic, the formula, ○Form, is satisfied at present if Form is satisfied at the next time moment. Similarly, ◊Form is satisfied at present if Form is true at some time in the future, and FG is satisfied if F is true since G has been true.
The specification of the controller agent is defined in Concurrent METATEM as follows:
These three formulas can be interpreted in English as follows:
If an agent asks, then eventually give the resource to that agent;
Do not give to anyone unless they have asked since you last gave to them;
If you give to two people, then they must be the same person.
Concurrent METATEM is attractive because it is directly executable so that no time-consuming and error-prone refinement is required in the process from specification to implementation. In addition, specifications have a concrete computational semantics defined by the sequence of models that arise through the constant interplay of the dynamic environment and the action of the agent. However, since the execution of Concurrent METATEM is based on theorem-proving, some standard problems arise such as the undecidability of first-order logic and the complexity involved in simple propositional logic. In addition, it is not clear how other modal operators commonly associated with agents such as belief, desire, and intention could be incorporated into the language while maintaining its directly executable property.
In the examples so far, the focus has been on agent architectures to support the design of agent-based systems. Another approach to building agents is to design a programming language with semantics based on some theory of rational or intentional agency, such as the BDI model, and to program the desired behavior of individual agents directly using mental attitudes. Such a technique is referred to as agent-oriented programming (AOP). An agent-oriented program comprises a set of transition rules that specify how an agent in a given mental state will respond to an input, which may be a set of messages from other agents, by defining its new mental state and any outputs.
The language we describe here is PLACA [44], which extends the expressive power of AGENTO [45] developed by Shoham, who introduced the concept of agent-oriented programs. We do not describe PLACA in detail, but include it simply as an example of an alternative approach, especially because of its formal nature, which provides a counterpoint to the earlier architectures in this chapter. This section thus only includes brief summaries, and the interested reader can explore further through the references.
When it was first introduced, AGENTO (and later its successor, PLACA) represented a new programming paradigm that supported the notion of a societal view of computation in which agents would interact with each other in order to achieve their individual goals. Shoham asserted that an approach that could enable a system of agents to be built in this way needs three particular elements, as follows:
It requires a formal logical language to describe the mental state of each agent. Such mental state consists of beliefs and intentions, as we have seen in other systems described in this chapter.
It also needs a programming language with which to specify the agents and their behavior.
Finally, it needs a method for adapting nonagent legacy systems so that they can be incorporated into an agent system, thereby allowing agents to communicate with them by attributing beliefs and intentions.
A program in PLACA is defined by an initial mental state and a set of mental-state rules specifying how the mental state changes in various scenarios. At run time, an agent's state consists of capabilities and its mental state, which comprises beliefs, intentions, and plans (where plans, and possibly intentions, are initially empty). At every step, an agent collects messages that have been received from others from the input buffer, clears this buffer, and updates its mental state according to its defining program.
At the beginning of each execution step, those transition rules that are satisfied in the current state are identified, and applied to the current mental state and messages collected from the input buffer. Once the mental state is updated, messages that need to be sent are placed in the output buffer, and actions that need to be performed are recorded and executed in the next step. If there is sufficient time before the next tick of the clock, the planner may construct and refine current plans for satisfying intentions. A basic representation of the PLACA interpreter is shown in Figure 2.7.
Figure 2.7: The PLACA interpreter. (After: [44].)
PLACA is based on a modal logic specification of the relationship between the mental-state components. Its designers claim that PLACA agents satisfy all the axioms provided, but the relationship between the logic and the programming language is not well-defined. This is a persistent problem in the agent field: how to determine the correspondence between computational systems on the one hand, and formal models, typically constructed in modal logic, on the other. One approach to this problem is to carefully design temporal logics so that agents can be described using logic formulas that are directly executable, as we will see next.
2.6.2 Concurrent METATEM
The same societal view of computation as in AOP is also present in Concurrent METATEM [46], a programming language in which each agent is defined as a concurrently executing process, communicating with other such processes via message-passing. The distinct characteristic of this work is that agents are specified using a temporal logic that is directly executable. To directly execute a specification written in a temporal logic formula, the interpreter attempts to construct a model structure in which the agent specification is satisfied. However, since the environment is dynamic, it may constantly be altering that model, so that responses must be made to what has just become true or untrue because of this dynamism.
The way in which the model is constructed in a dynamic environment is referred to as the execution strategy. When a Concurrent METATEM program is executed, it produces a sequence of temporally ordered states, each labeled with a model structure stating those propositions that are true. This trace provides Concurrent METATEM specifications with a concrete computational interpretation.
The basic form of a specification in Concurrent METATEM is as follows: "on the basis of what has happened in the past do something in the future." Thus, each agent is specified by a set of rules of the form:
and at each cycle, the precedent of each rule is matched against an internal history, firing if a match can be found. Once a rule fires, the agent is committed to the antecedent, which typically involves trying to make some predicate true.
As an example, consider the following Concurrent METATEM program, specifying the behavior of a controller agent solely responsible for supplying an infinitely renewable resource [47]. While the resource cannot be used by two agents at the same time, it is possible that the controller may be asked by two different agents for the resource simultaneously. The predicates ask(x) and give(x) mean that agent x has been asked for the resource, and that agent x has been given the resource, respectively.
In temporal logic, the formula, ○Form, is satisfied at present if Form is satisfied at the next time moment. Similarly, ◊Form is satisfied at present if Form is true at some time in the future, and FG is satisfied if F is true since G has been true.
The specification of the controller agent is defined in Concurrent METATEM as follows:
These three formulas can be interpreted in English as follows:
If an agent asks, then eventually give the resource to that agent;
Do not give to anyone unless they have asked since you last gave to them;
If you give to two people, then they must be the same person.
Concurrent METATEM is attractive because it is directly executable so that no time-consuming and error-prone refinement is required in the process from specification to implementation. In addition, specifications have a concrete computational semantics defined by the sequence of models that arise through the constant interplay of the dynamic environment and the action of the agent. However, since the execution of Concurrent METATEM is based on theorem-proving, some standard problems arise such as the undecidability of first-order logic and the complexity involved in simple propositional logic. In addition, it is not clear how other modal operators commonly associated with agents such as belief, desire, and intention could be incorporated into the language while maintaining its directly executable property.
Subscribe to:
Posts (Atom)
