Enterprise Abstraction

On a simple EA governance process

When we first embarked on the "EA Adventure" we'd received a goodly deal of in-house training and consultation - and we followed up on that with lots of reading and conference attendance.  All of these sources stressed the importance of architecture governance - and we agreed - an architecture that isn't followed or implemented is less than worthless.  However, few of our sources offered any suggestions on how to actually execute that portion of the process. 

We were particularly concerned that our organization is a rather small one - and our architecture group is proportionately diminutive - we wanted to reduce the amount of time spent on governance to a minimum.  Moreover - there was concern that the architecture group itself does not really have any direct decision making authority with regard to projects.

We established what we'd considered a pragmatic process for governing architecture alignment (then called compliance) in projects.  It essentially involved reviewing a project at 1) the planning stage, and during the 2) software design and 3) development stages.  Performing the review would be a committee - the Architecture Review Team - consisting of the architects and various heads within the technology department.  The project manager or technical lead would demonstrate the relevant parts of the project to the committee, and we would evaluate the project's alignment based on our standards, models and principles.  Good right? 

Of dozens of small projects - we successfully completed 2 partial reviews in 3 years - and neither of those reviews were actually successful in creating any sort of positive change in the projects.  So, what were some of the problems?

Firstly - we'd not really established the fact that our efforts should be guided from the very beginning of an initiative -- at the project proposal stage -- by architecture.

Secondly, the section heads that we'd invited to participate in the committee were not particularly well versed in the architecture, or architecture process.  They felt that they could contribute very little to the governance process.  Only the architects - involved in the depth and breadth of the architecture could really have an up-to-date understanding of the architecture to make informed decisions. 

Finally - getting that large group together to perform a review was an exercise in logistics.

Hence, not a terribly successful first attempt.  We couldn't review the projects early - when we could have the most positive impact before a project had already established a scope and plan - and we couldn't perform reviews in a timely fashion - it taking so much time to arrange meetings, review the material and keep up to date on architecture.  Most importantly - we felt we were wasting a lot of time for little impact.

How did we make it work better?  We abolished the malfunctioning Architecture Review Team - and replaced it by having an architect assigned to each project that was responsible from beginning to end in tracking the alignment - from proposal to close.  The architect ideally helps to write or consult on the project proposal and plan - and establishes which architecture components would be implemented by the project (if any), and keep track that the project was meeting established technology standards and architecture principles.

If there is an alignment issue - it's brought first to the technology lead or project manager - either directly or in a team meeting.  Failing a resolution at that level the issue can be reported to the project review committee (a committee that's assigned to oversee individual projects -- to help remove roadblocks and provide guidance).  Failing a resolution of the issue at that level - the problem is presented to the project sponsor and/or the corporate management team.

It's a reasonably nimble process - relying mainly on the resource availability of the architect (easily managed), and taps into the authority already existing in the project management and project portfolio processes already in place.

At this point in our second attempt at architecture governance - things seem to be going well.  We're currently tracking the alignment of at least 50% of the ongoing projects - and likely 100% of the architecturally significant ones.  Recommendations are more easily accepted - and in the few cases where there have been serious issues - they have been resolved reasonably quickly.

I'm crossing my fingers for continued success.

Metadata management and EA

DM Review Magazine has an article this month entitled Metadata Management & Enterprise Architecture:  Understanding Data Governance and Stewardship, Part 1. In part 1 David Marco and Anne Marie Smite discuss data governance, and the benefits therein.

It is a topic that holds much importance for most organizations, but particularly for us at CISTI, and our constant battle with sets of bibliographic metadata, along with the financial/client/transactional data of our day to day service provision.  Traditionally data was the domain of a particular silo, but as we move forward, it's becoming clear to everyone that management of data is exceptionally important to achieve the most efficient and effective use of the information across business lines.  "Who owns this data?" is likely to be as much a question as "who owns this service?"  We're finding that the answer to that question is not as obvious as it used to be.

 

ITIL and SOA

More on the relationship between ITIL and EA from IBM.

 

In this paper, delve a little deeper into the ITIL processes and their sub-modules. Then take a look at how you can leverage ITIL-based canonical domain models and an Enterprise Service Bus (ESB) to provide a messaging infrastructure that can integrate and automate applications catering to IT service management concerns, while still helping your enterprise realize its vision of Service-Oriented Architecture (SOA).

-- Link via Real World SOA: Is SOA Boring by Dave Linthicum.

ZDNet's Joe McKendrick posts regarding (North) American ITIL through a survey by OVUM:

...the survey found a high correlation between a business' level of satisfaction with SOA and their commitment to managing IT as a set of services. Leading-edge SOA adopters, in fact, are more likely to be also adopting IT infrastructure library (ITIL), service desks, asset and configuration management tools, IT portfolio management tools and business service management performance monitoring dashboards.

He's followed up his first post on the topic with some endorsements of the linkage between SOA with ITIL as an appropriate service management compliment.

Silicon.com follows up with an article that succinctly describes some of the newly exposed IT management issues exposed via SOA adoption:

The Ovum survey found a high correlation between a business' level of satisfaction with SOA and their commitment to managing IT as a set of services. Examples of best practice amongst leading edge adopters include IT infrastructure library (Itil), service desks, asset and configuration management tools, IT portfolio management tools and business service management performance monitoring dashboards.

See my previous post on this topic.

SOA and communication with the business

I recently came across an article in CIO magazine entitled The Truth about SOA where they raise the question of communicating with the business about SOA.  The answer they provide is:  don't.

Q: SOA is a technology architecture. How do you make a business case for a new technology architecture?

A: You don't. "Don't talk to the business about SOA because the business doesn't care," says Forrester Research analyst Randy Heffner. The business's interest in SOA extends only as far as it cuts the cost of applications and gets them running faster. But simply rewriting code in the form of a service doesn't deliver those kinds of benefits. To start down the road to building an SOA, there needs to be multiple redundant IT applications that can be consolidated into a single service, or the possibility for multiple areas of the business to use a single service.

I couldn't disagree more.  Let's leave aside the apparent mixing of SOA (business planning as it relates to technology) and web services (a particular implementation technology that happens to complement SOA very nicely), and the narrow definition of SOA as specifically a technology architecture.

You must communicate at least the broad concept of SOA, because in its implementation you're going to change the IT (and possibly) business governance structure substantially.  The business must be involved in this process, and they must understand the implications of moving from (in most cases) tightly integrated silo applications and infrastructure to an SOA environment with all its advantages, and disadvantages.

One of the disadvantages are the potential implications of change and change management.  Someone needs to be responsible for these services, and to ensure that changes and impacts within the business are met.  The must ensure the service levels, extensions to the service, and predict and plan change as the business changes. 

For the business to agree and endorse these changes they must understand some of the fundamentals.  In particular, in my organization, the difference between the presentation (offering) layer, the service layer(s).  For most businesses used to running applications in silos and managing the silo from top to bottom (as a "product") this likely implies a major change -- or at least a consideration -- in the way the business, in addition to the technology, is governed.

Richard just posted on SOA Service Ownership, and some of the specific issues we're having with regard to SOA service governance.

As in most business/technology communication efforts one of the primary issues has been coming up with terms and definitions that we can all agree upon amongst the business and technology folk -- and those of us stuck between.  We've just recently come to a consensus on some of the key terms in my organization.  Perhaps I'll post some of these terms and definitions once they're a little more stable.

Rolling stone of governance

My experience with a large library/small organization is that the best you can do when starting architecture is to get the stone rolling in the right direction, and subsequently, and only if absolutely necessary, nudge it from that point forward.  It's impossible to stop the stone (any particular project or initiative) once it's built up some inertia.

We've tried "after the fact" governance processes that were too heavyweight and too late to be effective, no matter how logical or economical our arguements.  My experience is that unless you start projects on a firm footing, there's very little chance of having anything but a limited impact on success of a project post-initiation.

Hence, we've put most of our effort in providing a service to business of helping them construct their proposals and manage the project initiation process.  In some cases we'll initiate projects ourselves.  In other cases, we'll parachute in to offer a prospective project proposer assistance with scoping, tasking and estimating a project early, and most importantly, with improved speed and efficiency as compared with the more traditional method. 

We find that in our practice this reduces uncertainty and eliminates most of the "investigation" projects spawned from fear of not knowing the complete solution up front.  Most importantly, ideas usually manage to move from thought to proposal stage an order of magnitude faster than they had  otherwise.

The biggest challenge thus far has been to have architecture perceived as primarily an enterprise  project management and business-related process than the common view of it as a technology-related process.  This acceptance has proven to be achievable, but it's taken uncounted weeks and months of communication and demonstration to convey. And still the march goes on.

SOA and Governance

Dan Foody of Sonic Software provides some useful insights into SOA Governance.  He highlights the issues, and dangers, of increasing complexity in governance as organizations move forward in their efforts.  He describes two main ways to avoid complexity:  automation of compliance; and having fewer but more important rules.

He illustrates some of the issues with automating compliance, and the issues of rogue services.

Of particular interest to me is the issue of service reuse, and the practicality of the reuse model.  That is to say, services are good in theory, but whilst we've eliminated complexity from some areas, we're likely adding complexity to others now that a service or set of services can now be a part of any number of applications, all with competing rules and desires for functionality and performance:

Service-oriented architectures dramatically change the way you need to think about your production applications. When a service goes into production, that's not the end -- it's just the beginning. The reason is that every time a service is reused, it essentially becomes part of a new application -- a new business process -- and that business process may have an entirely new set of rules to obey.

We used to have product managers... now we seem to need another layer of service managers... or at least some really good processes and automation.

What's an ESB

ZDNet asks:  What's an ESB?  It seems to me the best definition is that it's a burgeoning new market for a lot of software companies.  That always makes me a bit wary.

Wikipedia defines an ESB as:

an enterprise service bus refers to a software architecture construct, implemented by technologies found in a category of middleware infrastructure products usually based on Web services standards, that provides foundational services for more complex service-oriented architectures via an event-driven and XML-based messaging engine (the bus). An enterprise service bus generally provides an abstraction layer on top of an Enterprise Messaging System which allows integration architects to exploit the value of messaging without writing code. Contrary to the more classical EAI approach of a monolithic stack in a hub and spoke architecture, the foundation of an enterprise service bus is built of base functions broken up into their constituent parts, with distributed deployment where needed, working in harmony as necessary.

perhaps we should all contribute our thoughts in that forum.

Ron Ten-Hove offers the more succinct:  " An Enterprise Service Bus (ESB) is a distributed middleware system for integrating enterprise IT assets using a service-oriented approach."   It may be worth mentioning that Ron and others have been working on the Open ESB project.

Do I need one?  I'm not sure.  I've heard many say that with good planning of services, and a well thought out infrastructure, depending on your needs and circumstances, and ESB is not required, but that it could certainly make life easier.

A strategy I've heard is that it's wise to plan your services as if you don't have one, and are not going to have one... essentially treating the ESB as an addition or an afterthought, thereby encouraging robust and efficient service design.  Sounds like sage advice.  I've also heard that a good governance structure around SOA would be important before implementing an ESB. 

Perhaps then it would be best to wait until things are a bit more mature before diving into the still undefined world of the ESB?  It seems pretty clear that a good strategy would be to concern ourselves with the architecture first, and ESBs second.

Development capacity for improvement

Nick Malik brings up a good point regarding the need to have extra - IT driven - capacity to apply purely to the task of simplifying and reducing the complexity and maintenance of systems.  In my experience, nature abhors a vacuum, and it's particularly difficult to have a full team dedicated to reducing complexity, even if you could justify the expense by having the team pay for itself through reduction of costs.

I think it would be particularly important in the establishment of such a team to ensure that it was driven by it's mandate to reduce complexity, and to ensure savings over the long term.  I agree that it would be best to be IT driven, and not driven (directly) by the ongoing business concerns.

International EA Survey

This might be old news.  I just stumbled across this site that has made available a survey covering the governmental adoption of EA across a number of countries, including Canada, UK, US.  Note that the survey is really talking about governmental adoption of EA, not the amount of adoption in a given country.

Key survey findings:                   

• EA on a national level is emerging fast
• Limited realisation of EA goals
• Less than one half of the governments are measuring EA program performance
• Less than one fifth of the governments are calculating the ratio EA benefits to cost

The survey is a partial continuation of an international EA survey started in 2003 by The International Council for Information Technology in Government Administration (ICA, http:/www.ica-it.org). Throughout 2003 – 2004 ICA's Enterprise Architecture study group, consisting of EA competencies from ten ICA member countries, conducted a conceptual survey of the national EA activities in the ICA member countries.

The survey seems to particularly illustrate a lack of metrics and governance, likely due to the relative lack of maturity of most of the federal level EA's at this point in time.  Also, with regard to tools, the data suggests to me that a lot of places are using Visio or similar modelling tools as opposed to more rigorous architecture specific tools.  Here's a direct link to the survey findings PDF.

-- Via GotzeTagged

SOA and Agile - Peas in a pod

Brenda posted about the compatibility of SOA and Agile, comparing them to apples and oranges.  I'm not sure of the best metaphor to use, but in my opinion they're undoubtedly compatible methodologies. 

It's human nature to want to 'waterfall' processes.  From architecture to software design and development.  EA, SOA and, of course, Agile, all benefit from an iterative and incremental approach.  Another key practice I've heard applied to both is to, "do just enough, just in time" -- or just in time/agile architecture.

Amongst the strengths of agile methods are the strong assertion that you don't know all the answers up front, that you're willing to accept and embrace change at any point.  These are excellent principles to apply to SOA as well, within reason.  Assuming you're going to start from zero and work out a complete and workable architecture, followed by a representative development and implementation is probably folly. 

Architecture, among other things, is meant to constrain and guide development.  In this regard, EA, SOA and agile should be able to live together harmoniously as long as the architects and developers come to a pragmatic consensus about boundaries and collaboration.  This means having a good working relationship between the two camps, and a good idea of the reasons and methods behind the respective schools of thought.  This, perhaps, is the hardest thing of all to accomplish.  It certainly requires a lot of trust on both sides.

The approach we've taken thus far is to have architecture define a set of well validated services, interactions and general interfaces based on our EA practice, which is directly traceable to business desires.  During implementation we have used an agile methodology to actually construct the innards of the services, integrate, and to build human interfaces and detailed components of the desired system. 

The architects take care to guide the direction, and assert control only over the critical (architecturally relevant) portions.  Usually this means staying at a fairly high level of abstraction. The remainder (the details of implementation, integration, detailed data structures) are left for the designers/developers, and if they choose, an Agile approach.  This works best if the designers/developers and architects will interact as the project proceeds to make adjustments and clarifications as needed.  In this way the architecture, and the implemented system both come closer to meeting the true needs of the business in the end.

I'm keen to hear other opinions in this regard, as admittedly my SOA and Agile experiences are still quite new, and only partially tested in the real world.