Booz Allen Hamilton Reports on Customs

As we reported last month, Customs Chief Executive Michael Carmody recently announced release of the findings of the independent review of the Integrated Cargo System.  The 66 page final report is posted, with a press release, on the Customs web site (www.customs.gov.au/site/page.cfm?c=7198), and can be downloaded for viewing by all interested parties.

We’ve taken some time to study the report in detail.  We’ve also perused some of the press commentary.  Below, you will find Quentin Addison’s discussion of how the report’s findings and recommendations point, over and over, to the value of applying the six principles for good corporate governance of ICT set out in AS8015.

Of course, AS8015 was not published when Customs started its Cargo Management Re-engineering initiative.  Indeed, AS8015 was only formalised in January 2005, when most of the ICS development work had been finalised.  So it would be unfair for anybody to suggest that Customs failed to observe the standard.

Notwithstanding, the standard was developed by a highly experienced group of industry professionals and academics, who had enough experience of ICT to know what works and what doesn’t, and to understand the areas where attention is needed to ensure that the use of ICT is successful.  Their experience came from real situations and real learning.  Their insight was not unique, and reflects much of the published literature available today.  Unfortunately, that means that Customs, undertaking a massive business re-engineering initiative enabled by use of ICT, had access to enough widely available learning, and should have taken on board the lessons.

It seems fairly clear now that, while there were many flaws in the process from the beginning, the most significant flaw was that Customs viewed its responsibilities as ending at the boundaries of the Customs Department.  The problem with that view was of course that the initiative was changing an entire industry, where Customs is the central governing agency.  If Customs wanted to change the way the industry operated, it had a clear and continuing responsibility to govern that change across the entire industry.  As with many failed software projects, the key failure was not in managing the technology, but in controlling and maximising effectiveness of the broader change.

For those organisations coming after Customs, there is no excuse for repeating the fundamental errors made in Cargo Management Re-engineering – and especially for federal government organisations.  AS8015 is now well established, and if organisations consider the lessons from the Booz Allen Hamilton review of Customs in the light of AS8015, they should greatly reduce their risk of projects going awry.

The Price of Spin

Some readers will recall our comments in March, under the heading “Alternative Views Emerge”, where we said “Fascinating stories appeared in the February 2006 edition of MIS magazine (The Unforgiven), and in the ACS Journal, Information Age (Australian Customs – More Flak than Facts?).  Seemingly written by the same pen, or at least in the same classroom, these stories presented a view that the events of October 12^th and beyond were not at all serious, and certainly did not warrant the criticisms levelled in the press”. 

Well, the Booz Allen Hamilton report puts a very clear perspective on those stories.  In the executive summary, it says:

“The ICS Imports was implemented on 12 October 2005, with some transition problems for air cargo but with severe short-term consequences for the movement of sea cargo.  In the first days after implementation, a large proportion of containers were held by Customs on the docks, resulting in delays in imports in the lead-up to Christmas.

Many customs brokers and freight forwarders experienced grave difficulties interacting with Customs through their third-party software and attempted instead to use the ICS through the online Customs Interactive (CI) facility.  CI proved difficult to use and very slow under the additional load, further exacerbating user frustration with the system.  These difficulties were not general, some operators had relatively minor issues, but the problems were widespread”.

There can be no doubt from these statements and the detail provided in the body of the report, that the problems arising on and shortly after October 12^th 2005 were very serious, that they certainly warranted all of the press discussion that occurred then and subsequently, and that the Customs Department bears a great deal of responsibility for the situation.

We are left to wonder about the intent and effect of the spin-doctoring that lead two respected journals to publish articles that are now demonstrated to have been quite unrealistic in their representation of what happened.  Of course, the unfortunate reality of life is that this sort of spin-doctoring happens, in all sorts of situations, and many anecdotes tell us that the phenomenon is certainly not restricted to government circles.  It’s human nature that we want to be seen as successful, and most will seek to refute and trivialise any suggestion that they are, or will not be, completely successful.

It takes a very mature organisation to look beneath the primitive emotions, and to face facts.  Those that do will naturally have a very clear understanding of what success looks like, and will be able to consider dispassionately whether success is achievable at every point of the journey.  They will be able to understand that the definition of success may, legitimately, change as circumstances evolve, and they will understand that sometimes the underlying justification for an investment may evaporate.

There are many questions that organisations can ask to help themselves ensure that they are in control of their ICT investments.  The questions should address all six of the principles espoused in AS8015, and they should all be asked in such an open way that they demand full and comprehensive answers, underpinned by sufficient and relevant facts.  Quentin’s discussion below should provide those who need to ask questions with plenty of opening points. 

The Press View

As we said last month, Simon Hayes of the Australian called the Customs report “damning” (http://australianit.news.com.au/articles/0,7204,19416305%5e16123%5e%5enbv%5e,00.html).  In many ways it is.  Though the report very carefully avoids naming any individuals, it does make it clear that there were many aspects of the overall project that were flawed from the outset, and that these flaws persisted over several years.

Selina Mitchell’s discussion in The Australian on June 6 (http://australianit.news.com.au/articles/0,7204,19373437%5E15319%5E%5Enbv%5E,00.html), just prior to release of the report, reminded us that the Australian National Audit Office will deliver its findings early in 2007.  And it reinforced the messages about damage to the import industry, with comments from the Customs Brokers and Forwarders Council regarding the productivity and cost impacts of the system.

On June 13, Simon Hayes continued his discussion (http://australianit.news.com.au/common/print/0,7208,19450087^15415^^nbv^15309,00.html), highlighting key conclusions of the report, and quoting both industry and customs leaders in discussing the aftermath – an aftermath in which all parties are apparently positioning to work together in a more positive and well managed framework to achieve better defined objectives.  Hayes refers to his conclusion that communication was the main issue, underpinned by now acknowledged lack of appropriate testing and the inappropriateness of the “big bang” rollout strategy.

Of interest in the article is reference to $9m of compensation claims that Customs is working through.

Independent commentator and Australian columnist Bruce McCabe wrote “No shortcuts to project success” on June 20 (http://australianit.news.com.au/common/print/0,7208,19495069^15302^^nbv^,00.html).  He quoted the “scathing” Booz Allen Hamilton report in a wider discussion of big software projects that fail.  He positioned his discussion around the views of two academics who have reviewed some 400 software projects.  We are surprised when McCabe says that their findings challenge conventional thinking.  The academics said that “many initiatives are doomed from the start because project managers work to a vision quite different from that of senior executives in the business”.  That, of course, is one of the major elements that we see time and time again.  It results in projects being technically successful while utterly failing to deliver an intended business outcome – where the business change is forgotten in the intensity of the effort to deliver a technology based “silver bullet”.

Roll forward to July 4^th.  Simon Hayes presents an extensive article discussing the same (http://australianit.news.com.au/common/print/0,7208,19644392^24170^^nbv^24169,00.html) research that underpinned Bruce McCabe’s article.  His article refers to “The roll-call of failures, problems and budget blowouts” – all of which are familiar to us, and now with Customs Integrated Cargo System at the head of an extensive list of Australian experience.  He explores other aspects of the research, pointing out omissions in process that are often linked to failed projects – omissions such as failure to include project managers in the development of the project budget.  But the emphasis in this story is on the extent to which the project managers understand the business, and the tendency of those with insufficient business experience to focus on the technology – leaving the business implementation at risk. 

Hayes appears to have the view (and we absolutely agree with him) that a most fundamental pre-requisite for IT success is a clear understanding that there is no such thing as an IT project : there are only business projects that involve the use of IT.  He quotes St George Bank as an organisation that clearly understands the broad base of accountability for success with IT initiatives.  He also quotes Bendigo Bank CIO Vicky Kelly, who has a very clear view that IT projects are part of business projects and not separate activities.  And he quotes Melbourne University’s Linda O’Brien, who expresses the core view that IT is an enabler to process change in the workplace.

Discussion of the Report

by Quentin Addison, Infonomics Associate

The Booz Allen Hamilton Review of the Integrated Cargo System is a comprehensive and detailed report which identifies issues, causes, distributes responsibility and recommends a series of measures which Infonomics can only endorse.  We might have preferred that it had been a little more hard hitting and some detailed root cause analysis had been performed.  Similarly, we can only hope that Customs will take the opportunity and time to address the underlying issues.  For example, we would have thought that an organisation with an appropriate and effective system of ICT governance would have testing as a core discipline and if adequate testing was not observed in the case of this project then we would want to know why. 

In following the Booz Allen Hamilton recommendations to establish “… a sound governance base for the overall program of work, including clear business ownership:”, Customs would significantly benefit from adoption of AS8015-2005 Corporate Governance of Information and Communication Technology to underpin its efforts.

For those not completely familiar with AS8015, it defines Corporate Governance of ICT as: “The system by which the current and future use of ICT is directed and controlled”.  It continues: “(Corporate Governance of ICT) involves evaluating and directing the plans for the use of ICT to support the organisation and monitoring this use to achieve plans.  It includes the strategy and policies for using ICT within an organisation”.

AS8015 defines six guiding principles (see below), which we have used as the framework for presenting this Infonomics perspective on the development and deployment of the Customs Integrated Cargo System. 

1. Establish clearly understood responsibilities for ICT (P1)

AS8015 recommends ensuring that individuals and groups within the organisation understand and accept their responsibilities for ICT.  Significantly, business leaders in the organisation must understand that they are responsible for how ICT is used in the business.

2. Plan ICT to best support the organisation (P2)

AS8015 recommends that ICT plans should fit the current and ongoing needs of the organisation and that the ICT plans should support the corporate plans.  Ideally, IT strategy should be an integral part of the business strategy process.

3. Acquire ICT validly (P3)

AS8015 recommends that ICT acquisitions be made for the right reasons in the right way; based on appropriate and ongoing analysis, with appropriate balance between costs, risks, long term and short term benefits.  It is a good idea for ALL expenditure, even the recurrent type, to be explicitly linked to achievement of business objectives.  The principle also seeks proper and appropriate practices in selection and engagement of suppliers, and for fundamental decisions such as outsourcing.

4. Ensure that ICT performs well, whenever required (P4)

AS8015 recommends ensuring that ICT is fit for its purpose in supporting the organisation, is kept responsive to changing business requirements, and provides support to the business at all times when required by the business.  This should start with clear specification of the performance standards that are required - and not just for classical operational measures such as capacity and response time.  Specifications should cover matters such as responsiveness in support of new business requirements, resilience against adverse conditions and reliable access to current and historical business information.

5. Ensure ICT conforms with formal rules (P5)

AS8015 recommends ensuring that ICT conforms with all external regulations and all internal policies and practices.  Note that this principle is not merely about technical rules such as software licensing.  In many organisations, ICT is entwined with and fundamental to compliance with general legislation (eg privacy, trade practices) and industry specific legislation such as for financial institutions.  Directors should seek assurance that all relevant formal rules are identified, and that appropriate conformance programs exist.

6. Ensure ICT use respects human factors (P6)

AS8015 recommends ensuring that ICT meets the current and evolving needs of all the “people in the process”.  The key word is “all”.  The scope of human factors includes people who deliver and operate ICT as much as those who use it and depend on it.  And, it includes the people who will be involved in the future as much as those involved in the present.

Our review of the Booz Allen Hamilton report looks primarily at sections 3, 4 and 7 and discusses which of the six ICT governance principles are applicable.  It seems quite clear that a more positive outcome might have been delivered, if AS8015 had been in place and used to guide Customs approach to IT governance.

We hope that this approach to highlighting the value of AS8015 will help you to further appreciate the relevance of the standard to your organisation.   (Extracts fro the Booz Allen Hamilton report are shown in italics.  Infonomics comments are in plain text – indented as bullets to improve clarity).

Section – “3.2 The Implementation of Imports” - page 9.

“Customs continued to introduce changes to the software up until 6 October, only one week prior to the go-live date.  This required software vendors to complete their corresponding changes, conduct their final testing and release their software to their customers in a very short period of time.  In some cases, this resulted in the customers of some software vendors receiving their updates for the launch of ICS Imports after 12 October.”

• This suggests strongly that the ICS system itself was likely to have been insufficiently tested and therefore likely to generate some unknown and unexpected problems when deployed, making it less likely to perform well, as required – let alone as planned (P4).  Similarly, giving the software vendors insufficient time to make and test changes on their own interfacing systems, was likely to have a similar outcome (P4).  This also suggests that Customs did not show an appreciation of all of the human factors (P6), which need to be considered.  This includes ensuring that not only the systems for which Customs is responsible including ICS, are ready for use, but also that all interfacing systems and process have been rigorously  tested and are known to be ready for productive use. 

“Problems with some third-party software forced many companies to attempt to use the ICS directly through Customs Interactive.  This showed very poor usability and very slow performance based on the unexpected number of companies attempting to use it and based on the fact that Customs Interactive was not designed to accommodate a large number of users;”

• Poor usability and very slow performance of the Customs Interactive (CI) facility and the fact that it was not designed to accommodate a large number of users, indicates a lack of consideration of matters of functionality, operability and capacity – all of which fit under P4.  Poor usability also brings into question whether there was adequate consideration of the skills of the intended users (P6).  The lack of resilience against adverse conditions in the third-party software (P4) might have been a trigger for the CI problems, but highlights further gaps in attention to ensuring that the entire system performed well whenever required.  Being apparently unprepared for clients switching from third-party software to CI also suggests a failure to properly consider all of the relevant human factors (P6) particularly those over which Customs had no control (external agents and third party software vendors), yet upon which they have been demonstrably critically dependant.

• Both of the preceding comments highlight the fact that it is critical for organisations to understand and act to ensure that the systems and applications for which they are directly responsible perform appropriately and to look at the “whole system” end to end, including all of the interfacing elements upon which they are dependant.

• To step outside of Customs for a moment, consider the likely implications on the Australian financial system and economy if major changes were made to Australian Taxation Office ICT systems and they, like Customs, failed to look at “all of the people in the process”.  If there was a failure to include the necessary changes and consequential testing and support of the hundreds of thousands of businesses, banks, tax agents and consumers, that interface with the ATO’s IT systems, what might be the impact? It doesn’t really bear thinking about, does it?

 “Incorrect data matching as well as profiling of sea containers resulting in a large number of containers being held as high risk in the first few days after implementation;”

• Although in this case, a set of conformance standards for data input were in place (P5), no apparent consideration had been given to the fact that they were, particularly from the end user perspective (P6), unexpectedly and unrealistically high.  Again, little or no consideration appears to have been given either to the people who actually had to use the system or to the quality of data they were accustomed to providing or which was in fact available.  For good ICT governance, these and other human factors (P6) must be incorporated into system design, development, testing, delivery and operation. 

“Industry responded to these problems with calls to the Customs Help Desk.  The Help Desk was overwhelmed by this additional traffic and was unable to handle the volume, resulting in many calls being unanswered.”

• Inability of the Help Desk to deal with all calls suggests that the likely support requirements were not appropriately considered (P4).  The fact that calls went unanswered demonstrates a lack of appreciation of the need of people (P6) to at least be heard, if not to actually have problems solved.  Good ICT governance does not depend on good luck or the hope that it will be all right on the day.  Rather it mandates that support for “all of the people in the process” (P6) and their reasonable needs for information, service and reassurance are met. 

Section “4.1 Progress Against Planned Outcomes” - page 11

“In conducting this review we have been unable to locate a clear and quantified set of outcomes and benefits expected from the introduction of the ICS”

• We acknowledge that the report goes on to identify “..  a number of consistent high-level objectives have been stated..”, however in many respects, this opening statement by Booz Allen Hamilton  goes to the heart of this particular project’s near failure.  These are also significant contributing factors to the failure of many ICT initiatives to deliver what was expected – or, as in some cases, the outright failure of projects.  Such problems often reflect a lack of accountability and responsibility – an area where P1 recommends that individuals and groups within the organisation understand and accept their responsibilities for ICT.  Significantly, business leaders in the organisation must understand that they are responsible for how ICT is used in the business.  Further, failure to ensure that ICT plans fit the current and ongoing needs of the organisation (P2) almost inevitably leave the organisation exposed to risk.

• Effective ICT outcomes are more likely if organisations adopt an approach in which objectives are clearly defined, with all activity aligned to the intent of the project; value to be obtained is understood by all and all parties are focussed on delivering the intended result; there is a clear and sound approach which ensures that there are sufficient resources to design, build, test, deliver and operate the system; progress can be determined using appropriate performance measures and a continuous  process of risk assessment monitors what could compromise success.

“Table 1 – Objectives of the CMR/ICS Implementation “- page 11

“Service effectiveness, as measured by efficient movement of cargo, is greater for some parts of industry (principally integrated air freight operators), and the same or less efficient for the remainder.

Some of the introduced functionality changes have been the cause of severe disruptions (e.g., matching of data on Ocean Bill of Lading) and has resulted in manual work-arounds and reduced process efficiency.  Industry involved in the clearance of sea cargo has reported an increase in labour in order to meet new process standards”.

• With the exception of air cargo movements which has seen some improvement in effectiveness, the impact on the remainder of the services indicates that there has been an apparent lack of consideration of the needs and potential impacts on “all of the people in the process.” (P6) with the outcome that users external to Customs are working less efficiently and productively.  Surely this was not a planned outcome of the system at its inception.

“The profiling engine is not currently “tuned” to be effective.  The effect of this is to provide “overmatching” and incorrect matches in addition to correct matches.

Sorting the “wheat” from the “chaff” is proving to be a challenge _ The data provided by various industry sources in key matching fields is not consistently of good standard.  This creates challenges for the profiling functionality.”

• This appears to be a clear case of the use of ICT not meeting the needs of the business and performing well whenever required (P4).

“Table 2 – Expected Benefits from the ICS”- Page 14

“Use of the Customs Interactive facility takes considerably longer than the same process under the legacy system

• Principle 2 says “Plan ICT to best suit the business”.  A system that increases process time and cost without a manifestly valuable and agreed offsetting benefit cannot be considered to have met this expectation. 

“The process to gain digital certificates is onerous.  Many small companies use family trust structures that are not allowed under the certificate issuance rules.”

• Again a failure to consider “all of the people” (P6) involved in the process, in this case the companies using family trust structures, has created problems which could probably have been avoided. 

Benefits for Customs and Government – Page 14

The ICS requires a much higher standard of data quality than the systems it replaces.  The data quality should therefore improve over time.  However, the poor quality of data currently in the system is the result of combining four legacy systems and data therefore requires extensive “cleaning”.  For example, ICS reference data includes ~1,200 spellings of a major retailer’s name.

The majority of the Corporate Research Environment (CRE) reports are currently not functional.

The high data integrity standards required for matching Ocean Bill of Lading (OBL) have not been able to be met by many companies, creating problems in gaining clearances for sea cargo.

The Customer Research Environment has substantial potential as a single source of client data.  This potential is currently not being realised due to low data quality (see above).

• While the attention to compliance with standards (P5) might focus on formal industry, legal or legislative standards, it is equally applicable to system and data integrity standards as shown in the preceding three items.  Attention to ensuring that existing data was of appropriate quality would have significantly reduced the impact on the new ICS system.  Also as noted previously, recognition of all of the likely users (P6) and previously acceptable data quality standards should have allowed Customs to incorporate these potential and probable impacts into their planning of the ICS implementation.

No noticeable decline in costs has taken place yet.  No benefits realisation plan exists to ensure capitalisation on cost efficiencies.  Also, no specific cost savings have been targeted or reported against.

• AS8015  (P3) says “Acquire ICT Validly – for the right reasons, in the right way”.  In the case of a major investment, observance of P3 would demand that a comprehensive business case be developed, in which the exact objectives to be attained are set out, and in which there is a comprehensive assessment of the work required and the costs of achieving the objectives.  In conjunction with clear objectives, there should always be a definite specification of benefits to be realised, along with a specific plan for managing the benefits through to realisation. 

The legacy systems replaced by the ICS operated on a Unisys mainframe for which support would not have been available from March 2006.

• On a more positive note, this observation from Booz Allen Hamilton, clearly shows that Customs in compliance with P2 have at least with respect to equipment, ensured that ICT plans “… fit the current  and ongoing needs of the organisation.”.

Table 2 – Expected Benefits from the ICS - Page 33

In the case of the ICS, there does not appear to have been an effective structure or process to direct and control the project, nor to make suitable risk decisions.  To fulfil this task, Customs has had at least 10 bodies responsible for different aspects of the management and governance of the ICS, including the interactions with industry (see Table 7).  These bodies overlap in their responsibilities and accountabilities, and overall the program has no single business owner and accountabilities for its delivery are unclear.  (Infonomics bolding)

• The specific intent of AS8015, Principle 1 is to ensure clarity of decisions about the use and delivery of ICT and that the decisions correspond to the real needs of the organisation.  Notwithstanding the considerable efforts that have obviously been made by all participants, during and subsequent to the implementation of ICS, it seems remarkable, given the above statement from the report, that Customs was able to deliver anything at all.  It doesn’t seem completely inappropriate to suggest that Customs has succeeded in spite of itself.

As noted earlier, the Booz Allen Hamilton report identifies governance gaps and makes governance recommendations that link directly to the six principles of AS8105.  As Customs moves to implementing the recommendations and establishing a more appropriate and effective Governance Model, we strongly recommend that AS8015-2005: Corporate Governance of Information and Communication Technology be employed to guide implementation.