From tracking to planning ... and back

xProcess provides many of the facilities of an issue tracking tool alongside its project planning/execution and process improvement functionality. Particularly with the new web client access, it's quite possible to use xProcess as your issue tracking server as well as your project execution environment. However many projects and organisation already have issue tracking systems in place and would like to integrate the two systems, allowing issues to transform into tasks in project plans - when they have been classified in the tracker system - and the status of issues to change when progress is reported by the project.

The mechanism xProcess provides for implementing such integrations is the Workflow Server. This can monitor the tracker system as well as changes within the xProcess data source and trigger actions when specific changes are detected. How does this work out in practice? Here's a brief summary of a current integration with a leading tracking tool.

• The tracker is used to create, modify and review issues
• xProcess is used for planning and executing the confirmed issues
• When issues are confirmed, work is planned by instantiating the correspondingpattern (say for a bug-fix) in the appropriate xProcess project. It is the user setting the status to “confirmed” that triggers the transfer into the plan.
• Updates made to the data in the tracker may cause an alert or update in xProcess and vice versa.
• When sub-tasks close in xProcess, the plan status in tracker is updated by the Workflow Server. For example states like: Specified, Code complete, and Accepted correspond to particular subtasks being completed and closed in the project.
• If an issue is deleted or closed in the tracker, data is transferred to xProcess and may cause an alert to the project manager or other flags to be set in the data.
• Viewing the task in xProcess allows data from the tracker screens to be viewed (and updated) directly. Reverse links also possible.
• The issue in tracker and its corresponding task or tasks in xProcess are independent objects but with linked lifecycles that are controlled and synchronized through the Workflow Server.

Much of this integration is independent of the particular tool you are using for tracking, so you may find providing your tracker integration can build directly on this work.

Comparing big patterns and small ones

When defining your processes in xProcess you need to decide on the level of granularity at which to define task patterns and the degree of complexity to build in. Compare these two examples of feature pattern.

This first one is nice and simple. Two people will work on the feature using and/or modifying the three associated artifacts (Word documents in this case) Specification, Design and Test Spec. It's simplicity is its greatest advantage since this allows a lot of flexibility when such features are planned in a real project.

The next pattern is more complex and goes to a finer level of granularity. It defines 5 subtasks, using 4 specific role types and having different artifacts for each of the tasks. The artifacts are a mix of Word documents Wiki pages and xProcess Forms (user-defined data held in internal XML format). The result is a more complex patterns which breaks down the work into more speciifc packages and provides more guidance in terms of the work to be done.

There are dependencies defined on this pattern and there is also a gateway ("Feature complete") which defines the QA requirements for its completion.

This more detailed view of the same pattern shows the allocation of tasks to role types and the specifics of the artifacts.

Which of these two patterns is most appropriate for your projects? I can only recommend you try them and see... and most importantly let me know what you find!

What makes processes agile?

The key thing that makes processes agile is "reduction of work in progress". Traditionally, projects' processes view and partition the whole project: for example software development processes that divide the life-cycle into phases like Requirements, High-level Design, Detailed Design, Code and Build, Integration and Testing, Maintenance. Each phase of the project transforms the whole output from the previous phase into the relevant output products for that phase. You could represent such a lifecycle in this way, where each step, T, takes the artifacts, A, from the previous step and delivers them to the next step.


The major problem with this model is that every step must be completed for every part of the project before any delivery (the so-called "big-bang" delivery). There's a lot of work in progress in this model, and that means a lot of risk and a high level of required investment.

In contrast, an agile lifecycle defines the necessary transformations in the process, not for the whole project, but for a single requirement (“features” in FDD, “user stories” in XP, or “use cases” in UP). The release cycle, which is built around the process for delivering single requirements, ensures that benefits can be delivered sooner and in priority order. Should requirements change it no longer involves the loss of such significant amounts of work. At each release there is much less work in progress and value is delivered continually, rather in the single - and very risky - big-bang.

New diagram in xProcess

A recent development in xProcess is the support for State Transition Diagrams to define the workflow that is triggered by changes to elements of xProcess data. The diagram attached to this entry is a sneak preview.

You can define states of the elements you are interested in (Tasks, Projects, Peoiple, etc.) by defining a rule in OGNL. When these objects change from one of the pre-defined states to another the Workflow Server will inspect the defined triggers and call the actions defined therein.

More later!

What's in a process?

This is an interesting question - particularly when you're involved in building tools to support them!

In fact the processes behind most projects boil down to

• what is to be done (tasks, activities, actions),
• what it is done with (resources, artifacts, tools) and
• what it is done for or to (products, artifacts and changed status).

Processes can be modularized and if you want to support priority-driven planning - an essential benefit provided by xProcess - you need to build modules of your process - task patterns - around single units of stake-holder benefit (that is around single requirements).

A process need not just address task planning but also aspects such as

• quality control,
• artifact management and templates (e.g. for requirements, issues, design and user documentation),
  
• workflow (for notification of team participants and for integration with other tools in the environment such as accounting and tracker packages) and
  
• human resources concerns such as the definitions of role and skill types.
  

As organizations grow the libraries of processes behind their projects they can share best practices, evaluate the effectiveness of process changes and optimize their standard approaches to a wide variety of project types. In the mean time, all the projects applying the processes are able to dynamically report status changes, target changes and scope changes achieving both improved agility, and demonstrable compliance and auditability.

If you're defining a process in xProcess here's a list of elements you should be considering:

• Project patterns - what's the essential structure of projects
• Task patterns - are their different patterns for things like features, defects , releases
• Folder patterns - what are the groupings of activities you want to support for planning and prioritising
• Artifact types - what documents and other artifacts are produced and to what template
• Role types - what roles do people play
• Gateway types - are there tasks that need a auditable quality checks applied?
• Category types - how are tasks and other elements categorised?
• Expense types - any expenses to build into the process?
• Actions - are they actions you want triggered by users or events?

'Killing development projects early and often...'

Esther Schindler's recent post what Gartner is telling your boss makes for interesting reading.

Gartner urges managers to consider better process control and governance, managing 'application portfolios' much as they do stock portfolios. Part of this discipline is 'killing development projects early and often'. While no doubt some managers will want to respond to this by applying for the "Ghegis Khan training course for mangers" what it really means is that managers need much better information than many currently have about the status and delivered benefits of their projects.

Now where might they begin to get that kind of information from? Not only does it need to be part of the process of managing a portfoio of projects, the information needs to immediately accessible through the process support environment.

Exactly.

Who needs what?

One of the key ideas in xProcess is that it puts planning and process control in everyone’s hands - at the level they need it. For example...

Project Participants can:

• see their tasks and priorities (potentially across multiple projects)
• update estimates and progress
• update their availability and overhead activities
• update their active tasks and close their tasks when complete
• see changes to priorities and target dates

Project Managers and Planners can:

• create project tasks and prioritize them
• change available resources for the project and assign them to roles or tasks
• set the scope and target dates for timeboxes and releases
• see immediately if tasks are forecast to miss targets and why
• view burndown or Gantt charts of the whole project or timeboxes
• select and/or customize the processes their projects use

Project Executives and Stakeholders can:

• overview the goals and progress of multiple projects
• monitor and improve the processes adopted by projects
• compare and contrast processes if they are different
• examine “what-if” scenarios by varying resource deployment
• redeploy resources between projects
• monitor financials spend and earned value of projects
• generate different format reports for projects and resources.

Process Improvement Managers can:

• monitor and compare the progress of projects applying different processes
• capture process improvement strategies from different projects
• define and improve company standard processes
• monitor process and quality compliance.

The new UI in version 2.5 will reflect the different emphasis required by different roles with tailored perspectives for Project Managers, Project Participants and Process Definers. Expect this to be developed further in future releases.

Summer Fruit

Mmm - no postings in June July and August! Blogging just doesn't seem to have been on the agenda this summer. So what has been happening?

Actually a lot, and the benefits of all that work is now close to fruition. xProcess 2.5 is released in October and has a plethora of new features and user improvements that really are exciting. The headlines for the release mention Workflow, Graphical Process Modelling, Financials and Web Client. But in fact there is so much more, and I'm sure our current users will be delighted while new users will be surprised by the richness of functionality that's available through a much sparser user interface.

Even before the release I've been hitting the road again and showing selected customers the new direction and it's encouraging to hear the feedback. There's a lot to live up to. One customer gave us an outstanding accolade from his use of the existing version:

xProcess has freed me from fighting with a tool to make a plan, and instead given me direct visibility of our progress against plan, allowing me to get on with my job of project managing. (John Andrews, iiPay, www.iipay.com)

Thanks John. That's very much what we've set out to do. I think you'll find this new version takes you further down that road.

A lot of new stuff in version 2.5 is focused on enabling our OEM partners and larger customers to build more integrated process improvement and project planning environments. Workflow let's notification flow round the company, as well as receiving and issuing updates to other tools in the environment like issue trackers, requirements repositories and test tools. The Financials package allows integration with accounting tools and the Web server and client means that pages can built that show xProcess data and allow it to be updated by authorized users. Expect to see some exciting developments in this department over the coming months.

Overall it's been a fruitful summer... even if the harvest is yet to come!

Share your Personal-Edition plans NOW!

It's great how many people are discovering the free Personal Edition of xProcess. I get to see the download figures occasionallly and it's very heartening to know so many of you are getting this version and putting in your own plans and processes.

Of course there is a snag with the Personal Edition - it's personal!

This means there's only one of you who can see it and that means you're missing out on the best aspects of xProcess - its support for collaborative processes and team planning. This month the development of Ivis' partnerships with CollabNet and Clearvision means we now have a very simple way to upgrade you to an evaluation version of the Enterprise Edition. Straightaway you'll be able to share your plans and interact with them as a team. Just go to www.ivis.com/evaluation if you'd like to try it out.

"Plan of Intent" and "Plan of Record"

Ron Lichty is well known in the Software Engineering community on the West Coast as a practitioner, as a seasoned project manager of many successful ventures and in a number of SIGs and conferences in which he is active. In spite of knowing Ron by correspondence over a long period of time it was only at JavaOne this year that we finally got together and I'm very glad we did.

Ron wrote to me after our meeting:

I told a number of people later at JavaOne, and even later that evening at the Software Engineering Management SIG, about xProcess. It really looks good. A question came up: It's a common technique in large organizations to keep a "Plan of Intent" and a "Plan of Record" - to have two project plans, one for the business partners and boss, one you actually execute to. Any support for that in xProcess?

Good question! Here's my reply...

There is support in xProcess for an arbitrary number of target levels through what we call (in the process definitions) Planning Boundaries (PBs). The patterns containing PBs can be called what you like in the process definition - typically timeboxes, milestones, releases, "sprints", commitments, etc. PBs consist of a "scope" (a set of tasks) and a target date. So your "plan of record" could be contained in a set of PBs showing your commitments to one or more parties and the status of the PBs would be different from the team target status even though it is derived from the same project forecasts. It's a useful technique and one we recommend to xProcess users.

By the way Ron has just completed a cross-country ski marathon in Anchorage, Alaska benefiting the Leukemia & Lymphoma Society. If you enjoy photos, maps, and stories of such adventures, see this site http://www.active.com/donate/tntgsf/RonLichty. Also check out Ron's blog at http://ronlichty.blogspot.com.

Interview with Artima: Planners and "Do-ers" (AUDIO)

JavaOne is all about meeting people and finding the connections between what they are doing and your own work. This was brought home to me when I spent some time talking to Frank Sommers and Bill Venners, who are editors of the Artima Developer site, while they were recording interviews with vendors of innovative technology at JavaOne. You can hear their interview with me by clicking on this link... Audio file.

Many of the developers that I met at the show immediately denied they had anything to do with project management, which they associated with people who drew charts and manipulated budgets but rarely interacted effectively with the teams they were working in. It was very clear that the kind of synchronized connection between planners and "do-ers" that xProcess supports finds a strong resonance with such developers. It's a solution with a corresponding problem that's high on a lot of people agendas right now. Maybe that's why traffic on our site hit an all time high during last week and downloads of the free Personal Edition of xProcess are growing exponentially.

The interview by the way lasts 5 minutes 20 seconds and is available as an MP3 pod-cast.

JavaOne Feedback

We're preparing to head for the airport in San Francisco so this seems like a good time time to look back on an eventful week at JavaOne. There are so many conversations, new people we've met and significant feedback that we received from people who have used xProcess or were using it for the first time. Ivis didn't have the most conspicuous booth in the Pavillion but somehow word got round that something goodwas going on there and we had excellent traffic through the booth and interest from nearly everyone who stopped by. We also had opportunities for meeting with valued Ivis partners like CollabNet, Apple, Sun and Serena while we were there and the building excitement around both the feature set and architecture of xProcess 2 was very heartening.

One highlight for me was Paul Kuzan's Birds of a Feather session on the Tuesday. Its timing was not ideal - 10.30 to 11.20 on the Tuesday night, competing with many vendor parties, restaurants, bars and simply being in bed ready for a 7am start the next day! Nevertheless we did have a small but very engaged audience to hear Paul explain the principles of xProcess's connected/disconnected architecture built on a file version control system. The penny is gradually dropping that this architecture has significant applications beyond the immediate extensions to xProcess's. Several of the audience who attended and came up to talk to Paul and me after the event had also seen this potential and were keen to discuss further.

The final proof of the JavaOne pudding will be in the eating. Not just the great conversations, record leads generated and subsequent record stats on the web site - it will bejudged on how this exposure leads to actual users of the technology and ultimately delivered business benefits.

Let me invite you to the Brewery!

Working on the principle that even I could organize an event in a brewery, we've decided the launch party for xProcess 2 should be at the old Whitbread Brewery in the City of London (see www.thebrewery.co.uk for more details of the venue). It's an historic old building and, in the hope that there could be something good stored down there, we're meeting in the Smeeton Vaults.

If you're in striking distance of the City I very much hope you'll join us for the event. Here are the details:

Date: Monday 24th April
Registration and coffee: 3pm
Programme: 3.30pm – 5pm followed by drinks reception and exhibition
Venue: The Brewery , Chiswell Street, London EC1Y 4SD
Underground: 3 minutes walk from Moorgate or Barbican; 10 minutes walk from Liverpool Street or Bank.

Come and hear about ground-breaking developments in xProcess for capturing and improving the processes behind your projects, and then supporting your whole team in estimating, planning, resourcing, scheduling and monitoring the live projects. Simple to install, xProcess gives the team access to all project artifacts and the latest process and scheduling data. Information from the team and other stakeholders concerning estimates, availability, priorities and risk is available to all other stakeholders, and every change is versioned and auditable. Find out how xProcess can empower your teams and increase agility, best practice conformance, visibility of plans and progress, auditability and quality.

Register for the event now at www.ivis.com/london-launch or call Laura on 023 8087 5899.

Versioning and control

It is worth noting that behind the xProcess workbenches used by the managers, process engineers, planners and participants on the teams there is an innovative distributed architecture that ensures all the changes made are shared throughout the team. Whether changes were made originally when the team member was connected to the network or temporarily working off-line, xProcess handles storing and merging changes as they arrive at the server and each distributed client. I'm beginning to wonder whether this particular aspect of xProcess is in fact the most important from the point of view of effective team management, even though it is probably the least conspicuous of its virtues.

Since documents and other artifacts can be included for management within the xProcess environment, the underlying architectureallows all project data to be safely managed and distributed with minimum intrusion for team members who simply experience that processes, plans and artifacts they are using are kept continuously in-step with the rest of the team. Subsequent auditing of the project, should it be required is very straightforward since there is full access to complete change histories of project files. It is a vital but inconspicuous benefit of the xProcess system.

If you'd like to know more about this particular aspect of xProcess, I'm sharing a platform with Paul Kuzan, Ivis's chief architect for xProcess, at JavaOne in May. We'll be sharing more about this aspect and lessons learned from the implementation of the system.

The xProcess Beta Programme

If you have collaborating teams of professionals that need to …

• plan, execute and monitor projects;
• define and improve their processes in terms of the tasks and documents that make up typical projects (and yet not be constrained by a “process straight-jacket”);
• control projects by prioritizing requirements, varying resources and defining target dates;
• bring together all the plans, resources and documents for their projects in one versioned and auditable repository;

…then considering process improvement through xProcess is an obvious next step. As we get closer to the release of version 2.0 of the product, one way to get a really close view of the new features that this version will bring is to participate in the beta programme now under way. Note that the beta programme will continue after the 2.0 release, allowing participants to see the very latest features coming through into the product.

If you think this programme might be for you, do let me know.

Falling over the waterfall

I'm amazed by how many people I meet that tell me their project, or projects in their organisation, use some kind of variant of the waterfall lifecycle. It happened to me again today while conversing with a friend who works for one of the major international banks. The waterfall is as ubiquitous as death, taxes and project overruns!

[And now I discover there's an unmissable conference on the subject - stop reading this blog and go now to waterfall2006.com! But more on this later.]

The interesting thing about the waterfall lifecycle is that it has very few proponents among the luminaries of the industry. Even Winston Royce, who is usually credited with inventing the waterfall in his 1970 paper to the IEEE was actually criticising the approach of trying to deliver a complete system in one iteration. While I'm in name-dropping mode, I could mention that I met Winston when he was working at TRW. He had a most distinguished pedigree in software engineering but he was hardly the unqualified supporter of monolithic processes. His son follows in his tradition and provided a very interesting quote on the waterfall process:

• "Across the software industry, we characterize modern software lifecycles using many different terms, including spiral development, incremental development, evolutionary development, and iterative development (my preference). In spirit, these terms all stand for the same thing, namely anti-waterfall development."” Walker Royce (2000)

More striking criticism of the waterfall approach comes from all quarters. F. P. Brooks for example:

• "Much of present-day software acquisition procedure rests upon the assumption that one can specify a satisfactory system in advance, get bids for its construction, have it built, and install it. I think this assumption is fundamentally wrong, and that many software acquisition problems spring from that fallacy.”" F. P. Brooks (1986).

Or Tom Gilb:

• "There is nothing (no complex thing) that can't be delivered in an evolutionary fashion; conversely no (complex) thing can be delivered in one go." Tom Gilb.

The waterfall has a significant advantage. It's simple to explain and understand. For those tempted to adopt it for that reason should heed this warning from the celebrated journalist, political commentator [and cynic], Henry Mencken:

• "For every complex problem, there is a solution that is simple, neat, and wrong!"” H. L. Mencken “.

However I would want you to think that I'm biased in any way. I recently was pointed to this new conference on "Waterfall Unified Process" by a friend and former colleague Jon Kern. Whatever your view on waterfalls, I urge you to visit this site. It's great!

I've already mentioned the link. It's http://www.waterfall2006.com/.

Why should you use xProcess with your development teams?

Occasionally I get asked questions that force me to re-consider my elevator-pitch for xProcess. So somebody just asked me why he should use xProcess with his development teams. Here's my reply:

"xProcess merges a continual process improvement capability (dynamic process management) with fully connected project collaboration and management (dynamic project execution).

"If you capture your processes in xProcess it means that the patterns of tasks, deliverables, quality criteria and workflows in the process can be realized by project managers and participants as actual elements in running projects. (Since all data in xProcess is fully versioned you also have a full audit trail of who changed what, when and even in some cases why.) This means it is much easier for people to be compliant with your corporate processes than not to be! Given the investment you are making in process improvement this is great news. However xProcess is not a straight-jacket. If project managers choose non-standard ways to do things they probably have good reasons (especially as it’s easier for them to be compliant than not!). Such non-standard approaches are captured and visible within xProcess so that different approaches can be analyzed and compared. If they are successful it is easy in xProcess to take these new patterns of working (and new templates for deliverables, and new quality criteria, etc.) and make them part of a new or extended process.

"Why do it? Because having the best possible process gives you the highest competitive advantage. Having no standard process is wasteful, confusing and lacking in control. Having the wrong standard process – even constraining teams to quite a good process – is costly in resources, quality and morale. xProcess delivers flexible processes to running projects in a way that they can easily follow them and modify them (in a fully auditable way) as they need to. It leverages the process knowledge in your teams as well as providing them with the best possible support from process experts. It’s the best way to start the optimization of processes that will make your teams more effective than the opposition."

3PE - why I use three estimates where one might do!

xProcess uses three-point estimating (3PE) for tasks and projects. Why?

Firstly what are the 3 points? Simply the best case, the worst case and the most likely. Just like when you ask how long the repair work will last and you get the reply - "I think it'll take 5 days but if we're lucky it could be 4. Or maybe it'll be much more difficult than I think. It might even take 10 days if there are unexpected difficulties."

In this case the person carrying out the work has given you more information than simply saying it'll probably take 5 days. We have a handle on the anticipated risk associated with this task. The risk associated with the estimate is captured in the gap between best and worst cases. A very wide range shows that he really has no idea how much effort is required for that
task!

The nice thing about giving xProcess 3PE's is that it give you back 3 potential end dates - the 50% end date (you're as likely to finish by then as to overrun), the 75% end date (just a 25% estimated risk of overrrun) and the 95% end date (overrun risk down to just 5%). It give you as a manager or custmoner more information to allow you to handle contingencies. These dates are sensitive to on-going updates to information about resource availability, other task priorities, task dependencies and so on, and if these change the forecast dates change immediately.

The one caution to make about the three-point estimating technique is that it comes up with such authoritative estimates for what we know is a risky and uncertain business. xProcess is only manipulating the estimates that you give it and there are a string of assumptions involved in both the mechanism of making the estimates and the higher mathematics that are used under the covers. My conclusion? Well if you can find a bookie to give you odds of 20 to 1 that the project won't slip its 95% end date I'd take it every time! However the main thing is not just to look at the forecast end date (the 50% date). If the 75% and 95% dates are much later start planning for overruns or marshall your resources to reduce the risks.

Two old friends ALF and BIRT

xProcess is an environment that covers a wide area since it is involved in all aspects of processes for projects. As a consequence it often overlaps or interacts with information that may be held in other tools or data sources. Its ability to integrate smoothly with information from elsewhere is a key to the products on-going success. There are 2 Eclipse initiatives that Ivis and the xProcess team are currently involved with, that will be bearing fruit in the product over the next few months. They are called ALF and BIRT.

ALF stands for the Application Lifecycle Framework. It's a Project is a subproject of the Eclipse Technology Project. Its purpose is to create a technology framework that will enable a diverse set of vendor tools, irrespective of architecture or platform, to exchange user data, manage business processes and collaborate in support the chosen ALM infrastructure technologies in use by development communities. After initial discussions with the ALF team (see Kevin Parker's Blog) we're expecting to be participating in the project and ensuring that xProcess is able to both generate ALF events and be called from ALF Service Flows. You can find out more about the ALF project from http://www.eclipse.org/alf/.

BIRT is an open source, Eclipse-based reporting system that produces compelling reports for both the web and printed formats. It provides core reporting features such as report layout, data access and scripting. It's integration with the next version xProcess means that it will be straightforward for customers to specific and generate their own documents and reports. BIRT's home page is at http://www.eclipse.org/birt/.

Both these projects will be represented at EclipseCon - an event where you'll also be able to meet the Ivis team of course, and see xProcess in action!