The headaches of centralised data

The UK government's woes concerning the potential loss of personal data has received a lot of publicity in recent months. It calls into question the whole strategy adopted by governments to build ever larger centralised databases containing more and more integrated data from multiple applications such as passports, social security, driving licences and health service. An insurmountable security nightmare arises from developing such databases. Just who can you trust absolutely to control access to such universal data sources?

Here's a radical thought - instead of centralising the data and building multiple applications around the same centralised database, why not build applications that can safely access multiple data sources, allowing departmental control of the more localised data sources with full version control and audit of all data access. Interestingly when we built xProcess, the data architecture we adopted lends itself to just such an approach. We've since named the framework XAPPA (eXtensible APPlication Architecture) and its continued development promises much in terms of rapid, model-driven application development for distributed systems.

Key benefits include:

• Lets multiple “centres” manage the data they own and freely link to data owned by other centres.
• Each centre manages access and audit
• All data versioned
• All access auditable
• Every change recorded
• Uses industry standard protocols for versioning
• Uses XML files (optionally encrypted)
• Transfer objects support Web2 applications and tools
• Applications built with MDA/DSM
• Applications can share data sources.

Frameworks like XAPPA may yet provide the means for both the integration and distributed management of data that future personal data systems will need. While all data storage and retrieval systems suffer from the risk of penetration, distributed systems at least limit the risk of total data penetration, which is the doomsday scenario few seem to realise is only too likely with conventional database approaches.

Document-driven processes

Many processes are driven by documents. Documents are the resulting artifacts from tasks of the process, as well as being the inputs or informing artifacts for subsequent tasks. Examples of such processes range from drug administration, planning applications, legal processes of many kinds and financial control. Since many of these processes are also subject to regulatory control and audits it makes sense to define and monitor them within an environment like xProcess, which not only provides feedback on the forecasts and targets for task completion, but also can handle the version control of the document templates, and the documents themselves.

Modelling these processes requires understanding of what the key documents are, their format (which can be captured as document templates) and the transformation work which takes place through the process (which can be captured as task patterns). The quality control to be executed at each stage can be captured as gateways.

Here's a simple example.

Midway through Sprint 01

Time to revisit our example Scrum Project started a couple of weeks back. How's our team doing? Here's the burndown as it was 2 weeks ago.

When we left it, the Sprint was red indicating an overcommitment based on current estimates. Either the team would need to negotiate a slight reduction in scope for the Sprint or risk disappointing the project stakeholders at the end of the period.

Here's the situation today (click for an enlarged view).

The status of the Sprint is now amber. Comparing the charts carefully one can tell from the reduced size of the work that there is a reduction in scope (also visible from the task lists) but there is good progress visible from the closed tasks to date. Other things to note in the screenshot are:

• Alerts (showing where some open tasks have exceeded their "best-case" estimates - they are prompts that the 3-point estimates for these tasks may need review);
  
• Closed tasks in the Explorer view (show with check mark); and
  
• The task list for the Sprint (sorted by end date so that the at-risk backlog item is highlighted at the top).

Participating in the project

The key thing about defining processes and project in xProcess is that the participants on a project can interact with the plans day by day, making them not just a view of what someone once hoped might happen, but an up to date record of the most likely outcome given progress so far.

The easiest way for project members to update their estimates, progress and plans is to use the web client of xProcess. By installing the xProcess server, this becomes accessible to team members using just a browser. They log in using their Subversion password and can then see their task assignments and update time bookings, which tasks are active, task estimates and descriptions, and other aspects of the plan relevant to their work. Here's a typical screen shot taken from our Scrum Project example (click on the image to enlarge).

Different filters can be used to show either just tasks scheduled for this week or all tasks assigned to the participant. Clicking on a task bring the task details in to the lower panel where they can be reviewed and edited. If there are documents or other artifacts associated with the task these can be seen and downloaded from this screen.

This article is part of a series of posts on Using the pre-defined Scrum Process. Check out the next article in the series here: Midway through Sprint 01.

Adding structure, artifacts and gateways to task patterns

Task patterns can be very simple (just one task with no extras) or a quite complex template controlled by several parameters and providing the implementer of the task with template documents, quality checklists (gateways), variable estimates and a family of subtasks. It's important to note that often simplest is best! Or at least (following Einstein) "as simple as possible but no simpler". We already discussed this issue in the blog with regards patterns for FDD (see Comparing big patterns and small ones). In Scrum, we've shown that different patterns can be used in addition to or instead of the Backlog Item pattern (see previous posting). Now let's look at what additional features might be useful to add to our Enhancement and Defect patterns.

Depending on the size of Scrum team you may have specialist roles that go beyond those of

• Product Owner
• ScrumMaster
• Team Member

For example the specialist role of Tester is useful on nearly all but very small projects. Developers must also write tests and test their code but Testers are generally more focused on end-to-end tests, user acceptance tests and full system tests. They may also have a role to play in accepting enhancements and defect-fixes. So for example we could identify subtasks in the Enhancement pattern with these different roles.

The alternative to multiple tasks, each requiring a single person with a single role, is to specify one task with multiple people, possibly with multiple roles. This would mean that each of these people would book time to the same task. Although this results in fewer tasks ,it's not necessarily simpler since the completion of the task is less well defined - basically the last one to finish closes the task. Experiment with what works best in your team - and let us know the outcome. We find such feedback invaluable.

In subsequent posts we'll look at:

• Adding gateways
• Artifact templates
• Using parameters

These are all ways to enhance the task patterns and make them easier for managers and team members to use and be productive.

Different patterns for Backlog Items

If you've followed through the series of articles on Scrum (see Using the pre-defined Scrum Process) you'll know they've covered making a Scrum project in xProcess and setting up a backlog of tasks that you can prioritise, forecast and display in Burndown or other charts. Participants on the project can access their tasks in the plan and update progress or modify estimates to complete them. But the pre-defined "Basic Scrum" process is just that - basic. It doesn't take into account the type of project you are doing with Scrum (it may not even be a software development project), nor the specifics of your company's or your team's process. With a little process engineering we can make things much more specific, and sometimes that's very useful.

The types of things you might want to consider once you're up and running with the basic process include:

• Adding different role types if there are areas of specialisation within the team
  
• Adding artifact templates for essential documents
• Adding gateways to tasks (quality checklists)
• Adding structure to the "Backlog Item" pattern, for example subtasks of a standard form
• Adding different types of Backlog Item with different roles, estimates, gateways, structure and so on.

For example you might want to distinguish between a defect and an enhancement and use a slightly different structure of subtasks and roles for each. If you want both patterns to be used as backlog items you need to modify the Scrum Project pattern so that they can be added by Project Managers / Scrum Masters to Sprints and the Unscheduled Backlog.

This diagram shows the hierarchy in the Scrum Project pattern. It consists of a project, named in the pattern as "$Name$" (this string will be substituted with the name the user supplies for the project). It has a General Overheads... task and a Scrum administration... task, both of which cover background work not related to backlog items. The Backlog Item pattern appears 3 times within the project pattern: in the first sprint, Sprint 01; within the Sprint pattern (which can be instantiated within the Sprints parent task as many times as required for Sprint 02, Sprint 03, etc.); and within the Unscheduled Backlog task. In order to add additional or alternative patterns, all 3 of these locations will need to be modified slightly so that the new patterns appear in place of, or additional to, the Backlog Item pattern.

Here's the situation after adding 2 new patterns for Defect and Enhancement as alternatives to the general Backlog Item. This will give the Project Manager a choice of the types of backlog item she wants to make in each case.

One word of confession here: to add extra pattern shortcuts like this means that the composite task that contains them (Sprint 01 for example) must be a selection composite rather than a collection composite. In version Basic Scrum 2.9.0 they are collections so you either need to change this or, to save hassle, use version 2.9.0a or later of the process (if this is not yet available for download, email support@xprocess.eu.com to get it.)

Next we'll look at adding structure, artifacts and gateways to our new patterns. Or check out the next article in the Using Scrum series: Participating in the Project.

Agile 42 at the Scrum Gathering

It was great to get together last week with old friends and colleagues Alex Aptus and Andrea Tomasini, who were over in London for the Scrum Gathering 2007. They are both doing Scrum consulting and training in Germany. Andrea's company is called Agile42 and if you think about it the name has some logic to it - "agile" is the undisputed answer to all the major questions of software engineering (mmm...?) in the same way that 42 is the answer to that only slightly larger chestnut: life, the universe and everything. Actually I'd recommend Alex and Andrea's work precisely because they understand that the answers to large questions are not trite or formulaic. If you manage to engage them for your project you'll not only get excellent training in standard methods but thoughtful and experienced consultants that will help you solve the hard questions too.

Another highlight of our meeting was seeing the newly opened St Pancras station on the day that the first Eurostar train arrived there from Paris. The queues at Europe's longest champagne bar meant that we weren't able to order their most expensive bottle at (literally) £2800 a pop!

By the way Andrea has a blog on Software Development Process in Practice. Why not check it out. He has a thoughtful article for example on requirements management. Here's the link: Software Development Process in Practice: Agile Product Development... How to manage Requirements? Effective requirements management has been shown time and again to be the most crucial aspect of successful projects so it's a worthy topic for discussion. I think it's an area that all of us involved with agile processes should be examining. My feeling is that agile methods stop too readily at the level of "features" without also trying to capture other aspects of requirements. 3 Types of Requirement tries to explain this - it summarises the results of some consulting in this area for a client adopting agile methods on very large projects.

Top-down or bottom up planning for Sprints

xProcess supports two ways to specify the required resources for a parent task: top-down or bottom-up.

The simpler approach is bottom-up. A parent task has subtasks with their own estimates of size and effort. The parent task size is therefore just the sum of subtasks' sizes.

What if you don't yet know what all the subtasks are? In this case - or if you want to specify some contingency or slack without inflating the estimates for the subtasks (Parkinson's Law would suggest this is a good idea) - then you can supply an independent estimate for the parent task by specifying it as top-down.

A previous entry, Parent Tasks and Top-down Planning, has already discussed this feature and shows the user-interface for using top-down estimates. The question here (since this is part of the series on Using the pre-defined Scrum Process) is how to use it for Sprints in Scrum.

Let's take a look at the current state of "Sprint 01" in our example Scrum project. Although this is quite a detailed screenshot, just look particularly at the red figures in the middle and the Alerts at the bottom of the screen (click on image to make larger).

The negative numbers for the estimates of size and effort for the "REST" of this task, and the Alerts, tell the same story. The backlog items in this sprint exceed the size of the top-down estimate. (See Prioritising the backlog for how and why these items were added to the Sprint.)

In this state the top-down estimate really is of no use since the larger amount of effort required for the child tasks will determine the time it takes and the top-down estimate of the parent task will effectively be ignored by the auto-scheduler. The auto-fix for the Alerts provides 2 ways to solve this situation:

You can either increase the estimates so that once again the top-down estimate is greater than (or equal to) the sum of the subtasks, or simply convert it to bottom-up so that the estimate for the Sprint just reflects the size of its parts.

So Sprints which already have a complete set of tasks added to them should usually be set to bottom up. However Sprints which do not yet have their backlog items defined should be top-down. That's why when you create Sprints for the first time they are top-down, and as a result you can see the forecast assignments for people and the projected cost of each Sprint of the project, based on its top-down estimate.

The general rule for how to decide whether to use top-down or bottom-up is this:

• if you think that you have captured all the work required in the subtasks, then use bottom-up
• if you think that having completed all the identified subtasks that there may be some others, then use top-down and provide an estimate for how large these other subtasks are in the top-down estimate.
  

These other subtasks are what is referred to in xProcess as the "REST" of a parent task, or the Remaining Essential Sub-Tasks. When you discover them you can create them in the parent task and task-by-task the size of the REST will be reduced, Or you may become confident as time progresses that you have indeed identified all the subtasks so you can then convert the parent task to bottom-up estimating and the REST will be removed.

One more point about parent tasks: team members cannot book their time directly to a parent task, only to a leaf-level task. So if they find that they are carrying out part of the parent task, but not one of the identified subtasks, they may need to create a subtask so they can book to it. This is another way in which the REST of the parent task may be reduced.

The next article on using the Scrum process in xProcess concerns configuring your process with Different Patterns for Backlog Items. Or if you're more interested in how participants on your project can use the xProcess web client, see Participating in the Project.

Prioritising the backlog

The key point about priority-driven planning approaches like Scrum - common in fact to all agile processes - is that the set of tasks undertaken by the team are ordered by priority rather than by a pre-determined procedure. (Priority is usually assessed based on business benefit and risk.) This article focuses on this particular aspect of using Scrum with xProcess and it covers:

• Adding backlog items to specific Sprints
• Showing a Sprint in Burndown and Gantt charts, and finally
  
• Prioritising the backlog

Having created a Scrum project with resources and a set of backlog items (see the previous post in this series) our next step is to prioritise the backlog. Our starting point here is a project which has three Sprints defined and a number of backlog items that have been estimated for approximate size. For simplicity in this example I've created 26 backlog items (with single character names from 'a' to 'z') with sizes of between 1 and 5 "points" (points being nominal units of size corresponding approximately to an ideal day's work for the average worker - see earlier post for more about this topic). Here's a view of my project at this point (click on the image to make it larger).

For the backlog items to be scheduled they must be moved from the "Unscheduled Backlog" task to one of the Sprints. Maybe you have an idea at this stage about the most important tasks, or maybe you want to see how long they would all take given the current estimates and resources. In either case just select the backlog items in the task list and drag-and-drop them to the first sprint ("Sprint 01"). As you'll see in the screenshot, I've opened the Project Explorer pane to do this. It's a view that can be selected from the "Window" menu - or simply click on the project icon next to the project name on the "Tools" pane. Once the backlog items are in a Sprint you'll begin to see the forecast start and end dates for them. You can now hit the "Priorities" button on the Tools pane and begin to put them in priority order. First though let's look at the Gantt chart for "Sprint 01" having added backlog items 'a' to 'v' to it. Double-click on "Sprint 01" to see its details and click on the "Gantt Chart" button. You can view other charts for the Sprint in a similar way, for example by clicking on the "Burndown Chart" button. In the screenshot below you'll also see that I've put the task list in reverse End Date order (by clicking on the column heading) and on the Gantt chart I've scrolled down to show the tasks at the end of the Sprint.

The Gantt Chart shows the forecast start and end dates of the tasks based on resources and estimates so far defined. As time progresses and tasks are started, re-estimated, completed or changed this chart will also change of course - it gives us a view of the likely outcomes based on information currently available. The risk element of the estimates is shown by the 75/95% extensions on the Gantt bars (see "Why I use 3 estimates..." for a discussion of this feature). Given that the target end date for this Sprint is December 19th, we can see that 2 items ('n' and 'a') are in the red zone and not forecast to finish in time and several others are in the amber zone and at risk of not finishing. This would therefore prompt review with the team and the client to whom commitments are being made so that expectations are set appropriately.

Now what if 'n' and 'a' are actually the most important items to get done in this Sprint, or what if there are dependencies between some of the tasks? How are these handled? Well dependencies can be added simply by dragging-and-dropping a task on the task it is dependent on. But it's important for flexibility to keep such dependencies to a minimum. The main way to order tasks in the Scrum process is by priority. To see and change priorities click on the "Priorities" button on the Tools pane. Here's what you might see...

The priorities of all the tasks in "Sprint 01" are all the same (they're all priority one!). As a result the scheduling of the tasks is effectively random. Let's prioritise then in alphabetical order. Here's what the priorities list looks like then...

Changing priority changes the order that the tasks are scheduled, so the Gantt chart now reflects the alphabetical priority ordering. Here's the resulting Gantt chart for "Sprint 01".

And here's the projected Burndown chart for the same Sprint. It still shows a red status as we've not removed tasks by prioritising, but it reflects when the newly ordered tasks are forecast to close. For more on how burndown charts are used as the Sprint progresses, see "Don't burn out - burn down!".

The next article in this series looks at the significance of Top-down or Bottom-up planning for Sprints.

Using the pre-defined Scrum process

I've recently been asked for more guidance about using the pre-defined Scrum process in version 2.9 of xProcess. It uses some of the new features for process definition (like Composite tasks) and also has "top-down" estimating for projects built-in, which is useful for resource planning and financial reporting. So I'm planning to post a series of blog entries on the process and in this one I'll focus on getting started with this process "out of the box". Here's what it covers:

• Creating a Scrum project
• Adding resources to the project
  
• Adding Sprints to the project (a Sprint in Scrum is like a "timebox" in other methods)
• Adding Backlog Items to the project (Backlog Items are the tasks the project must achieve)

So step one is to select the Project Manager perspective (the default) and hit the "New" button so we can make a Scrum project. The New project dialog shows the processes that are available for import, so if you've not already done so this is the point to import "Basic Scrum 2.9.0" and then select "Scrum Project" as your type of project. You can fill in the name and other details of the project (such as the number of 4-week Sprints and a location of a Wiki if you are using one) on this dialog.

At this point take a look at the set of tasks that are already in the project (hitting the "Tasks" button is the quickest way to do this. Here's what you might see (click on the image to make it full size).


There are:

• some overhead tasks (green because they have fixed start and end dates and are within the target dates of the project)
• A "Sprints" parent task containing one sprint "Sprint 01" (these tasks are red since they have NEVER end dates, clearly after the project target)
• An "Unscheduled Backlog" (which is grey as this task is specified as not to be scheduled, also resulting in NEVER end dates).

In the pane below the project details, you can also see there are some alerts displayed telling us that this project needs resources with particular role types. That's a clue to our next step - add resources. Hit the "Resources" button on the "Tools" pane and you'll see an "Add/New Project Resource" button. If you've already defined or imported a set of people you can select those who are going to be on the project, or you can create new persons at this point. Add six team members (Participant role) and one team leader (Scrum Master role) to the project.

At this point we should see actual end dates appearing for the Sprints and the overall project and we can even see a Gantt chart and projected Burndown chart. The Burndown chart is slightly more interesting if we add 2 more Sprints to the project (hit the "New" button and select "Sprint"). Then hit the "Burndown Chart" button on the project pane and you should see something like this:

Now it's important to note that as yet we've entered very little information about this project other than the number of Sprints and the people on the project and their availability. The projected burndown is only reflecting this simple data showing when the work in each Sprint (based on estimates for the amount of work in a Sprint which is modifiable in the process definition or, for a specific Sprint by modifying the estimates in its "Required Resources"). To use Scrum in earnest we need some actual "Backlog Items". Backlog Items are identifiable tasks that in a software development project for example could correspond to features to be built or defects to be corrected. The next step therefore is to add such items.

Hit the "New" button again and select "Backlog Item" (and then "Next"). We have a choice of the 3 Sprints we've defined or the "Unscheduled Backlog" as valid parent tasks for these items. Setting the parent task as Unscheduled Backlog means that we can create a set of tasks without them changing the scheduling of the project. We can then decide later in which Sprint specific tasks should be completed. So with Unscheduled Backlog as the parent task you can give the backlog iten a name and a size (size is measured in "ideal days" by the way), and hit "Finish" and the task will created. If you already have a list of tasks you can copy and paste a comma-separated list into the name field of the "New" dialog, check the "Create Multiple" check-box and several items will be created in one go.

Now we have a project some resources and set of tasks to be completed. The next step is prioritising. We'll look at that in the next posting in this series.

Searching the blog by topic

Blogspot offers some useful facilities for finding blog entries by topic. Below most entries there are a series of labels. Clicking on any of these will bring up other related articles on the same topic. Try it our with the labels below this post.

xProcess Europe

On November 6th Ivis announced that it had demerged its European operation (see Press Release) . The new company is known as xProcess Europe and it's owned by former members of the UK team including Andy Carmichael and Paul Kuzan both seminal thinkers in the evolution of xProcess.

Andy Carmichael: "Paul Kuzan and I are both very pleased to be involved with the on-going evolution of the xProcess product and brand, and we believe xProcess Europe provides a great opportunity, particular for developing the services offering that customers adopting xProcess are often looking for. Providing better support and expertise for companies wanting to improve the processes behind their project and portfolio management will be one of the main focuses of the new company. While our product sales and support work will primarily be targeting the European market, we will continue to offer process improvement services world-wide, wherever the needs exist."

xProcess Europe and Ivis Technologies are both supporters and contributors to this blog. From time to time you'll find links and references to both companies here. For more details try their web sites: www.ivis.com and www.xprocess.eu.com.

The Simple Process... or is it?

The Simple Process in xProcess is the first one you are likely to use. It contains basic patterns for a project, a task a timebox and so on, and it also has some useful actions in there that can be used in your own patterns or even exposed as "User Actions" so they can be called on your projects.

The Simple Process has several elements that you cannot delete as for example xProcess assumes that even if no other Role Types exist (such as Project Manager, Developer, Tester and so on) at the very least the Participant role type will exist. When you define your own process it's likely that you will inherit Simple Process patterns (though you can hide them by defining your own hidden patterns with the same name. So it's definitely worth taking a snoop at all the elements in there. You might even want to cut and paste some bits into your own processes and modify their behaviour.

Latest patterns for Scrum

xProcess 2.8 is being released this week and among the exciting new capabilities are some new sample processes, including updates to the Scrum patterns. Here are some of the highlights.

Firstly the project pattern for the "Basic Scrum" process is now built from composite tasks so that there is a standard structure that is provided for all its projects. The Hierarchy diagram shows the structure, having top-level tasks for: General and Scrum administration overheads (overheads are unplanned or unplannable tasks that take up a proportion of everyone's time); the Sprints (a composite task which contains an iteration of Sprint instances); and the Unscheduled Backlog (a composite which contains a collection of Backlog Items).


You can also see from this diagram the structure of the Sprint and Backlog Item patterns. Sprint is a collection of the Backlog Items to be scheduled over the period, and consists of a single task. The beuty of xProcess of course is that these patterns can be modified to match the way your team is working or the particular application of Scrum that you are using. For example if you are using Scrum with XP you might want to rename the Backlog Item pattern to User Story and perhaps provide some structure to the set of tasks and role types that corresponds to0 the way you are using the method.

When you make a new Scrum Project in xProcess you'll be prompted for values of the parameters of the pattern. You can see these in this screenshot, where as well as defining the name of the project the start date, duration (the number of 4 week Sprints) and the Wiki location.
The Wiki location is not a required field but we've found many people find wikis are a convenient way to group documentation. By providing the location of the wiki in this dialog the descriptions of your backlog items are automatically transferred to a wiki page while also being browsable and editable in xProcess views.

Having created your Scrum project from this pattern, the next thing to do is add your resources. You need to add someone with a Scrum Master role and the others with Participant roles - let's say six participants in this example.

Because the tasks in the Scrum pattern are defined as top-down, even at this stage xProcess is able to provide a project schedule, based on the specified duration of the project. More usefully though, this is the time to create the first Sprint (right-click on the Sprints task and select New...) and then create some Backlog Items, either in the Sprint or in the Unscheduled Backlog if we don't yet know which Sprint we want them to be completed in. Backlog Items are prioritized, firstly from the particular Sprint into which they are created or moved, and secondly by the priority order of the tasks within a Sprint. You can edit this priority order by opening the Sprint editor, clicking on the priorities tab and either dragging and dropping or changing the priority number.

Once you have a Sprint which contains the items the team will be implementing, open the Burndown view. This provides both the history of how much work has been completed so far in the Sprint and a forecast of what should be completed. You'll see immediately for example if there's too much work in there and forecast dates are beyond the end of the Sprint. Here's an example of just one such Sprint... (click on the image to see more detail)Looks like some discussion needs to go on as to whether all the functionality is vital this time round!

Latest patterns for FDD

The release of 2.8 this week sees some interesting updates to the FDD patterns. Like the Scrum updates for xProcess, the new FDD process uses "composite tasks" to define the structure of FDD projects, allowing them to be expanded top down as the details of feature sets and features are defined.

This hierarchy diagram shows the structure of the FDD project pattern. For those familiar with the method, the 5 stage process is visible in the top-level tasks:

1. Develop an Overall Model
2. Build a Features List
3. Plan by Feature and
   
4. combined with 5. Design by Feature - Build by Feature.

Note that the darker blue nodes are patterns, so these can be further expanded in the hierarchy diagram or viewed in the graphical pattern editor.

Here for example is the hierarchy view of the Feature Set pattern, which groups together a set of related features into a deliverable package which can be scheduled for a particular release. Note that here the "Features" composite task, as well as containing Features, may also group Defects (work on correcting issues from previous releases) and Tasks (any other work related to the Feature Set).

The Feature pattern has already been the subject of a number of postings on this blog (see for example...). The pattern diagram shown here uses a quite traditional view of the tasks required to deliver a feature with the emphasis on Specify (the "design by feature" part) and Develop (the "build by feature" part). We can see more detail of this task pattern by using "Go Into" in the pattern editor. Here the specific artifacts and quality gateways that are associated with each task, as well as the number and types of roles required to carry out the task, can be reviewed and modified.
Clearly other ways of breaking these tasks down are possible, and some of the FDD literature (for example Palmer and Felsing's book Feature Driven Development) provide alternative schemes that place stronger emphasis for example on design and code walkthroughs rather than the test-driven approach implied here.

The beauty of xProcess is that the task patterns are all easily and graphically editable so you can make this method match exactly what you want the FDD teams to carry out. As all plans, tasks, artifacts, processes and time records are stored in the xProcess versioned data stores, the compliance with your process can be monitored at any time, either to improve the process patterns where teams have discovered better ways of working, or to improve the teams' approach by following the best practice captured in your process patterns.

FDD has a three-level hierarchy of functionality: Features, Feature Sets and Major Feature Sets (also referred to as Business Areas). So far we've seen where Features and Feature Sets appear within the hierarchy of a project. Major Feature Sets (MFS's) are handled slightly differently. MFS's are created by an xProcess pattern in a similar way to the other patterns we've looked at, but they result in categorized folders rather than tasks which are part of the project hierarchy. This is so that the Feature Sets are visible within the Gantt charts and release schedules. Because of the scope of major Feature Sets, they generally do not have such a clearly delineated start and end date and so it would not make the project schedule clearer to include them in that way. So opening a MFS folder shows you all the feature sets and features in that category. You can then review and prioritize the features in just this one business area. Here's an example of a MFS in a particular project with its corresponding Gantt chart (click on the diagram to see more detail).
Finally in this brief review of xProcess FDD, the pattern for a Release is also instructive. Again a prioritized folder is used for the Release pattern, rather than a task in the project task hierarchy (see the Scrum method for an example of a process that uses this alternative). By default just one release will be created in the project (with a target close to the end of the project). However the Project Manager can create as many additional interim releases as he requires. The "scope" of the release is defined by moving the required features/feature sets/tasks into the release folder. The scheduler of xProcess uses the implied and defined priorities of these tasks to calculate completion dates and provide alerts if targets are unlikely to be met. As the project progresses, the scheduler uses the input from team members completing and possibly re-estimating the required effort for tasks to give immediate visibility of targets' status and costs providing all stakeholders of the FDD projects timely and detailed information to support decision making and further planning.

Set size to match effort

I explained in another entry that size and effort are distinct metrics for tasks. However it's useful to keep the overhead of estimating two closely related items to an absolute minimum. This is why the patterns in the "Simple Process" uses the user's input of a size metric (in "ideal days") to automatically complete the effort estimates (usually displayed in person-hours or person-days). They can be changed subsequently though and if you want to re-synchronize them at some point - say when you a re baselining a plan - a utility for doing so is useful.

Here's a simple Action that you can define in your process (make it a "UI Action" by setting this field to True).

Action Name: Set size to match effort
Expression: setSize(getEffortIncludingChildren()/480.0)
Applicable to: Task
UI Action: True
Parameters: (none)

Having done this you can expand your task hierarchy in the explorer, select the tasks you wish to set size on, and then right-click and select UI Action -> Set size to match effort .

Note: From version 2.9 of xProcess this action is included in the Simple Process.

ITIL We Meet Again

I was talking with Charlie Betz recently about ITIL, the best practices framework for delivery of IT services, and how xProcess might be used to support service delivery applying these practices. xProcess is well suited to "case management" type projects because of its ability to handle relatively small task patterns and create composite patterns of collections, iterations or selections of smaller patterns. With v3 of ITIL taking a more "lifecycle" approach, it brings a greater emphasis on process and stronger motivation to understand and improve underlying process models. That's why I'm particularly interested to find ITIL professionals wanting to model their processes with xProcess.

Any way I've been browsing Charlie's blog which has some interesting observations, and not just on ITIL. This entry highlights some frustrations with ITIL from an SDLC perspective: My first ITIL v3 presentation. It brought a smile to my face at least.

(Apologies to Diana Ross for the title of this entry.)

xProcess wins in the "Tools and Environments" category of the SD Times 100 Awards

The 5th annual SD Times 100 recognizes the leaders and innovators of the software development industry. The 2007 SD Times 100 was posted on the sdtimes.com Web site today, and it names Ivis's xProcess as a winner in the Tools and Environments category. The award citation reads:

xProcess changes those lovely best practices into die-cast working frameworks for your developers. The easiest way to push some sanity into the development process.

David Rubinstein, Editor-in-Chief of SD Times, said "The winners of this year's SD Times 100 awards have demonstrated their leadership in shaping the software development industry. We took into account each nominee's products and services, its reputation among development managers, and the new ideas it brought out. These select individuals and organizations are the ones we've identified as helping to move the art of development forward."

The team at Ivis are proud to be a part of this select band creating the new breed of tools that will equip organizations and projects to deliver more, faster with less!

Parent Tasks and Top-Down Planning

A key aspect of agile planning is that you expect the plan to change. That doesn't make it a bad plan, it's just the baseline from which you can judge whether to accept certain changes and what impact can be expected. To help you build flexible plans xProcess supports a very important feature - top-down planning. Parent or composite tasks can have their own effort and size estimates which are independent from the size of their children. Alternatively you can select Bottom-Up planning of the parent task and its size will be defined entirely by its children.The above screenshot is part of a task editor that is defined as a top-down parent task. It displays the size (15 points) and effort estimate (120 hours) but also shows how much of this is already allocated to subtasks and how much remains in the REST of the parent task.

The advantage of this approach is that parent tasks can be defined and estimated before the details of their subtasks have been defined. This works for both parent tasks and composite tasks which are collections, iteration or selections of task patterns. As Project Manager you can plan near-term activities in fine detail while leaving tasks that are further into the future as larger activities that will be planned in more detail as time progresses.

Outsourcing for quality

Some people think that outsourcing is about minimizing the costs by moving work to wherever rates happen to be lowest at the time. The costs of distributing software teams can be high however and such a strategy may fail to reduce costs at all, and worst of all it can impact quality and agility.

There are good reasons to distribute teams however - for example to maximize quality by picking the best available expertise.
Ivis found such a partner in SwiftTeams who are based in St Petersburg. Their expertise in graphical editors for development tools is second to none, and the xProcess product has benefited enormously as a result.

How then do you overcome the difficulties of planning and coordinating teams that are geographically distributed? This is one of the key motivations of course for deploying xProcess. It provides the key visibility and communication layer and minimizes the negative impact of having a few thousand miles between the sponsors, planners and specifiers and the implementers. A number of key enhancements to xProcess in recent months have been motivated by improving our application of the Scrum process with SwiftTeams. We'll be looking for more in the coming months too.

Have you earned value today?

There are many ways to track progress on projects - see "Don't burn out - burndown" for example. Burndown is particularly useful in agile projects with timeboxes where the targets are set at quite short notice - month by month for example. Earned Value (EV) analysis on the other hand is a great technique for tracking progress against a stable plan (ultra-agile teams go and read that burndown article instead of this one - stable plans are a luxury you rarely see!).

With EV tracking the value of completing tasks is calculated from its originally planned cost. There are some disadvantages of this - not least that the whole method relies on having the well-estimated plan to start from - however its advantage is that efficiency of execution can be measured day by day on the project.There are two relevant measures of efficiency that EV analysis provide: cost efficiency (how much better or worse you are doing than the baseline plan with regards the cost of the tasks you have delivered so far); and schedule efficiency (how much more or less value you have delivered at this point in time compared to the baseline plan).


xProcess supports EV analysis in the Executive Dashboards client (see screenshot). The Executive Dashboards provide an overview of multiple projects (or just one project if you like) so it's an ideal place to look at these efficiency levels in different projects. We may revisit how to use the views provided in later blog entries so watch this space.

Great resources for estimating

There are two new books on estimating that I've been finding very useful recently, and one older one. The older one first; and here you understand I'm not referring to Larry Putnam Senior himself! It's his book, co-authored with Ware Myers, "Measures for Excelence: Reliable software on time on budget" that remains a classic, full of helpful advice and packed with actual measures from real projects. I met Larry in 1992 just after his book came out, when I was in Washington with Ed Yourdon presenting some research on software reuse from Japan. He kindly gave me a signed copy and it has been in frequent use since that time, and was a key source for my own book "Better Software Faster" published in 2002. Putnam and Myers' latest book "Five Core Metrics" is another excellent resource which highlights the essentials that all software projects should measure, with practical information about how to do it. In case you're wondering the five are:

• Time
• Effort
• Quality
• Productivity
• Size

While Putnam and Myers experience covers projects ranging from the small to the truly massive, using a wide range of methodologies and covering five decades of software projects, Mike Cohn's book "Agile Estimating and Planning" is focused very much on the agile development processes of more recent history. It is similarly an excellent read with a mass of very practical advice for project managers of agile projects. I particularly liked the "Planning Poker" variant of the Wideband Delphi approach (!) but there are many other practical suggestions and insights, including discussions of burndown charts, velocity and prioritisation.

The challenge for me, working as I am on the product direction of xProcess, is to ensure that the full range of techniques and processes represented in these books is well supported by the underlying model and user interface of the product. xProcess has always been about agility, and in particular bringing powerful techniques from established project management methods and making them available to managers and participants of agile projects in such a way that they remain agile. These books similarly contribute to that drive to achieve controlled, productive and agile software projects.

Hierachy Diagrams

Hierarchy diagrams in xProcess are a good way to see an overview of a project or of process patterns. The diagram on the left is a project that is following a Prince2 style project - actually using a process defined by the WPM Group, leading project management consultants and a company I've been doing some work with recently.

The Hierachy Diagram is a good way to show Work Breakdown Structures for projects, and they are also useful editor views, where you can expand/collapse regions of the hierarchy and edit tasks via their right-click menu.

One more thing to try with Hierachy Diagrams... turn on the constraints on the diagram via the filter button and (as in this example) you'll see all the dependencies between the various tasks. Very useful for understanding the constraints imposed on the ordering of tasks.

Estimating size and effort - Why are they different?

A common question I get asked about estimating in xProcess (after three-point estimating of course, which is another popular question!) is: "What's the difference between a task's estimate for size and its (three-point) estimate for effort... and why do I need both?".

There are several ways to answer this, but perhaps a starting point is to ask why we think they're the same. If I ask you the size of your swimming pool you're more like to give me an answer in feet or metres that to reply with the number of person days it took to build the pool. (Building swimming pools is an xProcess application by the way - but that's another story!) When we are discussing the planning of a project the first thing to consider is the size (in whatever units are appropriate) of the deliverables of that project. Then we need a function (based on previous experience) to map from size to the estimate of the effort and consumables required to provide those deliverables.

Because the size of the things we want to deliver are measured in many different units, all of which we want to map eventually to an estimate of effort, it is common for projects to estimate directly in effort units such as person-days, person-hours or person-years. However if we do this, we lose a very important measure of the effectiveness of the project. Questions of team velocity or productivity become somewhat meaningless (except as a correction factor on original estimates of effort, compared to actual effort expended).

So if we can agree size is an important metric for projects, the next question is what units should it be estimated in, and can we compare the estimates with actuals at the end of the project. Software projects in particular are notoriously difficult to find an appropriate measure for size. Lines of source code is commonly used (because they are easy to count) but they don't readily map either to effective functionality delivered nor to effort required to develop them. Function points is another option and more recently "story points" (for an XP user story) or simply "points" have gained popularity where these are arbitrary units that are comparable only within the single project where they are being used. Another useful measure is "ideal-days" - in other words the amount that can be produced by a typical team member in a completely uninterrupted day. We rarely get such days (if ever) so it will take most of us more than a day to implement an ideal-day's worth of size. Nevertheless it is an understandable and reasonably verifiable measure of size.

In xProcess the units of size are common to the whole Data Source (they can be set in Data Source Preferences) and the value is used in several important places. When creating tasks the size value is often used in process patterns to set initial values for three-point estimates. The size units are used in burndown charts and earned-value plots to show progress of a project towards creating its deliverables. And size is also used to calculate productivity measures such as team velocity and software productivity.

Project Challenge! On your bike...

The National Motorcycle Museum in Birmingham was the setting for last week's Project Challenge exhibition. It was the first opprotunity to see the new Executive Dashboard for xProcess in action, as well as to hear yours truly speaking about process modeling as a route to improve project performance. Needless to say the Executive Dashboard was clearly the highlight!


Guy's interest in the dashboard in this photo is not merely a pose for the camera. The show was also the first opportunity for the sales team to see this feature, which was turning a lot of heads (not just ours!). The ability to create additional views of interesting project data "on-the-fly" is a particularly impressive aspect of the facility.

Don't burn out... Burn Down!

Following in the tradition of you heard it here first, here's another sneak preview of soon to be released functionality on xProcess. The burndown chart is a key visualisation that was developed in Scrum and now used with a number of agile methods. Currently you can use the reporting feature in xProcess to generate a burndown chart for your project or timebox. However there's a new feature in the wings that shows you the burndown instantly (history and forecast) on any project, parent task or folder. Here's an example:


This timebox (Timebox 01) runs for about 3 weeks and the scope of work targeted is forecast to complete right at the end of the timebox -- indeed a couple of the tasks are alread critical. (This diagram is looking forward, predicting when tasks will close, and thus when the work in each task will burn down). Look at the same time box a couple of days later.

Now -- perhaps because tasks have been re-estimated or resources changed -- we can see some tasks are forecast to finish beyound the end of the timebox. The chart is consequently shown as "red" status. Even at this early stage in the timebox we can predict that some deliverables are at risk and so take appropriate action. In this case some additional resources have been found and the some tasks excluded from the targets. Once the changes have been made, the effect can be seen and assessed immediately...

The timebox is "green" and all systems are go! Day by day as the timebox progresses we can monitor our progress and ensure the whole team stays on track!

When we visit the team half way through the timebox we can see from the burn down chart that they are still very much on track and confidence will be high that this team will deliver on its promises.

New project Pattern for FDD

The project pattern for the Basic FDD process, which is delivered with xProcess is changing in version 2.7.2. Here's a preview.


This puts the emphasis back on the familiar 5 stages of FDD at the top level (Develop an Overall Model, Build a Features List, Plan by Feature, Design by Feature and Design by Feature). The first 3 stages are included in Timebox 00 which covers the start up period of the project and Features and Feature Sets are created as subtasks of combined final stages: Design by Feature - Design by Feature. Major Feature Sets are modeled as Folders with membership defined by Category.

3 Types of Requirement

Priority-driven processes only work if you can model the process around task patterns for a single requirement. There is little point in prioritizing say, coding over testing, or design over specification, since all these activities must take place at some stage, even if they happen inside an activity that is called something else. But are all requirements the same (in the sense of using the same processes) or are some requirements more the same than others?!

Recently I was asked to advise on the requirements capture process for an organization wanting to apply agile techniques in a controlled environment. I came up with some slightly different names for types of requirement, though I think the concepts will be familiar to you if you've analyzed other software development methods. The requirements types (shortened to PCF) are as follows:

• Business problem statements (Problems)
• Solution contraint statements (Constraints)
• Solution feature statements (Features)

I've written elsewhere about FDD, which provides a very good process for managing and prioritizing features. However often projects applying processes like FDD do not pay enough attention to the other types of requirement that precede features.

Problems, for example, should identify issues in terms of a measurable aspect of the current solution and the degree to which a performance improvement would overcome it. Constraints express aspects of the solution that the current designers are not expected to change (though they should also state the rationale behind the constraint, and how and by whom it may be changed if justification exists). Only by considering all of these three types of requirement can the essential requirements of a system be captured.

Action stations

One of the most powerful aspects of xProcess is its flexibility and configurability. Processes are defined in terms of patterns: for example project patterns, task patterns and artifact patterns. Configurable enough. However you can do much more to enhance these and make them easier to use, by using xProcess Actions. These are bits of "OGNL" code that do things when xProcess is working. They can be triggered by user actions, by instantiating a pattern, by workflow or a number of other events.

For example Sam DerKazaryan of Isobar Global asked me recently for an Instantiation Action to increment the issue number on newly created issues. For those of you who like a spot of programming with their process engineering, here's the code snippet that does it...

#root = getPersistenceHelper().getRootExchangeElementContainer(),
#counter = #root.getIntProperty("currentCounterValue"),
#counter = #counter + 1,
setName("Issue " + #counter),
#root.setIntProperty("currentCounterValue", #counter)

Note: you need version 2.7.2 of xProcess or later to run this code.

Finalist for Eclipse Community Awards 2007

Just received great news - xProcess has been shortlisted (1 of 2) for the prestigious Eclipse Technology Awards in the category Best Commercial RCP Application. The two Technology Awards aim to recognize the best Eclipse-based commercial and open source products in the world.

"Finalists were selected from a pool of 63 nominations by a panel of judges from the Eclipse community". Way to go!

New white paper on priority-driven processes

Are your teams working on the most important issues to your business? If not you are losing potential earned value every day that goes by!

I've just finished a paper on this topic called Priority-Driven Processes, which you can download from the Ivis website. Such processes use the priority of deliverables as a key management control, enabling teams to focus on what provides greatest payback soonest. The paper is designed to help people to model and implement processes with priority, and draws on examples from FDD, XP and other agile methods.

If you do download, be sure to send us your feedback!

Do What You Do, But Better...

A recent discussion with Dave Garrett of Gantthead on the essentials of process improvement provoked some very interesting questions and answers. Please check it out on his blog under this title: Do What You Do, But Better...