Folders are changing

There's a new spec for Folders in xProcess that should be released shortly in the v3 programme. The new feature let's you define a hierarchy of Folders and it clarifies the difference between folders with "explicit" members (the tasks are in the folder because someone put them there) and "implicit" members (the tasks are there because they fulfil the criteria for the folder - usually that they are in the appropriate category.

Email me or post a comment below if you would like a preview of this spec.

Rescheduling tasks to the date you want

I was asked by someone recently to help them get their task to be scheduled on the date they wanted. One thing xProcess doesn't give you (except for overhead tasks whose size is defined by their duration rather than the estimated effort) is the ability to state when a task should start and stop. This is what the auto-scheduler does, based on your estimates, constraints (e.g. dependencies on other tasks or specific dates), your priorities and your available resources. So what if you have a task which is not being scheduled when you think it should?

Here's the checklist I gave my colleague:

1. If the task being scheduled too soon you can add a constraint to the task by opening its editor (double-click) and selecting the “Constraints” where you can add the constraint (e.g. a dependency on another task or a “start after < date >” constraint).
   
2. It may be though that the task is being scheduled too late and you want it scheduled sooner. A quick and easy way to achieve this is to make it “active” as then it will be scheduled immediately. (Just right-click on the task and select “Make active”. This can also be done from the web client too and it’s worth checking that the team always make the tasks they are working on active so that they are always scheduled immediately and not postponed when higher priority tasks are defined unless they make them inactive.)
3. If it’s not currently being worked on (so not “active”), but you want it scheduled earlier (for example as soon as current active tasks are complete) then increase it's priority. Here's how:

If the task is not in the current sprint (assuming you're using the Scrum process) it will be scheduled after this sprint, so drag it into the current sprint from its current location

If it is in the current sprint but still not being scheduled soon enough, use the “Priorities” button on the tool bar, which takes you to the “priority groups” tab for the project. Maximize the window so you can see it all and select the current sprint. In the lower list you can see the relative priorities of all the tasks in the sprint. Change them so your task is near the top.


Check in the first tab of this window “All tasks in Schedule Order” that your task is high enough. (You will see for example all the active tasks near the top of this list.)

Finally one more thing to check. Make sure the task doesn't have constraints on it's start or end, for example a dependency on another task that won't complete for some time. This will also cause it to be scheduled later.

IT&T Conference Galway

Ireland seems to be a second home for me at present with a number of clients using my services for migrating to agile practices through partners Neueda Technologies. So it was a pleasure to be back in sunny Galway (in the gaps between the gales!) for this conference at GMIT. My tutorial was on "Agility: driving software projects for value and quality" and apart from giving an overview of agile methods, particularly Scrum, it looked at some of the key issues for scaling agile processes including design, continuous integration and team structures.

Project Home Page

xProcess allows you to define a wiki, blog or other site as your project's home page. If you can see this page from with your project in xProcess you are pointed at the default page. Why not change this to a site where you can add details of your own project and invite team members to contribute.

Some processes are configured to expect a Wiki site rather than a blog. These sites generally allow pages which do not yet exist to be referenced by url. In such processes a reference to a new page is generated when a new task or project is created, allowing the user to create the corresponding page, at the same time as and already linked to the task or project.

New Scrum Process for v3

With the release of xProcess v3 drawing closer here's an overview of some of the improvements that are in the pipeline enabled by new features in the product. In this article we look at the steps to create Scrum projects and the backlog, then to prioritize them and allocate them to Sprints. In subsequent articles we'll consider how you can use the same features used in the Scrum process to tailor development processes for your organisation.

Here's the dialog for a new Scrum project. It shows the parameters used to set up the project including the location where Wiki pages may be created (including a home page for the project) and the Sprint duration which is used when Sprints (the timeboxes of the Scrum process) are created.

As the next screenshot shows there are now more patterns to choose from when you hit the "New" button in the project toolbar. We'll be looking at exactly what all these patterns are for in subsequent articles. For now let's just look at the "backlog items". The patterns for these are User Story, Technical Task, Defect Fix and Delivery. In fact you can see just this menu of items if you right-click on the "Backlog..." task (or a Sprint if you've created one) and select "New" there.

Defects kill Sprint Planning

Burning down user stories sprint by sprint at a nice constant velocity is what we hope to be seeing on agile projects. What happens though if there are defects in the work we've delivered and we have to start planning the defect fixing as well as the new work? How should this situation be approached?

Firstly, don't deliver defects! Easy to say but we know that all teams deliver defects at some points. If you start getting a lot of defects back though it's a pretty sure sign that you have a false velocity - you're reporting work as done when it isn't really. It simply hasn't been tested enough. So the second point is that you have to face reality about the velocity of your team. Get the testing done and automated before you deliver the next set of stories.

All well and good but what about planning the next few sprints? How do we plan reliably now when we are not sure how much defect fixing we're going to have to do along side the new user stories. Here's one approach that seems to work... 2 velocities!

The overall velocity of the team is how many "points" the team burns down in a sprint. We want to ensure this is pretty constant, always assuming that the team stays constant and we're not carrying over too much work in progress at sprint boundaries (see here for a previous discussion of that problem). The second important velocity though is "effective velocity" - the amount of new client-required work that is burned down per sprint. This excludes work on defects (which means the higher velocity reported in sprints when you delivered the defects is balanced) and work on refactoring and process improvement (necessary but not what the client is paying for). You can see the effective velocity sprint by sprint by creating a new folder each sprint to contain just the new work completed/targeted).

Giving the team visibility of these metrics is important so they can see the impact of defects and also appreciate improvements when the effective velocity is restired.

Running xProcess with a console

It seems I missed out a vital piece of information in the previous blog - how do you create a shortcut that runs xProcess with a console. Actually it's pretty simple: add "- console" to the target command. So on a Windows machine the Target field of the shortcut should look something like this...

"C:\Program Files\Ivis\xProcess 2\xProcess\xprocess.exe" -console

Reporting and exporting from xProcess 2.9

A project I've been working with recently has been needing to get reports and data out of xProcess to integrate with other management reports. This hasn't proved as easy as I would have hoped with the product's standard reports, custom reports through BIRT, exporting and other options failing to give us what we needed. I decided to do some more experimenting with "UI Actions", which use OGNL to navigate over the data. Turns out this is a pretty good option with writing the required scripts being quite straightforward. The hardest thing turned out to be actually getting the export file which was a bit more of a hassle. I'm told all this will be great in v3 but for now here's my solution.

First you need to create a new Action in your process, and set it applicable to "Task" (or whatever type of object you want to report on) and "UI Action to "true". The only other crucial bit in this stage is writing the OGNL Expression. My script is below so you can always paste that in, see what it does and then change it!

We wanted to extract a set of key attributes of tasks (like dates and size) and display them in a spreadsheet.The script produces a set of comma-separated values (csv format). Unfortunately v2.9 doesn't give you an easy way to save this in a file. Instead the script outputs the values to the console (so you have to start xProcess with a console window). Once the output has been produced you then have to cut and paste the data into a csv file -- yes that's the "clonky" bit. Still it doesn't take a moment and you then have the data in a spreadsheet where you can mess with it to your heart's content. Brill!

All I want now is v3 so I can get this sort of report through a nice clean user interface. Here's the script...

"-- Output to console action - summarises task information --",

#categoryName ='Importance',
#category = #this.getProcess().getCategoryTypeByName(#categoryName),

#output='\n\nTabular output for: ' + getName() +'\n'+
'Name,'+
'Parent Task,'+
'Start,'+
'End,'+
'Target End,'+
#categoryName + (#category==null ? ' (WARNING - no such category)' : '') +','+
'Size (points),'+
'Estimate (person-hours),'+
'Actual/Planned (person-hours),'+
'Actual to date (person-hours),'+
'Closed?,'+
'Assigned to' +
'\n',

"-- First define the different functions for Leaf and Parent tasks --",

#leafOutput = :[
#output = #output +
name +','+
parent.name +','+
start +','+
end50 +','+
(targetEnd==null ? '' : targetEnd) +','+
(#cat=getCategoryInType(#category), #cat ==null ? '' : #cat.label) +','+
size + ','+
estimateOfEffort/60.0 +','+
bookedTime/60.0 +','+
bookedTimeToDateIncludingConfirmedTimeToday/60.0 +','+
closed,
getCurrentManualAssignments().{#output = #output + ',' + person.label},
#output = #output + '\n'
],

#parentOutput = :[
#output = #output +
'REST OF ' + name +','+
(parent==null ? '' : parent.name) +','+
start +','+
end50 +','+
(targetEnd==null ? '' : targetEnd) +','+
(#cat=getCategoryInType(#deliveryCategory), #cat ==null ? '' : #cat.label) +','+
(topDown ? (#result = size - sizeRolledUpFromChildren, #result>0 ? #result : 0) : '') + ','+
(topDown ? (#effort = estimateOfEffort - estimateOfEffortRolledUpFromChildren, #effort>0 ? #effort/60.0 : 0) : '') + ','+
bookedTime/60.0 +','+
bookedTimeToDateIncludingConfirmedTimeToday/60.0 +','+
closed,
getCurrentManualAssignments().{#output = #output + ',' + person.label},
#output = #output + '\n'
],

"-- Then process all the tasks, including the one selected --",

#tasks = getAllTasks(),
#tasks.add(#this),
#tasks.{
#this.isDesignatedAsParent() ? #parentOutput(#this): #leafOutput(#this)
},
#$sysout.println(#output),

"-- Inform the user where the output is",

#$dialog.informUser('Output to console: ' + getName(), 'The result of this action has been sent to the console.\nCut and paste the output into a Spreadsheet and use Data->Text to Columns... to create the table.\n\n'+
'Note: If you do not have a visible console, close xProcess and restart it from a command line window, or run xProcess with a console by adding "- console" to the target command. (E.g. on a Windows machine the Target field of the shortcut should look something like:\n\n"C:\\Program Files\\Ivis\\xProcess 2\\xProcess\\xprocess.exe" -console')

Note: one of the columns this scripts outputs is a category value ("Importance"). You could use the same pattern to display other relevant categories - "Target Delivery" for example if this was a category defined in your data.

How long should a Sprint be?

Generally I'd I'd say between 2 and 4 weeks. How long can the business leave the team alone to get on with what they've planned (the "no interruptions"rule) is one constraint. The other is how frequently do we need to update the release plan - we only really get feedback on velocity at sprint boundaries.

I've been working with a team recently where establishing a constant velocity has been very difficult. They are basically dealing with a large backlog of defects from previous iterations where they weren't using an agile process. Until this backlog is cleared it's largely academic how long the sprint is - they still get interrupted with a significant number of urgent defect fix tasks every week. Perhaps going to a weekly sprint, at least until this backlog is effectively cleared would help. That way the defects being worked on in that week would be known, with analysis only being done on new defect reports during the week. The sprint planning for the following week can then re-prioritize the work for the following week. This is an example of where the sprint needs to be short in order to handle rapidly changing priorities. However it also demonstrates the difficulty of trying to apply an agile process when a clear quality baseline has not yet been established.

In other circumstances I favour a longer sprint rather than a short one - at least 3 weeks. It gives the team a chance to establish it rhythm of delivering backlog items without the overheads of sprint planning and retrospectives interrupting.

Requirements, requirements, requirements

Is it possible to implement an agile project without overhauling your requirements process? I think not. The key economic benefits flow from agile methods ability to reduce "work in progress" compared to traditional methods [Anderson]. This means specifiers must take a much lighter approach to distant requirements and have much greater involvement, for example as "on-site customers" or product owners, in requirements currently being implemented.

[Anderson] Agile Management, Prentice-Hall (2003)

Availability - how much and who to?

When you're specifying your availability in xProcess there are 2 parts to the information required: when are you available? and to which projects? If you are wanting to change availability for people in your project from the assumed defaults (usually 8 hours each weekday), then here are a few hints to get you started.

The first part - your overall availability - can be specified for both organisations and for individuals Open an editor for either a person or organisation and you'll find (at the bottom of the editor pane) a tab labelled "Availability". This shows the availability records that have been defined. Each record has a from and to date ("..." means "forever" in this context, either in the past for the "from" date, or in the future for the "to" date). For each day of the week you can either specify a specific number of hours or inherit the hours specified on another person or organisation.

By default organisations at the top level (that is organisations with no parent organisation) are created with availability of 8 hours on Monday to Friday. Sub-organisations or persons within an organisation are created to inherit availability from their parent organisation. So for example public holidays can be defined by adding availability records at the top level and personal holidays defined on the individual person in the same way.

The second part of defining availability is to divide your time between one or more projects. When the Project Manager adds a resource to his team he must specify the percentage availability (default 100%) and the from and to dates for this person (defaults to the start and end of the project). To change the percentage of time available to any one project, hit the "Resources" button in the Project Manager perspective "Tools" panel, or open the project editor and go to the "Available Resources" tab. You can then select the relevant person and use the "Manage Project Availability for..." button. Again availability records can be added for a period specifying a percentage of the overall time the person is available on each day of the week. (The default in this case is 100% each day.)

Process patterns: what are "composites"?

The key to defining the task structure in a process definition is the task pattern. Task patterns consist of prototype tasks, optionally contained in a prototype project. Here's an example:

This is the task pattern for a "Feature" in the FDD process and it results in the creation of a set of subtasks and artifacts when the Feature pattern is "instantiated". This is quite a simple pattern and it doesn't include any composite tasks, which are structures to allow multiple tasks, optional tasks and iterations of tasks (repeated tasks that follow on one after the other). It is composites that allow arbitary process diagrams, drawn say using UML's activity diagram format, to be translated into xProcess processes. Here's an example of a task pattern, this time shown in the Hierarchy Diagram view, which does contain a composite. It's the "Feature Set" pattern from the same process.

The composite task in this example is called "Features: $Name$" (this is a pattern so the $Name$ will be substituted by the user supplied name when the pattern is instantiated). It contains a choice of 3 patterns: "Defect", "Feature" and "Task". Any of these patterns may be instantiated multiple times within the composite task.

Composite tasks are parent tasks, in other words they do (or will) contain child tasks, and they also define constraints on what type of child tasks can be created through the patterns they contain. In the latest version of xProcess composites can be instances (only one pattern can be instantiated in the composite), collections (an arbitrary number of patterns can be instantiated in them) or iterations (similar to collections but each pattern will be constrained to start after previously instantiated tasks have completed). Composites contain one or more patterns which define the options available when creating new tasks in the composite. In the example above, Defects, Features or Tasks can all be instantiated within the composite.

So the composite mechanism gives the Process Engineer the ability to define task patterns with loops, sequences and optional paths. More or less any pattern that can be represented in a flow chart or activity diagram can be created in an xProcess pattern by using composites. Composites go further though. Because of xProcess' support for top-down estimation, composites have their own estimates and resource requirements so that they can be scheduled even before the detailed decisions about how many tasks, iterations or options have been made.

To understand how these top-down estimates work in practice, see this blog article: Latest Patterns for Scrum which shows how the top-down estimate for a Sprint is used for scheduling resources, up to the point that all the backlog items have been created within it.

Footnote Question (for the techies among you!):
Are all parent tasks also composites as well as vice versa?
Well firstly all composites are parent tasks. Even if they as yet contain no child tasks they are classified as parent tasks because they allow task patterns to be instantiated in them. On the other hand parent tasks are only composites if they contain a set of patterns. This set defines the patterns that can be instantiated in the composite /parent task. So a parent task without such a set of patterns is not a composite and (in version 2.x) any non-hidden task pattern maybe instantiated in it.
[Note in version 3 this behaviour is likely to be changed, effectively removing any distinction between parents and composites. If the parent task does not contain any patterns, no patterns will be able to be instantiated in it. This will allow Process Engineers to create project patterns where, for example, certain tasks have a fixed set of children which cannot be added to.]

Not just project management - not just process improvement

Explaining xProcess's position in the market of agile management products is something I never find straightforward. On the one hand it's important to emphasize the way you can simply define a set of tasks a team will be working on and immediately generate a plan (which you can then monitor in real time as tasks are re-prioritised, started and completed). On the other hand there's all that cool stuff you can do defining the task patterns, artifact templates, gateways and workflows that make up the process definitions behind different project approaches. I guess it all depends who you're talking to. If they're project managers needing something to get them started quickly, emphasize the simplest possible agile approach (prioritizing a list of tasks, defining the resources available and generating the plan). If they're process gurus - it has to be said these guys are thinner on the ground! - try impressing them the process modelling facilities xProcess provides for you. Producing the graphical task patterns and even workflows is a great starting point.

If you're going for the "simplest possible" approach, try these steps:

• Create a new Data Source (or open an existing one) and switch to Project Manager perspective
• Create a new Project (hit the "New" button in the "Tools" panel)
• Define your tasks (hit "New", select "Task" and create multiple tasks in a comma-separated list)
• Define your resources (hit the "Resources" button, then "Add/New Project Resource" and select yourself as the project resource
• Hit the "Tasks" button to see forecast dates for your new tasks
• View Gantt and Burndown charts for your plan
• Add further resources if available and see the forecast dates and charts change
• Hit the "Priorities" button and make one of the later scheduled tasks highest priority (drag-and-drop or type "0" in its priority field and save the change using Ctrl-S)
• View the change to the task order in the Gantt chart.
  

SpringSource Application Platform released

An interesting announcement this week that you may have missed is the release of the SpringSource Application Platform. Effectively this is a lightweight but high-performance platform providing most of the functionality of existing application servers in a much simpler, small-footprint package. SpringSource CEO Rod Johnson has been quoted as saying that it will provide a "leaner and more powerful solution." He continued. "The SpringSource Application Platform, comprised of the dm-Kernel we developed, along with the proven Spring, Eclipse Equinox and Apache Tomcat technologies inside, provides the industry with the leanest, most flexible server for running enterprise Java applications". Paul Kuzan, a leading software architect and part of the team that put together xProcess, has been contributing to the open source code in the SpringSource platform. "It's a really exciting development in the app server world", said Paul, "and one that is likely to make a very big impact in the years ahead. Watch this space!"

UK Earned Value Conference announced

For another opportunity to hear my thoughts on using Earned Value analysis within agile develop- ment processes, check out this conference in London in June. My presentation is on the Wednesday (11th June) entitled Applying Earned Value in Agile Projects. I'll be reporting on the work to integrate EV analysis into the xProcess project-process improvement environment, highlighting essential modifications that make it possible to apply EV within agile projects.

Sample project revisited

The last time we looked at progress on the blog's sample xProcess project we were looking at the way that project velocity measured simply as "points per sprint" can vary significantly (Calculating team velocity, 24.02.08). If we can gather accurate data about team availability and overhead activity, using a simple measure of productivity produces much better forecasts. Here's the burndown chart as last seen on 24th February.


This sprint completed the target 69 points although as can be seen by the slightly changed positions of the steps on the burndown, some individual tasks completed on different days to the forecast.

The following sprint also completed 77 points, again showing the problem of "yesterday's weather" forecasting, albeit within a reasonably acceptable 10% margin of error. However the productivity measure is much more stable, only varying from 0.91 to 0.90.

Here's the state of the current sprint. In this case we're forecasting 64 points to be completed.


To follow through a series of articles on Scrum in xProcess start with this article on "Using the pre-defined Scrum process".

Converting Eclipse RCP applications to Web apps

We're exploring a new technology called the Rich AJAX Platform (RAP) at present which converts Eclipse Rich Client Platform (RCP) applications into AJAX-enabled Web applications. It looks very interesting. For those of you interested to find out more here's a link to a useful webinar...

http://live.eclipse.org/node/450

Since xProcess is an Eclipse RCP application you might gues the reason for my interest. The main advantage of this technology is the ability to share the same Java code base for the RCP and web applications.

Whitepapers available for download

A number of whitepapers relating to process modelling and agile development are now available for download via the xProcess Europe web site. These include:

• EVM and Agile Processes – an investigation of applicability and benefits
  
• Priority-Driven Processes
• Planning by Priority
• Managing Processes with Agility and Transparency
• Building Agility into a CMMI-oriented culture - the BT experience

The last of these is a report into the BT experience of transforming a traditional CMMI-oriented culture with the adoption of agile techniques. This paper was published in 2005 and since that time more experience papers relating to the cross-over between software engineering focused organisations and the agile community have appeared. It remains though a gulf that is only rarely bridged by companies and communities - generally people prefer to stay one side or the other. I heard recently of a major bank that had implemented a major new project management methodology, standardising the approach for all its IT projects with the exception of - wait for it - agile projects! Surely it behoves us to understand the strengths and weaknesses of each other's approaches rather than merely excluding whole classes of projects. On the other hand I expect major gains to be made in the next few years by organisations who look to embrace both the lessons of agile methods and the best software engineering practices that have evolved over the lifetime of the discipline's existence. Thesis, antithesis, synthesis: it's a good pattern for progress.

Modelling processes to scale

Tasks in process models come in varying sizes. There is a key distinction that divides tasks in process models into two classes: is this task at the "plannable" or "sub-plannable" level. Plannable tasks appear on plans, Gantt charts, burndowns and crucially time-sheets. They are the kind of tasks that take from several hours to days or even weeks. Plannable tasks are the trackable planned, estimated and costed elements in xProcess. However in some processes we wish to define (and monitor) tasks at the sub-plannable level. For example if my tasks is to distil a certain compound from ingredients (in a chemical process), the steps in the process may need to separately defined and tracked to ensure quality compliance. xProcess provides a mechanism to support this in the "Gateway Type". While these steps form only a simple sequence in v2, we've recently been experimenting with a richer model for gateways that will allow the same kind of forms - selections, collections, iterations and sequences - that are supported in plannable tasks through the composite structures. Together these facilities make a continuum of process modelling from from large to small scale.

Better Faster Audit

I've recently been asked to outline the structure for our process audit service offering at xProcess Europe. The product is called the "Better Software Faster Process Audit" and is based on the structure of Dan Haywood's and my book of the same name(Better Software Faster). It covers 7 crucial aspects of any software development process:

• The last step - Build and Deploy
  
• The first step - Modelling and design
• The stakeholder step - Requirements
• The controlling step - Management
  
• The continuous step - Quality measurement
• The micro step - Development
• The macro step - Architecture

Starting with the last step usually throws people, but given one of the key criteria for improvement is agility, it makes sense to start with seeing how close a team is to fully automating their build and test suite. There's no excuse for not fixing this area if you're using languages like Java or C# but with some older languages it can be tough to provide continuous integration for every change made by a developer.

Quality Measurement is another related area which is sometimes overlooked. Every team has some means to assess quality. For some teams "release the product and monitor the bug reports" seems to be their preferred way of working. Not recommended!

EVM and Agile Processes

I am presenting a paper on EVM and xProcess next week at the 2nd EVM Conference in Birmingham, UK. I’d love to see you at the conference but if you’re not able to make it and would like a copy of the paper you can download it now from the xProcess Europe or Ivis web sites. Here's the abstract...

EVM and Agile Processes – an investigation of applicability and benefits

Earned Value Management (EVM) is a technique for understanding the degree to which a project is following a plan. A prerequisite for EVM as traditionally applied is therefore a stable baseline plan with a fixed scope. Agile processes for software development by contrast are designed to be responsive to a changing environment and enable discovery of requirements through their lifecycle. Agile plans are rarely stable or of fixed scope. However both approaches share the goal of producing plans that provide stakeholders with immediate feedback on the daily progress of projects and they share common needs in understanding and reporting progress and productivity. In addition the fact that agile processes map requirements to plannable tasks means that requirements may change without invalidating the baseline for progress reporting. This paper reports on work to develop tool support based on the xProcess process and planning product to apply EVM to agile processes. It shows how the metrics for schedule and cost efficiency can be modified for compatibility with agile approaches, and how they help effective management of agile projects, particularly when supplemented with tool support for project forecasting and monitoring.

Calculating team velocity

Why is it that agile methods use timeboxes containing broadly similar sets of activities - as opposed to phases favoured by more traditional methods, which contain different types of activity in each phase through the lifecycle? A key reason is that stakeholders in the project get quicker and more effective feedback on the progress and direction of the project. As a consequence planners can judge more accurately the effectiveness of the project and what it is likely to achieve over its full duration. One of the most useful metrics planners get from each timebox is the velocity of the team. Scrum projects usually use the velocity from previous sprints to forecast their likely progress - a technique often referred to as "yesterday's weather" since it effectively assumes the next period will be broadly the same as the previous. In this article we discuss what velocity is, how to calculate it in xProcess, and how to go beyond "yesterday's weather" by calculating and using simple productivity for improved forecasting.

Velocity is defined as the amount of required work delivered per time period. In the Scrum process for example, required work is called the Backlog, and its component parts, the Backlog Items, are individually sized and prioritised. The size of an item is not the same as the effort required to complete it, though size and effort are related and may - once the velocity of the team is known - be derived one from the other (see here for further discussion of size and effort). The size of backlog items is usually estimated in a team-specific measure called points (sometimes called Story Points, Feature Points or, if related to the amount of work a team member could carry out in a full uninterrupted day, Ideal Days). The size estimates therefore really only give a relative size for the backlog items. However once a timebox has been completed - in Scrum timeboxes are called Sprints - the team has a measure of how many points they were able to complete and so have the first reading for velocity in Points per Sprint.

When using xProcess for Scrum you can see the velocity of completed Sprints from the burndown chart. For example in the Sprint shown below you can see that a total of 91 points were completed, giving the team a velocity of 91 points per sprint.

Using the principle of yesterday's weather it might be natural for the team to commit to a similarly sized set of backlog items in the next sprint. Before doing so they might be wise to look at all the history on the project. Here are the burndown charts for the previous two months, first Sprint 02 and then Sprint 01...
As we can see from the respective velocities of 40 points per sprint and 73 points per sprint, velocity is not always constant and, if we look at the reasons behind this, we may be able to forecast the velocity for the next sprint more effectively. Why did Sprint 02 for example do so badly?

Well one clue is in the dates for this Sprint. This won't be the only team to have discovered their productivity was lower between mid-December and mid-January! All the team were on holiday for at least some of the Sprint, and when they were in they probably discovered someone they needed to speak to was away, or that there were other activities they had to attend to which were not related to the sprint backlog. Another factor was that this sprint had a number of items that had been worked on but not completed at the end of the sprint. Since they are not finished they cannot be counted in the size of work "done". Backlog items like this do not necessarily get done in the following sprint since other priorities may be introduced to the team at that point. However if they are carried forward (as they were in this case), there will be considerably less work left to complete in the following sprint. This at least partially explains why Sprint 03 was such a productive one.

So what velocity should the team predict for Sprint 04? This is where other information from xProcess comes in very handy in the planning process. xProcess allows team members to record when they were working on overhead tasks (like management meetings, email and admin, training courses and office parties) as opposed to backlog items themselves. xProcess also give us information about the amount of time spent on backlog items that were not completed in the sprint.

The key metrics we want to derive in order to better predict the next sprint are these:

• simple productivity (size of work done in points per ideal person-day)
• overhead percentage (what proportion of people's time is applied to non-backlog-item tasks)
• availability (when are people available to the project)

We'll leave discussion of overhead tasks and availability to another occasion, though they can be set simply in xProcess. The simple productivity measure of Size/Effort can be calculated automatically by xProcess. I say simple productivity measure since, as Putnam and Myers point out, it is more complex to derive a true process productivity for a project, taking into account that size and effort do not follow a linear relationship when the duration under consideration varies. This non-linearity can be broadly discounted however if the duration of the sprint does not vary. (One of the reasons why agile methods recommend fixed durations for timeboxes is to ensure that feedback from one timebox is applicable to the next.)

Simple productivity for a given period (a sprint for example or the whole project) takes into account all the non-overhead tasks booked to during the period. It also takes into account tasks which were incomplete at the start and end of the period. Once the simple productivity factor has been calculated for previous sprint (or for the project so far), the value can be use to set new effort estimates based on size using the Set effort to match size UI Action described in a previous article.

The simple productivity currently being used on this project is 1.0 - in other words a backlog item estimated as size 1 will be scheduled with an estimated required effort of 1.0 ideal person-days. Using this factor the current state of the burndown chart for Sprint 04 is as shown below.


However using the history from the previous 3 sprints, xProcess can calculate the actual productivity in points per ideal person-day. In this case it was found to be 0.91 - in other words historically a task of size 1 has taken on average about 8.8 person-hours to complete. Using the Set effort to match size action, we can quickly adjust the estimates of all the open tasks in the project to reflect this factor. Once all this is applied (see burndown chart below), the state of Sprint 04 now reflects the true situation that certain tasks are at higher risk of not completing.

Developing your own workflow monitor

Just occasionally I'm tempted to post a really techie article that will probably only be of interest to hard-core xProcess enthusiasts! This is one of those occasions. I've written elsewhere about some of the aspects of workflow support in xProcess (see for example the article on State Diagram support which explains how to define triggers for internal events graphically, and "From tracking to planning and back" which discusses how tracker products can be linked to xProcess via workflow). This article though will go a bit further and give you some hints about how the Workflow Server works and how you can link in your own programs to detect workflow events and trigger actions on them.

The Workflow Server sits in a Tomcat instance watching the outside world (and the inside world of the xProcess Data Source) through a set of monitors. When these monitors discover events that might be of interest, they tell the Workflow Server which checks whether any object in the xProcess Data Source has registered an interest in that type of event. If it has it invokes the defined action(s) for that event, potentially committing changes to the xProcess data or indeed to external systems. Several monitors are delivered with the xProcess Workflow Server. There's the DataSourceMonitor itself which monitors state transitions within the Data Source.There are also monitors for the Flyspray bug tracking system, for CollabNet's Project Tracker and a general purpose email monitor. However if you want to integrate external systems with xProcess this is exactly the way to do it - write your own monitor. In spite of my introduction to this blog article, this is quite straightforward to do. There are 3 key elements:

1. A monitor class (inheriting from com.ivis.xprocess.workflowserver.monitors.MonitorImpl)
2. Methods for this class which initialize it, find the events
   
3. [optionally] An event class (inheriting from com.ivis.xprocess.workflowserver.ExternalEvent) and possibly useful methods that may be called when the event is triggered.

Here (below) is a code snippet for a sample monitor which show the key methods init(), which is called when the Workflow Server starts up, and findEvents(), which is called periodically by the server in order for this monitor to query its external system and provide the events to the server.

public class SampleSystemMonitor extends MonitorImpl {
SortedSet events;
public static String EXTERNAL_SYSTEM_NAME = "My External System";
public static String EXTERNAL_EVENT_NAME = "My event name";

public String getSystemName() {
return EXTERNAL_SYSTEM_NAME;
}

public String[] getEventNames() {
return new String[] { EXTERNAL_EVENT_NAME };
}

@Override
public void init(DataSource dataSource) throws WorkflowException {
super.init(dataSource);
// TODO initialize connection with external system
}

@Override
protected SortedSet findEvents() throws WorkflowException {
events = new TreeSet();
// TODO query external system for new events, e.g.
// loop
ExternalEvent xevent = new SampleSystemEvent(new Date(), EXTERNAL_EVENT_NAME);
Map map = xevent.getMap();
//TODO set map values so they can be referenced in the triggered action, e.g. ...
map.put("name", "New Task");
events.add(xevent);
// end loop
return events;
}
}

The monitors are singleton objects and are instantiated within the server by Spring. So having written your monitor you can inform the Workflow Server that it should be called simply by modifying the xprocess.xml in the Tomcat WEB-INF directory (following the pattern of other monitors). Specific objects in the xProcess Data Source can respond to the events and trigger specific actions through their "WorkflowPackage" - which is declared in the process definition.

Set effort to match size

A previous entry provided the script required to define an action shortcut for setting the Size of tasks based on existing estimates for the effort required (see Set size to match effort). In some cases the inverse of this operation - Set effort to match size - is the useful operation. Here's the code you need for that one. Once again you should make it a "UI Action" by setting this field to True so that it appears on the right-click menu when you select a task or set of tasks.

Action Name: Set effort to match size
Expression:
#factor = getProject().getFloatProperty( "CurrentProductivity_pointsPerIdealPersonDay"),
#factor = (#factor < 0.01) ? 1.0 : #factor,
#size = getSize(),
setBest(#size/#factor*240.0),
setMostLikely(#size/#factor*480.0),
setWorst(#size/#factor*720.0)
Applicable to: Task
UI Action: True
Parameters: (none)

Note that the operation uses a factor to scale the mapping between size points and effort. If this factor is 1.0 the mapping is 8 hours effort (480 minutes) per size point.

Perhaps - like it's inverse operation, this will one day appear in the preconfigured operations of the Simple Process. Note that is very similar to the Set estimates action which is delivered in this process. However that operation is designed to take Size as a parameter (usually during the instantiation of a pattern) and so it is not suitable to make that one into a UI Action for operating directly on existing tasks.

Traceability through a simple metamodel

When people start defining processes with total traceability between every aspect, the sight is often not very pretty! When traceability adds multiple layers of complexity it rarely pays for itself in terms of improved quality or maintainability.

This is one of the reason why we defined what we termed the "minimum metamodel" in the Better Software Faster book on agile development. The diagram shown here is this model in its simplest form. The questions traceability should answer are:

• How is this requirement tested?
• How is this requirement implemented?
• How is this part of the design tested?
• Why is this part of the design needed?

It's a useful starting point when defining the artifacts required in a software development process, and the references they should contain to enable these questions to be answered.

The Role of the Process Engineer

Since xProcess provides configurability of processes, it also introduces a new role to projects - that of the Process Engineer. Of course the role may be played by a project manager or process consultant, or simply by one of the members of the project, but it's an important role and one that can greatly enhance the success of projects using xProcess.

Several Process Engineers have already created the processes that come bundled with xProcess (Basic Scrum, Basic FDD, Simple Process) and there are others such WPM Prince 2, Web Development, Issue Management and Basic UP that are available in the community. So the likelihood is that most Process Engineers will start with one of these processes as the baseline. As experience with a process is gained, the details of the way it is being used by the projects can be understood and the sorts of useful modifications or additions to the patterns, actions and workflows can be specified.

Using parameters in task patterns

Processes are defined in xProcess through Patterns - patterns for sets of tasks, for projects, for priority groupings (e.g. Folders) and patterns for artifacts. By constructing sets of patterns, including patterns within patterns (Composite Tasks), a complete definition of the process behind a family of projects can be built up. A key part of pattern definition is the specification the pattern's parameters. This is what this article looks at.
The diagram above shows the editor for a typical pattern from the Simple Process, showing the 4 parameters that this pattern has: Name, Project Home Page, Start Date, Duration. These are the values that the user can supply when creating an instance of the pattern. Parameters always have default values which are used if the user does not change them, but you can add an action to a parameter - a DefaultValueGetter - which will set the default value at run time based on teh context or time at which the pattern is being instantiated. You can see in this case that Start Date has such an action attached which sets the start date to the current date. Parameters also have a type which can be one of: Integer, Floating Point, Boolean, String or Rich Text.

Having set up the parameters of your pattern you can then use these values to set values within the elements that are created by your pattern. Typically these will be projects, tasks and artifacts any or all which can be modified from the state of the prototype elements in the pattern to the real elements made when a user uses the pattern.

Th first way to use a parameter is by string substitution. Any text field in your prototypes, for example the prototype project's name, can include a parameter substitution. Simply enclose the parameter name with 'Simple Project pattern, the name of the prototype project is $Name$, showing that when the pattern is used the name of the project comes from this parameter. It's also used in the case of the Project Home Page parameter, which sets the description field to point to this url.

You can even use string substitution with artifacts, provided that the artifact uses a text-based, rather than binary format (e.g. rtf, xml, csv formats). In this case just put the substitution string, say $Name$ inside the file attached to your pattern. You'll find when you use the pattern that the corresponding artifact has this string substitution applied. Several patterns in the Basic FDD signs in the field, and when the pattern is instantiated the parameter value will appear at this point instead. You can see an example of this type of parameter use in nearly all the pre-defined patterns delivered with xProcess. In our example here, the Simple Project pattern, the name of the prototype project is $Name$, showing that when the pattern is used the name of the project comes from this parameter. It's also used in the case of the Project Home Page parameter, which sets the description field to point to this url.

You can even use string substitution with artifacts, provided that the artifact uses a text-based, rather than binary format (e.g. rtf, xml, csv formats). In this case just put the substitution string, say $Name$, inside the file attached to your pattern. You'll find when you use the pattern that the corresponding artifact has this string substitution applied. Several patterns in the Basic FDD process have artifacts that use this feature.

The second important way that parameters are used is in actions. As well as DefaultValueGetters, patterns have another very useful type of action - InstantiationActions. These actions are called when a pattern is used. When you define InstantiationActions you specify which of the prototype tasks you want to be transformed by the the action, and which particular action you wish to be called. Here's the InstantiationAction used in the Simple Project pattern.It uses the Set project dates action which sets a number of dates (targets and scheduler dates) associated with the project. It is this code which ensures projects you create start on the date provided by the Start Date parameter. One thing to note about parameters appearing in OGNL code is that spaces in parameter names are replaced by underscores and non-alphanumeric codes are ignored. Also following OGNL conventions the parameter is preceded by a hash (#). Note that the Set project dates action uses both the Start Date and Duration parameters.

When you're starting to use parameters in your patterns the best approach is to look at patterns delivered with the pre-configured processes in xProcess and to re-use actions that these patterns use. The Simple Process for example provides the following actions that you can re-use in your own patterns:

• Next timebox date (Project): finds the latest finishing folder/timebox and adds a day to its end date
• Set required role type (Task): finds a RoleType matching the supplied string and sets this as the required role type for the task
• Set target dates (Task): sets targets
• Get today with offset: returns a date relative to the current date
• Set max participants (Task): sets the number of people for a task
• Set project dates (Project): sets target and schedule dates
• Assign to account name (Task): searches for a matching user and assigns to the task
• Get earliest start or end date (Task): returns either today or the task's project's scheduleStart if this is later
• Set target dates and weight (Folder): sets the folder's targets and priority weighting based on that of preceding folders/timeboxes
• Next timebox name (Project): finds the name of the latest finishing folder/timebox with the appropriate name and returns a name with its number incremented
• Set estimates (Task): sets three-point estimates based on Size
• Set importance (Task): categorizes task into High, Medium or Low categories.
  

Note: there are also some UI Actions in the set of actions supplied by the Simple Process. See for example this article: Set size to match effort.

Artifact templates

When you're defining the process for your project don't forget that a major part of this is defining the standards for the files, documents and records (artifacts) that are created, modified and qualified as a result of the process. xProcess provides excellent support for defining the types of artifacts that will be used in a process. Furthermore it stores both the document templates and the resulting documents within its version control system so you can not only reliably access the most up to date version, but also compare it with previous versions if necessary. Here's a task pattern showing artifact templates that have been added to each of the subtasks in the pattern.
You can do this simply by right-clicking on the task and adding a "reference" to a new artifact - an attached file, form or hyperlink. (To learn more about the different types of artifacts supported in xProcess see Creating Artifacts in the Help system). Once you've done this, every time you make an instance of this pattern a clone of your artifact template will be created for use with that instance. Another nice feature is that you can substitute text in the file when you instantiate it, using the values that the user enters into the parameters of the pattern.

Actually parameters of patterns are discussed in the next article in this series, so see Using parameters in task patterns for more details.

Adding gateways to task patterns

An important part of defining the process for your project is defining how to know when a task is complete. Is a development task complete before unit testing for example? Almost certainly not. But if acceptance testing for an enhancement is a separate task in your process, at what stage does the developer hand it over for acceptance testing?

One way to define this in xProcess is to define a gateway for the task so that the user is prompted by a checklist to ensure he has completed the required work.
In the task pattern shown above you can see that a gateway called "Feature complete" has been added to the "Develop" task. To do this in one of your patterns simply switch from the Project Manager perspective to the Process Engineer perspective, you can thenopen the pattern diagram for your pattern (right-click on the pattern). You can then select "Gateway Type" from the palette and drop it on the task in the pattern that you want to have a gateway.

The content of the gateway is defined by creating or editing a Gateway Type in your process. Right-clicking on your process and selecting New -> Type of Gateway opens an editor like the one shown here on the right. This is where you can enter in the questions / checklist items that are presented to the user when he closes one of these tasks. In this case the questions are:

• Tests complete and pass?
• Documentation complete?
• User approved?

In each case we must also specify the set of possible answers and whether the answer represents a "pass" or "fail" condition.

Let's now see how this shows up when a participant on the project closes one of these tasks using the web client. Here's the gateway screen that a user would complete before closing one of these tasks (click on the image to enlarge).
You can see that for questions where a "fail" response is selected a comment is required in order that a reviewer can understand why this was done. You can learn more about gateways in xProcess from the help files. See for example Answering Gateways.

The next article in this series concerns Artifact Templates.

EV for Agile

Earned Value analysis is really designed for showing how closely you are tracking a pre-defined plan. Schedule efficiency and cost efficiency give separate measures of the degree to which you are "on-time" and on-budget". The problem with agile plans is that the scope of the project is expected to change as it progresses, usually with the aim of keeping schedule and cost more or less constant. Is EV analysis simply not applicable in this case? Actually in the context of a forecasting environment like xProcess, EV can still provide some most valuable metrics to agile plans. I've been invited to present a paper on EV for agile projects in a few months time and so I'd be most interested to hear of others applying EV with plans that must vary over time. Drop me a line if that's you!

If you're interested in receiving a copy of the paper btw click here.