Breaking Down Software Development Roles

As a IT professional, I've often found myself at a loss when I've had to explain my job concisely.

Usually following the informal pleasantries, including hellos, name exchanges and handshakes is the dreaded "So, what do you do?"

Um' I'm an IT professional.

Okay, but what do you do?

I um.. develop software.

Develop software?

Well.. I do requirements analysis, design, architect, develop, document, test and deploy software applications based on customer needs, while working with various vendors and integrating with other applications and co-ordinating activities with offshore team members.

Huh?

I write code.

oh!

By this time, the pleasantries have ceased to be that, and the other person either looks at you and your family pitifully or warily.

The truth is that the software developer of today is expected to perform all these roles. Internet.com, has tried to break down the software development roles as they exist today.

Software development, the paper observes, is done differently at every organization, and recognizes that the process that one organization or person uses to develop software may not work perfectly in all circumstances.

While environments will change and with that the process that are being adhered would be adapted either marginally or dramatically.. however the multiple and multi-faceted practitioners of the fine art of software engineering remain largely the same - for the requirements are the same - There will always be a need to understand the business problem, convert that problem into an architecture, convert the architecture into a solution, test the solution, and deploy the solution. Although each of these processes may change to some extent based on the programming models and tools being used, fundamentally there are some roles, which every process has in one form or another.

One person may be filling all the roles or a handful of the roles, or one very specific role. Despite
this there is a need for all of the roles -- each serves a purpose.

From Subject Matter Experts, Functional Analysts, Solutions Architect, Development Lead,
Developer, Quality Assurance, Deployment (Deploy), Training, Project Manager, to the Development Manager, each role has a set of critical skills required.

Download the paper from here. (free registration required)

The Future of Software Development

Software development has come a full circle states Alex Iskold in a scathing, pull-no-punches broadside upon the Waterfall model.

Read it here: the future of software development

Alex states that arrogance was the main problem with the waterfall model. The arrogance he argues came from the fact that we believed that we could always engineer the perfect system on the first try.

37 years ago, W. W. Royce published a sequential development model, coining what is now known as the waterfall model. This approach applied the insights from mature engineering disciplines (mechanical, civil, etc.) to software. The idea was to construct systems by

1. first gathering requirements,
2. then doing the design,
3. then implementing it,
4. then testing,
5. and finally getting it out the door in one linear sequence.

Ironically, Royce actually argued in this presentation that this was a flawed and actually proposed what is known as iterative design.

37 years - a lot has happened, and we have come a long way and (hopefully) have learned a lot about making software.

However, in the real world, (and in other unreal places that I happen to work in) software projects have ill-defined and constantly evolving requirements, making it impossible to think everything through at once. This effect is compounded by the accelerating pace of business.

Older development methods completely fail to address business needs. The Waterfall Model is rigid and unrealistic and incapable of rapidly adapting to and keeping pace with the changes to the business.

Using the Waterfall Model, these

1. changes were impossible,
2. the development cycle was too long,
3. systems were over engineered and
4. ended up costing a fortune,
5. and often did not work right. ;-)

In nature, dynamic systems are not engineered, they evolve.

• Build small and then add on.
• Model the Happy day scenario. Leave the esoteric corner cases for another day, another release.
• You cannot and must not code for *all* the weird and *potential* use case scenarios
• Build pluggable frameworks
• Design to integrate
• Leave the system open, and flexible

The solution to achieve these niceties, it appears is an age old one. In 1975 - (a few years after the Waterfall term was coined), Fred Brooks famously propounded that "Adding manpower to a late software project makes it later" in his book titled The Mythical Man-Month: Essays on Software Engineering. However we see this mistake repeated everywhere; and in every project that is going sour. "Throw-more-resources-at-it", will probably get you a few more months out of your aging PC - but hardly ever speed up the project.

Alex evangelizes the adoption of agile methodology as proposes that in the future, software will be developed by "Just a Few Good Men" and provides a simple overview of Agile Software Development Principles:

• Have Fun
• Focus on Simplicity
• Adapt the code
• Embrace Change
• Get feedback
• Refactor
• Communicate
• Release Often
• Test

.. and concludes with his forecasts for the future:

• High-quality, passionate software engineers will be in very high demand and will make substantially more money. - Good!!!
• The developers who do not have great programming skills are going to have to look for jobs elsewhere. - hmmm... why are they in this industry in the first place?
• The changes that we are witnessing today in the social software market are going to reach the enterprise level. - SAS? the Web is the computer?
• Software off shoring will make less and less economical sense. - why so? does agile software development going to make the world less flat?
• Computer science is going to remain a highly competitive and prestigious field. - sure hope so.

While I agree with Alex's theme in general, with respect the Agile methodology adoption, I fail to see why off-shoring would get affected by this; Unless this this is was just a general comment.

On this topic, I had a brief chat with a colleague of mine, another proponent of the Agile methodology himself, and he argued

1. it would not be possible to have synergistic agile teams across the seas and across timezones.
2. Agility would be impeded due the overhead of heavy documentation; people would not understand the business properly otherwise.
3. it is not cost effective to have several delivery dates from offshore
   

I smiled in reply - it will evolve.

SAP to acquire Indian BRMS company Yasu tech

SAP announced on Wednesday that it is acquiring privately-held Hyderabad, India based Yasu Technologies, which creates business rules management software.

Yasu's flagship product is the QuickRules BRMS, launched in early 2000. The deal is designed to boost SAP's business process management (BPM) offerings and will be tucked into SAP NetWeaver, which channels the ebb and flow of data to software applications via SAP's back-end middleware.

The India-based company was founded in 1999 and has just over a 100 employees. Started by 6 entrepreneurs, it boasts of some of the best talent in India’s IT industry, with over 70% of the product development team comprised of IIT alumni.

The Yasu announcement comes just a little over a week after SAP announced that it will acquire Business Objects in a deal valued at more than $6.8 billion, its largest acquisition ever.

Help Line - More signs of out-sourcing

I was feeling a bit depressed the other day, so I called the Help Hotline.

I was put through to a 'call center' in Pakistan.

I explained that I was feeling suicidal.

They were very excited at this news and wanted to know if I could drive a truck or fly an airplane....

BPM Software shoot-out

The question on Slashdot was actually "Do You Like Your Workflow or BPM Software?".

The questioner was interested in "firsthand experiences with these kinds of products and in unbiased reviews" and requested information on following products:

• JBoss jBPM & Rules
• ILOG JViews & JRules
• webMethods & Blaze Advisor
• Oracle BPEL Process Manager
• COSA
• FLOWer
• Microsoft BizTalk
• Lotus Workflow
• Global360 BPM
• Bizflow
• WebSphere MQ Workflow
• Tibco BPM
• Fujitsu Interstage
• FileNet BPM
• Savvion
• Pegasystems
• BEA AquaLogic with Fuego

As is to be expected on Slash Dot, there were several comments of "Do you own homework", and some "unbiased" product plugs passed off as "opinions". Largely however, there well thought out answers from experienced industry veterans.

The general consensus seems to be that "it depends". It depends on what you want to do and how much money you want to spend :-)

Error Message: Your Password Must Be at Least 18770 Characters and Cannot Repeat Any of Your Previous 30689 Passwords

"Error Message: Your Password Must Be at Least 18770 Characters and Cannot Repeat Any of Your Previous 30689 Passwords"

No kidding... its a real error message....

The State of BPM: Perspectives of an Industry Insider By Kevin Spurway

"The BPM industry is awash in hype" declares Kevin Spurway, warning that vendor hype has created market confusion about the proper role of IT with respect to BPM.

I could not help but smile sadly and nod my head at this... IT unfortunately perceive BPM as a threat rather than the powerful tool that it is can be.

What's beyond the hype, and how is the industry addressing this dangerous situation, and help clear the smoke? Kevin identifies 3 intersting developments - "Standards", "Free Modelers" & "Communities and Social Network-based Approaches".

BPMN clearly stands out from amongst the various standards that mushroomed; however Kevin warns that "BPMN notation has to date gained little traction outside the relatively small BPM community" and observes that "Business users are highly unlikely to spontaneously adopt BPMN notation the way they spontaneously adopted the spreadsheet almost thirty years ago".

Free modelers are a relatively new concept. this allows the organization to take Process Modeling for a test drive.. without shelling out the $$$$s. Lombardi and Savvion seem to have it.. whereas the others are playing a wait and watch. The developer in me is definitely excited at the prospect of a test spin... however the process modeler in me argues that I already have M$ Visio installed with BPMN stencils- do I need more?

Communities and Social Network-based Approaches - again a new concept; Pega launched Pega Exchange recently to enable "customers and partners to create and exchange PegaRULES Process Commander (PRPC) content, from application frameworks, to plug-ins for common enterprise technologies, to utilities that make development easier", in order to "Leverage the community" and not "reinvent the wheel".

Savvion has their ProcessXChange - however, according to Kevin was a "stillborn".

What I would like to see is an open exchange market place where I could pick up a process modeled in one platform and use it with another seamlessly. Sort of like a sourceforge or a codehaus or component source for process models. For this to happen though, the standards such as XPDL or BPDM (the serialization standards in which you save the process models that you model in BPMN) must be widely adopted strictly adhered to (thanks for the correction Sandy).

J2EE v/s .Net

[Originally posted Friday, November 26, 2004 @ Caffeine Kick]

While I know that several Terabytes of data have been generated in the online and off-line feud over what's the better, I decided I wanted to join in with my own rant as well.

As you probably are aware, I earn my living writing J2EE web-apps. What you are probably not aware is that my better-half is a .Net developer with a top ERP company. So my point-of-view on this matter would be a no-brainer right? Hee hee.. Wrong!

I have several issues with J2EE and Java, at least the way we have it today. I'll talk about one of them here.

There are just too many ways of doing any single thing.

Take persistence frameworks, application servers, MVC frameworks, testing frameworks.. the list is virtually endless. For each and every thing that I want to do, I have at least two (usually more) options to choose from, each better, faster, lighter, easier, cheaper… xyz-er than the other. While I understand that Choice is a good thing.. but what we have today is definitely an overkill.

So much of time and effort expended on ‘deciding’ the ‘appropriate’ ‘stack’ to use… only have that sinking feeling 1-month into development, when an even better, even faster, even lighter, even easier, even cheaper and … even xyz-er, brand new way of working surfaces out of the blue (usually apache / SF or someplace similar)

To EJB or not to EJB is usually the first question, then follows a plethora of nerve wracking choices .. Hibernate or Spring, Struts or JSF, XML or Resource Bundles, Weblogic or Websphere… you get the drift.

.Net on the other hand is much more straight forward. You have one vendor, who gives you all the damn stuff that you need. You concentrate on the work at hand. In .Net we do this like this, and that like that. Over, Simple, Period, Full Stop. No further unpleasant DAR (Decision Analysis and Resolution documents), no further uneasy CAR (Causal Analysis and Resolution meetings).. No more irritating sales calls from SUN, BEA and IBM pushing their servers through, No more newbies in the organization with a conceited grin of mockery claiming "we could have done it better if we had used that 'other' framework".

[Originally posted Friday, November 26, 2004 @ Caffeine Kick]

Seriously, almost 3 years later, what has changed?

Caffeine Kick

Nostalgia time. tried to remember my old blog. Caffeine Kick. it used to be at caffeine.sabarish.com. Sniff sniff.. boo hoo...

However, thanks to the Way Back Machine, I was able to sneak a peek at my old blog.

Apache ODE

Apache ODE, as in "Apache Orchestration Director Engine" (wow thats a mouthful), is a Top Level Project under the aegis of the Apache Software Foundation.

The stated objective of the ODE is "to create a reliable, compact, and embeddable component capable of managing the execution of long-running business processes defined using the BPEL process description language", and the focus has been on "developing small modules with minimal dependencies that could be assembled (and easily reassembled) to construct a full featured BPMS". LEGO blocks of BPMS?

The key components of the ODE architecture include the ODE BPEL Compiler, ODE BPEL Engine Runtime, ODE Data Access Objects (DAOs), ODE Integration Layers (ILs), and user tooling

Based on the contribution from Intalio to the ASF in July 2006, (originally obtained by Intalio's acquisition of FiveSight Technologies) ODE left the incubator and took Top Level Project status on September 12 '07.

Already 4 projects including the Intalio|Server are using the ODE :

• Apache ServiceMix: Agile open-source ESB (JBI Container)
• Coghead: online platform to create your web-based applications.
• Intalio BPMS: a full open source BPMS solution including a BPMN designer, runtime components and tooling.
• SUPER: Integrated EU research project, aiming to raise BPM to the business level, where it belongs, from the IT level where it mostly resides now.

Built upon the ODE foundation,

"Intalio|Server is the fastest and most scalable process engine currently available on the market, capable of supporting hundreds of thousands of different process models deployed on the same server, and hundreds of millions of process instances running concurrently on a single CPU"

Sounds compelling enough to take it out for a spin.. what with Eclipse IDE support! Will keep everyone posted.

---

An Introduction to Apache ODE
Intalio|Server

---

Six Myths of Rules and Business Process Management

One of BPM's early pioneers, Dr. Setrag Khoshafian, Vice President of BPM Technology at Pegasystems (previously Senior Vice President of Technology with Savvion), surely knows a thing or two about BPMS. In this short and interesting white paper he diffuses what he identifies are Six Myths of Rules and Business Process Management.

The Six myths dispelled with insightful reality checks include:

1. We are focusing on BPM (automating processes) not rules [sigh… I hear you doc!]
2. We can handle the rules in Java or C# [my personal favorite]
3. "Best of Breed" is best of choice [but is’nt loose coupling a good thing?]
4. Let the BPMS handle 'simple' rules and the BRE handle 'complex' rules [grin grin]
5. Rules should be modeled separately from BPM modeling [yeah right!]
6. Rule-based BPM is a process engine written in a rule language that 'reduces' everything including processes to a rule [frankly, I didn’t get that myself at first]

Many of the myths seem to stem from the fact that historically, application architectures have defined and handled BREs and BPMS as separate entities.

Get the white paper (available as a PDF download) from pega.com

Iterative vs. waterfall software development

Nowadays, this question seems to figure at every technical interview that that I been involved with (at either side of the table). What would you choose - Waterfall model or the Iterative approach?

The correct answer - it depends.
The usual answer - some mumbling about extreme programming (followed barely disguised rant on how one was forced to use it).

Classically, in development we use either the Waterfall or the Iterative approach. While the Iterative approach is more adept at accommodating requirements change, the Waterfall model treats changing requirements as the exception. However, this does not preclude either model or approach from accommodating change.

In the waterfall mode - There is a long requirements definition and approval phase (the elusive sign off), followed by the subsequent life cycle steps (design / development / testing / deployment), and there is one huge grand ship date (the famed / dreaded go-live date). The requirements are considered sacrosanct post the sign-off and any change is expensive.

In the iterative mode - The development cycle is actually a series of sequential and repeated short release cycles. This lends itself to injecting changes to the application without disastrously affecting the bottom line.

While the iterative approach may seem the obvious choice; a panacea for all our problems, the choice isn’t really that easy. See Bill Walton’s take on iterative versus waterfall in Computer World.

Managing Requirements

FACT #1: The Standish Group’s 1994 Chaos Report found that the top three project impairment factors across 352 companies and 8,000 projects were

1. Lack of user input (12.8% of respondents),
2. Incomplete requirements and specifications (12.3%), and
3. Changing requirements and specifications (11.8%)

FACT #2: Almost 50% of defects identified during testing to be due to defects in requirements. Source: “Calculating your return on investment from more effective requirements management.”


How does one address this? Let’s first understand the processes and tools that are in place today.

• Development Process (the way we do it)
• Waterfall
  
• Iterative.
• eXtreme Programming
• RUP
• Tools (the hammer and tongs of it)
• Word, Visio, Excel, Power Point (all are documents)
• UML – Visual Paradigm, Rational
• Experience (hey cant dismiss that)
• Business experience (Knowing the business is critical)
• IT experience (What do we have, what do we need to build, the know how and prior experience with similar requirements)
  

Notice something?

While we have tools and processes that define the way we do it and have the tools and experience to execute, we seem to have ignored the big-fat-white-elephant-with-a-pink-ribbon sitting in the room all along. Change.

With every development project that we undertake, with the changing business, we are faced with change. From simple ones such as I-really-don’t-think-we-need-to-have-that-up-there-in-bold or can-you-make-the-text-label-SS#-instead-of-SSN?, to more esoteric and convoluted ones such as you-know-some-of-our-users-really-don’t-connect-to-the-internet-to-use-the-application or this-report-really-should-pull-data-from-our-old-Cobol-systems. Where are the tools required to control those ceaseless changes?

Simple question: So how does one control change?
Simple answer: You cannot. Change is permanent. You have to manage it.

In any requirements gathering exercise,

• Recognize the responsibilities of each group
• First, understand that requirements definition is a shared responsibility. Both the business and IT have equal stake in this process.
• Business team
• Describe business requirements in a simple language, preferably with ‘real world’ examples – and leave design and implementation to the IT.
• Prioritize. What do you really, really, really need? What do you really, really need? What do you really need? What can wait until the next release? Do you need to system to perform that function for you? Consider the bang for the buck – IT cannot tell you what is critical to your business.
• Signoff on all the documents, only after satisfying yourself that we have covered all that you need, accurately and appropriately. – RTFM
• Inform IT in advance about changes to business needs / scenario – do not wait until ship date to start hmm and hawing
• IT team
• Spend time to understand the business, the business goals and the business context – it’s not all about code.
• Identify functional and nonfunctional requirements – do not mix ‘em up.
• Inform the business and all stakeholders regarding development progress and problems in a timely fashion – if you are going to miss a ship date, then the business needs to know, and who knows they might re-prioritize, and you could get to push that nasty little corner-case functionality to the next release. Hey not all change is bad ;-)
• Get a good grip and over view of the entire system, so that you can effectively predict the downstream impact of changing requirements plus. You’ll know what can and can’t be done, and more importantly, what shouldn’t be done.
• Recognize the mode of expression
• Word documents don’t always tell the story effectively
• Not all requirements are describable in any one tool – mix ‘em up.
• Some requirements may be lying in undocumented live applications – legacy code is not always bad.
• Model Processes– work shops, walk-through’s, scripted role plays, Visio, flowcharts, UMLs
• Mock ups – WYSIWYG, but don’t get carried away
• Recognize the role of the business analyst
• At the crux of the business user and the IT staff lies this hybrid artificially created creature.
• Many business analysts sadly and erroneously confuse themselves as to be the business users – they need to align themselves as the bridge, the go-between, the translator.
• Other business analysts have their thinking clouded with prior application design knowledge, and try to ‘out think’ IT.
• IT personnel playing the business analyst begin to dictate requirements based on personal design principles or beliefs.
• Recognize the appropriate software tool
• There are several specialized tools out there in the market today that address specific needs
• Requirements definition
• Axure RP, Borland DefiniteIT, Compuware Optimal Trace, iRise, Ravenflow, Serena Composer, Sofea Profesy apart from modeling tools (such as Microsoft Visio) Microsoft Office, Graphics packages (e.g., Adobe Illustrator) HTML etc.
• Requirements management
• Borland CaliberRM, Compuware Optimal Trace, IBM Rational RequisitePro, Serena Dimensions RM, Telelogic DOORS, Test management tools, Change management tools and even and homegrown applications
• Research these tools and ascertain the best-fit in terms of
• Usability – are your business users, business analysts and developers all comfortable with the tool
• Life-cycle worthiness – can you use the tool from start (project conceptualization and initiation) to finish (several iterative changes later unto the final release and maintenance phases) seamlessly
• MS Project is not a tool for requirements management.
  

-----------
ref: The Root Of The Problem: Poor Requirements

e

Why 'e' ?

E is the fifth letter in the Latin alphabet. The letter e is the most commonly used letter in the English language.

e means different things to different people at different times.. electronic, enterprise, environment, energy, e=mc², excitement, entertainment, enthusiasm, empowering, engaging, explosion, exposition, example, evolution, e = 2.71828183, earth, exponential, exotic, evolutionary, electron, elementary change, explosive, enjoyment, .. so many more... you know more.. then you know what I'm referring to here.

The Problem with Pega...

In a seemingly innocuous note softly tucked away in a comparative discussion on Human-Centric Business Process Management Suites, analyst Connie Moore makes a this poignant and quietly thought provoking observation:

[most buyers consider Pegasystems a high-end business rules vendor with BPMS aspirations.8 In reality, Pegasystems - a successful company with 2004 revenues reaching $96.5 million — has completely morphed into a BPMS vendor that uses its business rules/process engine to tackle and simplify complex processes.

With SmartBPM Suite, the business rules engine (BRE) does more than codify and execute rules, which is how most other BPMS vendors use external BREs. SmartBPM Suite does that, but also uses business rules to empower rapid, iterative process design; to execute, monitor, and optimize dynamic processes; and automate processes in which changing conditions can drive each work item down a completely different path.

However, business rules can make the product overly complex. People sometimes conclude it’s nothing but a BRE because rules are used for everything - not just for business logic, but for processes, access control, data modeling, and system integration] - Connie Moore February 24, 2006 The Forrester Wave™: Human-Centric Business Process Management Suites, Q1 2006

(all emphasis mine)

On one hand Pegasystems does what is its purported to be (a BRE) much better than the other, and then delivers more than what the users think it can... however on the other hand, by handling all the tasks (access control, data modeling, system integration and the rest apart from business rules themselves) as rules.. causes users to dismiss it as
1. Either too complex or
2. Why do we need pega to do that?

a classic case of being too good for its own good? or a case of a misunderstood genius?

Human-Centric Business Process Management

Analyst Connie Moore devours 12 BPMS vendors across 215 criteria to understand the renewed trend towards Human-Centric business processes.

She segregates Business processes today into 4 major groupings:

• Integration intensive
  
• Order fulfillment, HIPAA transactions, Supply chain mgmt
  
• People intensive
• Employee on-boarding, Claims processing, Handling exceptions
  
• Decision intensive
  
• Mortgage loan origination, Underwriting, Retail inventory mgmt
  
• Document intensive
• Accounts payable, Contract mgmt, Proposal mgmt, SOX and other compliance processes
  

Given that most vendors play across these groupings and have offerings that could be classified under more than one group, she observes that the processes themselves are either

• Human-centric processes
  
• those that require people to get work done by relying on and interacting
  extensively with business applications, databases, collaboration tools, and documents
• ex. claims processing, loan approvals, accounts payable, and customer
  service
  
• System-intensive processes.
• manage interactions between packaged applications, custom applications,
  external applications
• typically involve millions of transactions per day that are handled
  on a straight-through basis with no to minimal human involvement and few exceptions
• ex. trade reconciliations, supply chain management, and line provisioning

Just as web application development in the late 90's and early 2000's went from a heavily data-centric (ERD's) to application-centric (J2EE, .NET) to finally user-centric design(Flash, Web 2.0, RIA) and architecture, the BPMS market seems to be headed towards the users of the tools.

Its the user. As long as the application caters to the user's needs, and allows him/her to do his/her job better and that tad bit easier, and can beat that nasty commute back home [or hopefully tele-commute], the application will be a success. It does not matter whether you built the application using J2EE, .NET, Oracle, Weblogic, Websphere, Lombardi, Savvion, PRPC or peppered it with the jargon flavor of the day - AJAX, SOA, Mashups, Web 2.0, CRM 2.0, BPM 2.0 etc., What matters is that the user feels comfortable using the tool, and feels that it has increased his/her productivity.

When you factor this into your design, and place the user and his/her actions at the center of your design .. and model the application to handle the processes that emanate from this core... you will have effective design. you will have an application that is intrinsically usable.

and in order to place individuals at the center of process analysis, Forrester suggests:

• Make the individual or small team the design point; focus on people in the process, rather than removing them from the process.
• Identify people/information-intensive processes
• Focus on the processes that are broken
• Think about what percentage of the IT budget is dedicated to empowering people vs. automating rote processes

All this focus on the user and people at the core of these processes, will surely interest vendors and IT departments that have been focused purely on automation and straight-through processing.

The result? Lombardi Software, Pegasystems, and Savvion lead with comprehensive suites that foster rapid, iterative process design

The message? the next wave of process optimization lies in empowering people.

---

the vendors evaluated : Appian, FileNet, Fuego, Fujitsu, Global 360, HandySoft, Lombardi, Metastorm, Pegasystems, Savvion, TIBCO, Ultimus

---