Often, as data grows, the number of desired outputs grows, and the time to execute these batch jobs grows too. Consequently tremendous efforts go into optimizing and squeezing out minimal gains in order to ensure the jobs complete by the time they are needed.

Enter Hadoop. Hadoop is an open source implementation of Google’s MapReduce and Google File System papers. It provides a distributed data store with very interesting parallel processing capabilities.

The parallel process is what you get when you setup a Hadoop cluster and write some MapReduce operations to do data analysis. The execution of your MapReduce operation gets triggered on all nodes of the cluster, and the results of the processing get consolidated into a common result set.

While the MapReduce implementation may not be ideal for generating the reports themselves, it can certainly help with the costly data crunching required to get to the end product.

I could mention other processing tools, but in the BigData landscape, Hadoop is enjoying a great deal of popularity, has an active community, has been around long enough to be mature and stable and has some interesting high-level tools for writing MapReduce operations.

I won’t assume that this is the solution to all your problems, but given that you could easily scale up a Hadoop cluster in cost effective increments, this can be an interesting alternative to buying traditional server horsepower.

If you’re wondering how you could improve your batch processing, then maybe it’s time you looked into Hadoop.

As per Randy Shoup’s classic (or personal favorite) presentation on eBay’s architecture and lessons learned, at any point in time at eBay there are systems down―the trick is to make a system that continues to run smoothly. As a consequence of this, their developers must actively design with failure in mind.

I have three thoughts on designing for failure:

• Asynchronous messaging is great for scaling up, but it is not cool to have important messages streaming into an untreated queue. To avoid this, it is important to have a degree of redundancy; enough systems to ensure that messages don’t appear to drop off into a void.

• A resilient design will help you smoothly roll over bumps in the road, but quick diagnosis and correction are still very important. As the number of moving parts increases, it can get harder to know what is broken. A good monitoring strategy should also be part of a resilient design. Design to make early detection possible. You don’t want your system to be a nightmare to troubleshoot, and traditional logging is probably not enough anymore―SNMP, UDP heartbeats, log consolidation and dashboards are not uncommon in these contexts.

• Having designed for failure, you may sleep well at night, but unless you are testing that the system continues to run as expected through a failure, you may be in for a rude awakening. Like backups, fault tolerance should be tested. I’ve heard of organizations that have test environments in which at least one node is down at any given moment. In theory, this should be transparent to the end user. Netflix is known to do fault testing and has made their chaos monkey testing software public.

Are you ready for a failure? If not, then maybe it’s time to start preparing. It’s coming.

Happy failing!

This rule proposes that, if scaling up is a concern, then design for 20 times capacity, implement for 3 times capacity and deploy for 1.5 times capacity.

While I can’t vouch for their numbers, nor am I going to re-explain all their thinking, I’d like to share how this aligns itself with architecture and design on Agile projects.

Firstly, the dilemma of the Agile architect is often how to design for possible scale up without investing a great deal of effort and code on it up front. Such an early investment in scaling code can cause software projects to deliver little or no value early on, and can result in great waste if the project never reaches a stage of success that requires such scalability.

The idea behind this design-implement-deploy progression is as follows. If there is a legitimate concern for scaling up at some point, then there is value in identifying obvious and avoidable performance bottlenecks and designing so that they can be isolated, avoided, or worked around. This means determining if synchronous processing, data locking, near-consistency or other factors will make it difficult to scale up and design to account for those issues with significant impacts. This does not mean plan optimizations in potential hotspots, but plan for frictionless performance where it can make a worthwhile difference.

Next, implement. But only enough to make the design work and validate the design. This means that, if you’ve identified that allowing for stale entail consistency will be the leverage for a scale up, then treat data as potentially stale when implementing. If a pub/sub architecture will be key to scaling up, then implement a simple pub/sub or use a lower end ESB; you can plug in a more robust or high-performance version later if needed. In short, make it work for the desired design, but don’t optimize it for the 20 times capacity. If you need to scale up, you will have created the entry points to introduce the necessary improvements to your system.

Finally, deploy for 1.5 time capacity. 1.5 is not a great deal more than your expected capacity, but it should buy you time to tune and adjust for greater scale should you find you really need it.

So basically KISS still applies, but be a little more critical of YAGNI. Your scalable design will still have cost more than the simplest possible design, but your implementation should not cost much more initially. Your deployment will pretty much compare to the one you need.

Stay tuned for scalable design tricks!

The world of Agility, more specifically the Scrum framework, is in high demand by organizations looking to improve their software development practices. Urban Turtle is a set of Agile modules for Microsoft’s Team Foundation Server platform. More specifically, it seamlessly integrates with Team Web Access (the platform’s web portal). In fact, Urban Turtle offers teams around the world a set of Agile tools that facilitate the planning, execution, and follow-up of their projects.

Our application of Agile methods allows us to deliver value every month. Here is our objective for 2013: 12 versions in 12 months.

“Since the very first version in 2008, we are working to make software development simpler. After 5 years and 3 versions, we are very proud of all that we have accomplished. I would like to thank the Urban Turtle team members and all Pyxissians who contribute to the commercial success of our product,” says Dominic Danis, Product Owner.

“We make considerable efforts to provide value to end users and to improve our product. Urban Turtle is in line with the /studio’s mission, and I am very proud of the work accomplished by the team,” indicates Martin Landreville, vice-president of solutions and product at Pyxis /studio.

The team is looking forward to May 25th to find out the winner in the “Commercial Success” category. We wish the entire Urban Turtle team good luck!

Failure can be translated into an exception, a runtime error, or even a runtime failure resulting in unstoppable application termination.

Traditionally, assertions are not compiled into production code. Instead, it’s being used as debug/test time safeguards. In such cases, the goal is to detect erroneous behaviour or non-compliant use of methods during application validation, and then remove such internal checks in order to reduce the drag on application performance and avoid any unsavoury failures in production use of the application.

One common place in which assertions are used is in unit test frameworks. All the unit testing frameworks provide a way of validating for the expected condition or result following a test. Once again, these are rarely moved into production, and the assertions are written outside the production code. Note that the intent of the test and the assertion differ somewhat, but the mechanics of the assertion are the same.

Times have changed, but popular views of assertions remain somewhat antiquated.

Even though we like to think of assertions as debug and test-time checks, the reality is that the assertion-model is alive and well in the core runtimes and classes of the Java and .NET platforms.

Take a look at the API documentation for any of the system classes. Most methods will throw an exception if one of the arguments passed to them is null or invalid in some way. Usually such exceptions identify which argument is at fault. This can save a lot of time troubleshooting—you don’t need to look inside a method to find where the problem is, it fails early instead of fails eventually.

Implementing such preventive APIs is pretty straightforward, starting each method with a block of argument validation. Typically these are written as a series of “if/throw” statements. These can take up a lot of vertical and horizontal space in the code that often obscure their intent.

This brings us back to the conciseness of the assertion. Rather than writing a verbose validation block or handling basic assertion failures, I prefer to write assertion-style helpers that throw standard exceptions on failure.

Thus, this allows this:

public void SomeVerboseOperation(string sin, PostalAddress address)
{
    if (address == null)
        throw new ArgumentNullException("address");
    if (string.IsNullOrEmpty(sin))
        throw new ArgumentNullException("sin");
    // do stuff
}

To become this:

public void SomeOperation(string sin, PostalAddress address)
{
    Validate.ArgumentIsNotNull(address, "address");
    Validate.ArgumentIsNotEmpty(sin, "sin");
    // do stuff
}

I expect this to throw an argument null exception that points to the argument named “postalCode”. Fairly self-explanatory.

Other possible forms that work well include “IsNotEmpty” for strings and “IsPositive” for integers. Other possibilities include regular expression checks and many more.

I can’t take the credit for this approach. I borrowed it from the DDD sample application that Eric Evans published some years back (dddsample.sourceforge.net).

As an aside, I would point out that some languages support notions of method pre-conditions, effectively providing a formal mechanism for building this concept into the program. Some tools exist to inject this kind of behavior into C# and Java, but I find that these take away from the clarity of the code.

Does anybody have any other tricks to clean up argument validation checks?

Now, funny enough, after a few weeks of “frost training”, I tend to enjoy the snowy strolls. I am also very proud to be part of this great organization. And this is why…

A vision

The team is led by François Beauregard whose vision is to enable businesses to create software intelligently and efficiently whilst ensuring a great experience for their teams and clients. There is a huge human component to it where highly satisfied teams and clients is the norm. He guides Pyxis to lead the Agile transition movement by pushing the boundaries of excellence and by creating a unique offer on the market.

A unique offer

Pyxis’ offer aims to help organizations transition to Agile by looking at it in a different way than the rest of the market. It benefits from huge amount of know-how and experiences accumulated over the last twelve years. It is also supported by highly skilled cross-functional teams; all sharing the same vision.

At Pyxis, we live what we offer, and within the organization, core teams are working intrinsically together to achieve our goals. We are an Agile development team (/Studio) and we create great solutions with our partners. As developers, we understand the tools and techniques, and we constantly look for ways to improve in order to be more efficient. It is our laboratory.

We are also trainers (/Campus) and we are eager to share our knowledge around. It is essential that our partners receive the best training possible so that they have the skills to support their organizations.

To support our clients further in their Agile transition, we are specialized coaches (/Consulting) who can help on organizational, team, and technical levels.

The combination of these core components creates a powerful driving force and a strong platform on which companies can build their Agile environments.

Our team members’ skills are continually strengthened by the interactions of the teams and the fact that they can rotate between them. This is what makes the Pyxis offer unique; the depth of our skills and the cohesion between our teams ensure that we can respond to our partner’s needs in the most meaningful way; we have “Savoir-Faire” and “Savoir-Être”.

Whilst we are focused to achieve our goals, we do not want to lose our values which are based on balance, quality of life, and unexpected results. It is essential that we keep them in check if we want to stay at the leading edge of our craft.

My contribution

By joining the /Consulting team, I aim to help Pyxis make a difference, reach its goals, and serve our partners the best way possible. I am very excited to be part of the journey and I have no doubt I will learn and grow as we put ourselves in innovative positions. This part aligns very well with my personal goals and will be a key component of my growth as a coach.

C’est trippant!

Because of this, its not uncommon to find oneself dropped into such projects. We grumble and groan and hope to get off them as quickly as possible. Well buck up, greenfields projects are a rarity so here is where we practice our trade.

The first hurdle to cleaning up (or even just modifying) a big system is having to worry about non-regression. Now there are no magic bullets here, the only way to add tests to the behemoth is to roll up your sleeves and dive in. Well, sort of. It’s useful to find a seam through which to inspect the system. One such seam is the gigabytes of XML that many systems produce for message exchanges. These XML messages are useful for non-regression testing—they are structure, are either input or output from the system and it is possible to validate them. For more on the subject of testing with legacy systems and seams, refer to Michael Feather’s treatise “Working Effectively with Legacy Code”.

Even though it’s been around for a while, XML validation is often limited to schema and DTDs. I’ve also seen XML validation done by scanning the XML for some expected sub-strings or the brute force comparison of the resulting XML against the expected XML. These options each have their merits, but all fall short in some way. The main irritant is not having all the differences highlighted upon assertion failure. I find that I need a better XML comparison/assertion. To that end I went looking for tools that can be easily used within JUnit tests.

This library has been around long enough to have stagnated, but it continues to save my hide on projects. I’m talking about XMLUnit (XMLUnit.sourceforge.net)

With the following helper method in place, I forge ahead with integration tests on my side.

import org.custommonkey.xmlunit.DetailedDiff;
import org.custommonkey.xmlunit.Diff;
import org.custommonkey.xmlunit.XMLUnit;
public class AssertXml {
    public static void areSimilar(String expected, String result) {
        XMLUnit.setIgnoreWhitespace(true);
        XMLUnit.setNormalizeWhitespace(true);
        DetailedDiff diff;
        try {
            diff = new DetailedDiff(new Diff(expected, result));
        } catch (Exception e) {
            throw new RuntimeException(e);
        }

        if (!diff.similar()) {
            StringBuffer differenceDescription = new StringBuffer();

            differenceDescription.append(diff.getAllDifferences().size()).append(" differences");
            differenceDescription.append(diff.toString());

            differenceDescription.append("expected => \n").append(expected);
            differenceDescription.append("result => \n").append(result);

            org.junit.Assert.fail(differenceDescription.toString());
        }

    }
}

This is useful when refactoring legacy systems that produce XML. It allows me to capture XML output from a system, validate it manually, and use the captured XML as the expected output to a given test. I test the execution output against my expected output to make sure that I haven’t changed any behaviour in the system while refactoring.

@Test
public void testSimila() {
    AssertXml.areSimilar("", "");
}

For best results, capture a variety of outputs and allow for a greater non-regression. Often this has problems with temporal data (as simple as current system date and time). This give a first target for refactoring—breaking this out. Some are easier to break than others, the simplest are the static contexts like the current system time (see my older post on this http://pyxis-tech.com/blog/en/2009/09/03/abstracting-out-static-contexts/).

Because testing a large XML typically involves varied data processed by varied rules, this is not a viable long-term solution for integration testing, the tests are fragile and not adequately targeted. It does give me a safety net to start breaking apart the tight core of code and splitting out smaller, unit tested portions while ensuring the whole essentially behaves the same.

This strategy can still be used with small, target bits of generated XML as part of a longer term test strategy, but the code needs to adapted to generate isolated fragments of the XML.

Xmlunit comes in a Java and .NET flavor (though I’ve not used the .NET version) and ranks as one of my favourite  old, dead, but very useful libraries.

At the core of the new Urban Turtle platform is the ability for companies, development teams, or individuals to purchase applications based on their goals and objectives. For instance, managers seeking to track project progress can purchase Urban Turtle’s newest add-on, a dashboard for TFS, without leaving the application. The dashboard offers configurable widgets, including an innovative Sunset Graph that project managers will love.

“The new Urban Turtle platform allows clients to extend TFS with incredible ease. Once initial software is installed, users pick and choose the applications they need and customize them, if need be,” says Dominic Danis, Product Director. The result allows users to avoid costly and time-consuming software acquisition and installation delays. Clients can now get the software they need to be successful—when they need it (24/7).

To support this new direction, and building on Urban Turtle’s hugely popular planning and task boards, the team has already introduced several new applications, including an estimation board to improve and speed up the estimation process and Fast Track, a highly configurable board that allows teams to manage interruptions (e.g. support tickets) and simultaneously maintain sprint commitments.

“This is just the beginning. Our team is committed to developing SMART innovations for our growing client community,” Dominic continued. Moving forward, Urban Turtle will regularly introduce new and valuable applications. More than ever, the team is dedicated to delivering awesome software to help Agile teams around the world be more successful.

About Urban Turtle

Urban Turtle helps over 700 companies in 21 countries build better software faster. Widely known as the leading Agile development tool for Microsoft Team Foundation Server, Urban Turtle has evolved into an application platform for TFS. The new platform, Urban Turtle 2012, is the most powerful, tightly-integrated, and comprehensive software offering that empowers companies to optimize software development—and have fun doing it. Applications in the software platform include backlog management, a task board, a dashboard, an estimation board, a daily stand-up timer, and much more. For the love of Agile software and to learn more about Urban Turtle TFS add-ons, find us at urbanturtle.com.

About Pyxis Studio

The Studio is the software development division of Pyxis Technologies inc. We develop sustainable applications, maintain existing ones, and assist development teams by providing Agile products as well as technical guidance and professional services on how to implement state-of-the-art development techniques. Our Studio lives for unique and personalized software designs and inspiring projects that we can carry out with or for our clients. We welcome our clients into our teams where we not only develop software but create unique experiences that bring software development to a higher level.

For additional information: Dominic Danis, 1.855.875.2072 ext. 1, ddanis@urbanturtle.com

# # #

The trademarks and trade names mentioned herein may be the trademarks of their respective owners and are hereby acknowledged.

I’m an Agile coach with 5 teams to assist and not much hair left on my noggin, mainly due to scratching it too often, which in turn is due to too much thinking. I’m stuck once again trying to figure out what item, from many, to tackle from my “Scrum Teams Notes & Observations” list. My first thought: How do I prevent hair loss? What I really mean to say is: How do I order my list?

What I had to work with was a set of “comfort and interest” self-assessment scores from a previous activity (Analyze This).

So I decided to try something different. I produced, with the self-assessment scores, a plot chart with numbers representing a Scrum activity and colours representing each Scrum Master. Low and behold, what I discovered were visual groupings in the chart that identified people that considered themselves as experts (blue), within the average (yellow), and below average (green, red, and black). I also noticed groupings of activities that were not of interest (5′s), interesting but not understood (7′s), interesting and understood (1′s).

I decided to work with the activities or individuals which or who had a low comfort level but which or who also had sufficient interest levels (greens and 7′s). My decision was based on the theory that working on interested individuals and activities needing assistance might positively influence the individuals’ future assessments.
Going through this exercise provided multiple benefits; no more hair loss, an ordered list, and more money in my wallet—ROGAINE^® costs a fortune here