ITK

The Nature of the Beast

If you did not understand the nature of the beasts,
it would be of little use to know the mechanics of their anatomy.

Dee Hock, One from Many: VISA and the Rise of Chaordic Organization

In a typical image analysis problem, a researcher or an engineer will take an input image, improve some characteristics of the image by, let's say, reducing noise or increasing contrast, and then proceed to identify some features in the image, such as corners and strong edges. This type of processing is naturally well-suited for a data pipeline architecture, as shown in Figure 9.1.

To illustrate this point, Figure 9.2 shows an image of a brain from a magnetic resonance image (MRI), and the result of processing it with a median filter to reduce its level of noise, as well as the outcome of an edge detection filter used to identify the borders of anatomical structures.

Process and Data Objects

Two main types of objects were designed to hold the basic structure of the pipeline. They are the DataObject and the ProcessObject. The DataObject is the abstraction of classes that carry data; for example, images and geometrical meshes. The ProcessObject provides an abstraction for the image filters and mesh filters that process such data. ProcessObjects take DataObjects as input and perform some type of algorithmic transformation on them, such as the ones illustrated in Figure 9.2.

DataObjects are generated by ProcessObjects. This chain typically starts by reading a DataObject from disk, for example by using a ImageFileReader which is a type of ProcessObject. The ProcessObject that created a given DataObject is the only one that should modify such DataObject. This output DataObject is typically connected as input to another ProcessObject downstream in the pipeline.

This sequence is illustrated in Figure 9.5. The same DataObject may be passed as input to multiple ProcessObjects, as it is shown in the figure, where the DataObject is produced by the file reader at the beginning of the pipeline. In this particular case, the file reader is an instance of the ImageFileReader class, and the DataObject that it produces as output is an instance of the Image class. It is also common for some filters to require two DataObjects as input, as it is the case of the subtract filter indicated in the right side of the same figure.

The ProcessObjects and DataObjects are connected together as a side effect of constructing the pipeline. From the application developer's point of view, the pipeline is linked together by invoking a sequence of calls involving the ProcessObjects such as:

writer->SetInput( canny->GetOutput() );
canny->SetInput( median->GetOutput() );
median->SetInput( reader->GetOutput() );

Internally, however, what is connected as a consequence of these calls is not one ProcessObject to the next ProcessObject, but the downstream ProcessObject to the DataObject that is produced by the upstream ProcessObject.

The internal chained structure of the pipeline is held together by three types of connections:

• The ProcessObject holds a list of pointers to its output DataObjects. Output DataObjects are owned and controlled by the ProcessObject that produces them.
• The ProcessObject holds a list of pointers to its input DataObjects. Input DataObjects are owned by the upstream ProcessObject.
• The DataObject holds a pointer to its producer ProcessObject. That happens to be the ProcessObject that also owns and control this DataObject.

This collection of internal links is later exploited to propagate calls upstream and downstream in the pipeline. During all these interactions, the ProcessObject retains control and ownership of the DataObject that it generates. The filters downstream gain access to the information about a given DataObject through the pointer links that are established as a consequence of the calls to the SetInput() and GetOutput() methods, without ever taking control of that input data. For practical purposes, filters should treat their input data as read-only objects. This is enforced in the API by using the C++ const keyword in the arguments of SetInput() methods. As a general rule, ITK embraces a const-correct external API, even though internally this const-correctness is overridden by some of the pipeline operations.

The Pipeline Class Hierarchy

The initial design and implementation of the Data Pipeline in ITK was derived from the Visualization Toolkit (VTK), a mature project at the time when ITK development began. (See The Architecture of Open Source Applications, Volume 1.)

Figure 9.6 shows the object-oriented hierarchy of the pipeline objects in ITK. In particular, note the relationship between the basic Object, ProcessObject, DataObject, and some of the classes in the filter family and the data family. In this abstraction, any object that is expected to be passed as input to a filter, or to be produced as output by a filter, must derive from the DataObject. All filters that produce and consume data are expected to derive from the ProcessObject. The data negotiations required to move data through the pipeline are implemented partly in the ProcessObject and partly in the DataObject.

The LightObject and Object classes are above the dichotomy of the ProcessObject and DataObject. The LightObject and Object classes provide common functionalities such as the API for communications of Events, and the support for multi-threading.

The Inner Workings of the Pipeline

Figure 9.7 presents a UML sequence diagram describing the interactions between ProcessObjects and DataObjects in a minimal pipeline composed of an ImageFileReader, MedianImageFilter and ImageFileWriter.

The full interaction consist of four passes:

• Update Output Information (upstream call sequence)
• Update Requested Region (upstream call sequence)
• Update Output Data (upstream call sequence)
• Generate Data (downstream call sequence)

The whole process is triggered when an application invokes the Update() method in the last filter of the pipeline; in this concrete example this is the ImageFileWriter. The Update() call initiates the first pass that goes in the upstream direction. That is, from the last filter in the pipeline, towards the first filter in the pipeline.

The goal of this first pass is to ask the question, "How much data can you generate for me?" This question is codified in the method UpdateOutputInformation(). In this method, every filter computes the amount of image data that can be produced as output with the given amount of data available to it as input. Given that the amount of data input must be known first before the filter can answer the question about the amount of data output, the question has to propagate to the filter upstream, until it reaches a source filter that can answer the first question by itself. In this concrete example, that source filter is the ImageFileReader. This filter can figure out the size of its output by gathering information from the image file that it has been assigned to read. Once the first filter of the pipeline answers the question, then the subsequent filters downstream can compute their respective amount of output one after another, until they make it to the last filter of the pipeline.

The second pass, which also travels in the upstream direction, informs filters as to the amount of output that they are requested to produce during pipeline execution. The concept of Requested Region is essential in supporting the streaming capabilities of ITK. It makes it possible to tell the filters in the pipeline not to generate the entire full image, but to focus instead in a subregion of the image, the Requested Region. This is very useful when the image at hand is larger than the RAM available in the system. The call propagates from the last filter to the first one, and at every intermediate filter the requested region size is modified to take into account any extra borders that a filter may need in the input so it can generate a given region size as output. In our concrete example, the median filter will typically have to add a 2-pixel border to the size of its own input. That is, if the writer requests a region of size 500 x 500 pixels to the median filter, the median filter in its turn will request a region of 502 x 502 pixels to the reader, because the median filter by default needs a 3 x 3 pixel neighborhood region to compute the value of one output pixel. The pass is encoded in the PropagateRequestedRegion() method.

The third pass is intended to trigger the computation on the data inside the Requested Region. This pass also goes in the upstream direction and it is codified in the UpdateOutputData() method. Since every filter needs its input data before it can compute its output data, the call is passed to the respective upstream filter first, hence the upstream propagation. Upon return the current filter actually proceeds to computes its data.

The fourth and final pass proceeds downstream, and consists of the actual execution of computation by every filter. The call is codified in the GenerateData() method. The downstream direction is not a consequence of one filter making calls on its downstream partner, but rather of the fact that the UpdateOutputData() calls are executing in order from the first filter to the last filter. That is, the sequence happens downstream due to timing of the calls, and not due to what filter is driving the calls. This clarification is important because the ITK pipeline is by nature a Pull Pipeline, in which data is requested from the end, and the logic is also controlled from the end.

Embracing Diversity with Facades

The image analysis community has developed a very large set of file formats to store image data. Many of these file formats are designed and implemented with specific uses in mind, and therefore are fine-tuned to specific types of images. As a consequence, on a regular basis, new image file formats are conceived and promoted across the community. Aware of this situation, the ITK development team designed an IO architecture suitable for ease of extensibility, in which it is easy to add support for more and more file formats on a regular basis.

This IO extensible architecture is built upon the Factory mechanism described in the previous section. The main difference is that in the case of IO, the IO Factories are registered in a specialized registry that is managed by the ImageIOFactory base class, shown on the upper left corner of Figure 9.8. The actual functionality of reading and writing data from image file formats is implemented in a family of ImageIO classes, shown on the right side of Figure 9.8. These service classes are intended to be instantiated on demand when the user requests to read or write an image. The service classes are not exposed to the application code. Instead, applications are expected to interact with the facade classes:

• ImageFileReader
• ImageFileWriter

These are the two classes with which the application will invoke code such as:

reader->SetFileName("../image1.png");
reader->Update();

or:

writer->SetFileName("../image2.jpg");
writer->Update();

In both cases the call to Update() triggers the execution of the upstream pipeline to which these ProcessObjects are connected. Both the reader and the writer behave as one filter more in a pipeline. In the particular case of the reader, the call to Update() triggers the reading of the corresponding image file into memory. In the case of the writer, the call to Update() triggers the execution of the upstream pipeline that is providing the input to the writer, and finally results in an image being written out to disk into a particular file format.

These facade classes hide from the application developer the internal differences that are inherent to the particularities of each file format. They even hide the existence of the file format itself. The facades are designed in such a way that most of the time application developers do not need to know what file formats are expected to be read by the application. The typical application will simply invoke code such as

std::string filename = this->GetFileNameFromGUI();
writer->SetFileName( filename );
writer->Update();

These calls will work fine regardless of whether the content of the filename variable is any of the following strings:

• image1.png
• image1.jpeg
• image1.tiff
• image1.dcm
• image1.mha
• image1.nii
• image1.nii.gz

where the file name extensions identify a different image file format in every case.

Know Thy Pixel Type

Despite the assistance that the file reader and writer facades provide, it is still up to the application developer to be aware of the pixel type that the application needs to process. In the context of medical imaging, it is reasonable to expect that the application developer will know whether the input image will contain a MRI, a mammogram or a CT scan, and therefore be mindful of selecting the appropriate pixel type and image dimensionality for each one of these different image modalities. This specificity of image type might not be convenient for application settings where users wants to read any image type, which are most commonly found in the scenarios of rapid prototyping and teaching. In the context of deploying a medical image application for production in a clinical setting, however, it is expected that the pixel type and dimension of the images will be clearly defined and specified based on the image modality to be processed. A concrete example, where an application manages 3D MRI scans, looks like:

typedef itk::Image< signed short, 3 >  MRImageType;
typedef itk::ImageFileWriter< MRImageType > MRIWriterType;
MRIWriterType::Pointer writer = MRIWriterType::New();
writer->Update();

There is a limit, however, to how much the particularities of the image file formats can be hidden from the application developer. For example, when reading images from DICOM files, or when reading RAW images, the application developer may have to insert extra calls to further specify the characteristics of the file format at hand. DICOM files will be the most commonly found in clinical environments, while RAW images are still a necessary evil for exchanging data in the research environment.

Together But Separate

The self-contained nature of every IO Factory and ImageIO service class is also reflected in the modularization. Typically, an ImageIO class depends on a specialized library that is dedicated to managing a specific file format. That is the case for PNG, JPEG, TIFF and DICOM, for example. In those cases, the third-party library is managed as a self-contained module, and the specialized ImageIO code that interfaces ITK to that third-party library is also put in a module by itself. In this way, specific applications may disable many file formats that are not relevant to their domain, and can focus on offering only those file formats that are useful for the anticipated scenarios of that application.

Just as with standard factories, the IO factories can be loaded at run-time from dynamic libraries. This flexibility facilitates the use of specialized and in-house developed file formats without requiring all such file formats to be incorporated directly into the ITK toolkit itself. The loadable IO factories has been one of the most successful features in the architectural design of ITK. It has made it possible to easily manage a challenging situation without placing a burden on the code or obscuring its implementation. More recently, the same IO architecture has been adapted to manage the process of reading and writing files containing spatial transformations represented by the Transform class family.

Reusability

The principle of reusability can also be read as "avoidance of redundancy". In the case of ITK, this has been achieved with a three-pronged approach.

• First, the adoption of object-oriented programming, and in particular the proper creation of class hierarchies where common functionalities are factorized in base classes.
• Second, the adoption of generic programming, implemented via the heavy use of C++ templates, factorizing behaviors that are identified as patterns.
• Third, the generous use of C++ macros has also permitted reuse of standard snippets of code that are needed in myriad places across the toolkit.

Many of these items may sound like platitudes and appear obvious today, but when ITK development started in 1999 some of them were not that obvious. In particular, at the time the support most C++ compilers offered for templates did not quite follow a consistent standard. Even today, decisions such as the adoption of generic programming and the use of a widely templated implementation continue to be controversial in the community. This is manifested in the communities that prefer to use ITK via the wrapping layers to Python, Tcl or Java.

typedef unsigned char PixelType;
const unsigned int Dimension = 3;
typedef itk::Image< PixelType, Dimension > ImageType;
ImageType::Pointer image = ImageType::New();

typedef itk::MedianImageFilter< ImageType, ImageType> FilterType;
FilterType::Pointer median = FilterType::New();

Maintainability

The architecture satisfies the constraints that minimize maintenance cost.

• Modularity (at the class level)
• Many small files
• Code reuse
• Repeated patterns

The Invisible Hand

Software should look like it was written by a single person. The best developers are the ones who write code that, should they be hit by the proverbial bus, can be taken over by anybody else. We have grown to recognize that any trace of a "personal touch" is an indication of a defect introduced in the software.

In order to enforce and promote code style uniformity, the following tools have proved to be very effective:

• KWStyle for automatic source code style checking. This is a simplified C++ parser that checks coding style and flags any violations.
• Gerrit for regular code reviews. This tools serves two purposes: On one hand, it prevents immature code from entering the code base by distilling its errors, deficiencies and imperfections during iterative review cycles where other developers contribute to improve the code. On the other hand, it provides a virtual training camp in which new developers get to learn from more experienced developers (read "experienced" as have made all the mistakes and know where the bodies are buried…) how to improve the code and avoid known problems that have been observed during maintenance cycles.
• Git hooks that enforce the use of the KWStyle and Gerrit and that also perform some of their own checks. For example, ITK uses Git hooks that prevent commits of code with tabs or with trailing blank spaces.
• The team has also explored the use of Uncrustify as a tool for enforcing a consistent style.

It is worth emphasizing that uniformity of style is not a simple matter of aesthetic appeal, it is really a matter of economics. Studies on the Total Cost of Ownership (TCO) of software projects have estimated that in the life-cycle of a project, the cost of maintenance will be about 75% of the TCO, and given that maintenance cost is applied on an annual basis, it typically surpasses the cost of initial development costs by the first five years of the life-cycle of a software project. (See "Software Development Cost Estimating Handbook", Volume I, Naval Center for Cost Analysis, Air Force Cost Analysis Agency, 2008.) Maintenance is estimated to be about 80% of what software developers actually do, and when engaged in that activity the large majority of the developer's time is dedicated to reading someone else's code, trying to figure out what it was supposed to do (see Clean Code, A Handbook of Agile Software Craftsmanship, Robert C. Martin, Prentice Hall, 2009). Uniform style does wonders for reducing the time it takes for developers to immerse themselves in a newly open source file and understand the code before they make any modifications to it. By the same token, it reduces the chances that developers will misinterpret the code and make modifications that end up introducing new bugs when they were honestly trying to fix old bugs (The Art of Readable Code, Dustin Boswell, Trevor Foucher, O'Reilly, 2012).

The key for making these tools effective is to make sure that they are:

• Available to all developers, hence our preference for Open Source tools.
• Run on a regular basis. In the case of ITK, these tools have been integrated in the Nightly and Continuous Dashboard builds managed by CDash.
• Run as close as possible to the point where the code is being written, so that deviations can be fixed immediately, and so developers quickly learn what kind of practices break style rules.

Refactoring

ITK started in 2000 and grew continuously until 2010. In 2011, thanks to an infusion of federal funding investment, the development team had the truly unique opportunity to embark on a refactoring effort. The funding was provided by the National Library of Medicine as part of the initiative of the American Recovery and Reinvestment Act (ARRA). This was not a minor undertaking. Imagine you have been working on a piece of software for over a decade, and you are offered the opportunity to clean it up; what would you change?

This opportunity for widespread refactoring is very rare. For the previous ten years, we had relied on the daily effort of performing small local refactorings, cleaning up specific corners of the toolkit as we ran into them. This continuous process of clean up and improvement takes advantage of the massive collaboration of open source communities, and it is safely enabled by the testing infrastructure driven by CDash, which regularly exercises about 84% of the code in the toolkit. Note that in contrast, the average code coverage of software industry testing is estimated to be only 50%.

Among the many things that were changed in the refactoring effort, the ones that are most relevant to the architecture are:

• Modularization was introduced in the toolkit
• Integer types were standardized
• Typedefs were fixed to allow management of images larger than 4 GB on all platforms
• The software process was revised:
  • Migrated from CVS to Git
  • Introduced code review with Gerrit
  • Introduced testing on demand with CDash@home
  • Improved method for downloading data for unit testing
• Deprecated support for obsolete compilers
• Improved support for many IO image file formats including:
  • DICOM
  • JPEG2000
  • TIFF (BigTIFF)
  • HDF5
• Introduced a framework for supporting GPU computation
• Introduced support for video processing
  • Added a bridge to OpenCV
  • Added a bridge to VXL

Maintenance based on incremental modifications—tasks such as adding features to an image filter, improving performance of a given algorithm, addressing bug reports, and improving documentation of specific image filters—works fine for the local improvement of specific C++ classes. However, a massive refactoring is needed for infrastructure modifications that affect a large number of classes across the board, such as the ones listed above. For example, the set of changes needed to support images larger than 4 GB was probably one of the largest patches ever applied to ITK. It required the modification of hundreds of classes and could not have been done incrementally without incurring in a great deal of pain. The modularization is another example of a task that could not have been done incrementally. It truly affected the entire organization of the toolkit, how its testing infrastructure works, how testing data is managed, how the toolkit is packaged and distributed, and how new contributions will be encapsulated to be added to the toolkit in the future.

Reproducibility

One of the early lessons learned in ITK was that the many papers published in the field were not as easy to implement as we were led to believe. The computational field tends to over-celebrate algorithms and to dismiss the practical work of writing software as "just an implementation detail".

That dismissive attitude is quite damaging to the field, since it diminishes the importance of the first-hand experience with the code and its proper use. The outcome is that most published papers are simply not reproducible, and when researchers and students attempt to use such techniques they end up spending a lot of time in the process and deliver variations of the original work. It is actually quite difficult, in practice, to verify if an implementation matches what was described in a paper.

ITK disrupted, for the good, that environment and restored a culture of DIY to a field that had grown accustomed to theoretical reasoning, and that had learned to dismiss experimental work. The new culture brought by ITK is a practical and pragmatic one in which the virtues of the software are judged by its practical results and not by the appearance of complexity that is celebrated in some scientific publications. It turns out that in practice the most effective processing methods are those that would appear to be too simple to be accepted for a scientific paper.

The culture of reproducibility is a continuation of the philosophy of test driven development, and systematically results in better software; higher clarity, readability, robustness and focus.

In order to fill the gap of lack of reproducible publications, the ITK community created the Insight Journal. It is an open-access, fully online publication in which contributions are required to include code, data, parameters, and tests in order to enable verification by reproducibility. Articles are published online less than 24 hours after submission. Then they are made available for peer-review by any member of the community. Readers get full access to all the materials accompanying the article, namely source code, data, parameters, and testing scripts. The Journal has provided a productive space for sharing new code contributions which from there make their way into the code base. The Journal recently received its 500th article, and continues to be used as the official gateway for new code to be added to ITK.