Saturday, September 26, 2009

Keeping your core stable

In my post from May 19 2009, titled "Extend, don't customize", I wrote about the importance of implementing customer specific adaptations in a way that does not interfere with the regular maintenance and upgrade operations. I also described how Openbravo ERP 2.50 addresses this challenge and how, through the thoughtful usage of modularity, users can implement even the most invasive adaptations though extensions.

This argument holds true as long as two conditions are met:
  1. Most adaptations are possible as extensions, without requiring changes to the application itself.
  2. The interface and the semantics of the application are stable and we can be guarantee that any extension that works against a specific version of Openbravo will continue to work when the application is updated to subsequent versions through maintenance packs.

It is my belief that both conditions are satisfied in Openbravo ERP 2.50 and today I would like to describe how we ensure that Openbravo is stable and that that the second condition holds true, leaving the exploration of the first point to future blog posts.

Let's start with some background information.

Openbravo ERP is itself delivered as a module that we call Core. All extensions, both base modules and templates, reference Core. Templates can directly change the attributes of artifacts included in Core; for example, a template can disable a field. Base modules, on the other hand, cannot change Core directly but can influence its behavior by adding artifacts that add to Core; for example, a module can add additional fields to a sales invoice line and consider those fields at invoice completion by adding to the extension point of the Invoice Post process.

In all cases, modules depend on Core to the extent that they reference artifacts that are defined in Core. It is therefore essential, to keep modules working, that Core artifacts are not changed or removed as part of the maintenance of the application. By guaranteeing that, we ensure that users can confidently apply mainteanance packs without the risk of breaking their adaptations.

We call the set of Core artifacts that other modules can reference the Openbravo Public Application Programming Interface or the Public API for shorts.

In other softwares, the term API usually refers to the signature of procedures and modules that can be invoked to execute specific functions. In the case of Openbravo, that definition is too restrictive as the application is meta data driven and, in many cases, modules interact with meta data rather than procedures and classes (actually, in a perfect meta data driven application, all interactions are at meta data level).
We are therefore using the term public API in a more generic sense, referring to all artifacts that other modules can reference in their definition, including:
  • Web services
  • Public Java classes and methods
  • Javascript libraries that developers can use in their manual UIs
  • All meta-data defined in the Application Dictionary
  • PL/SQL functions and procedures
  • Database tables and views

Soon after the launch of 2.50, we completed the formal definition of the public API. You can find the results of this work as part of the Developer's Guide.

Once the public API had been defined, the next step was to guarantee its consistency and perpetuity. This is particularly challenging in the case Openbravo given the very diverse nature of the artifacts it comprises. To achieve this objective, we have developed a tool that compares two revisions of the Openbravo code and performs a number of checks to ensure that there are no changes in the public API that could break already published modules.
For instance, this tool verifies that no mandatory column has been added to an existing table without a default value associated to it because that would break any module that attempts to insert new rows into that table and is not aware of the new column; similarly, it checks for addition of constraints, changes in column names or types, etc. When it comes to the Application Dictionary, the tool verifies that no meta data has been removed as other modules could reference it. It also verifies that the signature of a procedures, functions and methods has not been altered in any way.

With the tool in place, the logical next move was to integrate into our Continuous Integration framework. This way, any possible change that breaks the public API is discovered as soon as possible and in most cases only a few minutes after the developer has committed the change. This early detection system allows us to resolve any inconsistency well before it affects our users.
You can monitor the periodic execution of the API Consistency Check here.

Because the API consistency check is a static verification of consistency, it can only cover the formal integrity of the API at a syntactic level but there is still a small possibility that the API has changed semantically. To cover this eventuality, we have developed another automated test, also integrated in the Continuous Integration framework, that verifies that installing a few representative modules continues to be successful. We are also considering extending this test to cover at every execution a few randomly selected modules among the ones published by our community and possibly to integrate their regression testing as well.
You can monitor the period execution of the Module Installation Check here.

Of course there are always exceptions and, in a few cases, we need to accept small breaches in the API consistency. This typically happens when the API definition is wrong in the first place and we need to tread off addressing functional issues with the API stability. For example, we recently had a case where the product allowed the definition of duplicate accounting periods for the same calendar; this could have resulted in product failures under those circumstances as the accounting engine does not know under which period to post a transaction and the fix was to add a new unique constraint on the period table. Formally, this is an API change but the chances of this change breaking other modules are very small compared to the risk of leaving the defect in, so we accepted the exception.

In general, these exceptions represent very low risk to our consumers.
Whenever they happen, however, we publish them to our community using so that module owners can review them and assess if they are affected and adapt their code accordingly. We do so using as many communication channels as we can to reach the broadest possible audience:

With public API defined and this infrastructure in place, we can guarantee that modules published for 2.50 will continue to work unaffected throughout the whole life cycle of the release.

Additionally, by discovering and publishing API changes we can also ensure that, as we start the development of a new major version, our community is fully appraised of changes in our application and module developers have a very accurate description of the changes that affect their extensions.

In other words, this allows us to maximize the preservation of the investment that our community makes in the Openbravo platform and ensure the health of our ecosystem.

Sunday, September 13, 2009

Status update for August 2009

I apologize for the delay (these status updates are due by the 10th of the month) but here is the latest installment of the status update series, for the month of August 2009. As always, you can see the consolidated updates on our wiki in a status update page.

Status Update for August 2009

  • Maintenance packs
    • Openbravo ERP 2.40 MP8 was released to professional subscription customers on August 20, 2009
    • Openbravo ERP 2.35 MP14 was released to professional subscription customers on August 10, 2009
  • Modules
    • The following new 2.50 compatible modules have been released:
      • CIF and NIF validator
      • Inter-company Documents
      • Tax Report Launcher
  • Core enhancements:
    • The following enhancements have been completed and released as part of 2.50 MP4 (Sep 2nd):
      • Support for Extension Points for Order and Invoice Post procedures (feature request 10257)
    • The following enhancements are available in the pi code repository and will be released as part of 2.50 MP6:
      • Support for logging in the Eclipse Console 10329)
      • Ability for a module to add accounting support for a new document as part the standard Accounting Process 10265)
      • Adding a format.xml.template 6162)
  • Defects
    • A total of 130 defects have been resolved, with the following breakdown by severity:
      • 9 critical
      • 68 major
      • 45 minor
      • 8 trivial
  • Infrastructure
    • Completed the stabilization of the ERP smoke tests in the Continuous Integration framework at builds.openbravo.com

Finally, we made progress in the following areas that have not been released during the month of August:

Monday, August 10, 2009

Status Update for July 2009

This is the latest installment of the status update series, which refers to the month of July 2009. You can see the consolidated updates on our wiki in a status update page.

Status Update for July 2009
  • Maintenance packs
    • Openbravo ERP 2.50 MP3 was released to the community on July 30, 2009
    • Openbravo ERP 2.40 MP7 was released to professional subscription customers on July 17, 2009
  • Modules
    • The following new 2.50 compatible modules have been released:
      • CIF and NIF valiation (Spanish Localization)
      • Tax Report Launcher
      • Inter-company documents
  • Core enhancements:
    • The following enhancements have been completed and released as part of 2.50 MP3:
      • Performance: optimize Product selector on oracle to be faster with many products (feature request 9562)
      • Performance: remove extra request for the MessagesJS in generated pages (feature request 9727)
      • Application Dictionary: single record UI pattern (feature request 4113)
      • Disallow to create client record in Client window (feature request 3437)
      • Heartbeat Configuration and Registration windows are edit only (feature request 486)
      • Accounting: Create new Sub-Account per BPartner or Product (feature request 9888)
      • Asset Management: Asset report for depreciation schedule (feature request 9887)
    • The following enhancements are available in the pi code repository and will be released as part of 2.50 MP4:
      • Accounting: Add C_DOC_TYPE_ID as a dimension to the FACT_ACCT table (feature request 10002)
      • Accounting: Ability to add additional processes as post actions to the Accounting Process (feature request 10017)
      • API Enhancement: Include getChildOrg() and getChildTree() methods in OrganizationStructureProvider class (feature request 10126)
  • Defects
    • A total of 277 defects have been resolved, with the following breakdown by severity:
      • 17 critical
      • 166 major
      • 85 minor
      • 9 trivial
  • Infrastructure
    • Worked on stabilizing the ERP smoke tests in the Continuous Integration framework at builds.openbravo.com

In addition, the Openbravo Community released the following new 2.50 compatible modules:

  • Chart of Accounts France
  • Taxes: configuration for France
  • Translation: French France (fr_FR)
  • CRM
  • Phidias Pack Base
  • Phidias Skin

Finally, we made progress in the following areas that have not yet been released:




Saturday, July 11, 2009

Status Update for June 2009

This is the latest installment of the status update series, which refers to the month of May 2009. You can see the consolidated updates on our wiki in a status update page.

Status update for June 2009
  • Maintenance packs
    • Openbravo ERP 2.50 MP2 was released to the community on June 30, 2009
    • Openbravo ERP 2.40 MP5 was released to professional subscription customers on June 16, 2009
    • Openbravo ERP 2.40 MP6 was released to professional subscription customers on June 19, 2009
    • Openbravo ERP 2.35 MP13 was released to professional subscription customers on June 10, 2009
    • Note: Openbravo ERP 2.50 Professional Subscription SMB Appliance released on June 15
  • Core enhancements:
    • The following enhancements have been completed and released as part of 2.50 MP2:
      • Ability to generate a provisional Balance Sheet report for open periods (feature request 9069)
      • Ability to search Windows, Tabs and Fields by module (feature request 9022)
      • Support for BLOB data types in DAL (feature request 9313)
      • The publicly usable java classes should be defined and all non-public ones should not be accessible by modules (feature request 9317)
      • Change tab name from 'Salary Category' to 'Cost Salary category' (feature request 9044)
      • Performance: Add infrastructure to VariablesBase class to allow for technical validation of request parameters (feature request 9500)
      • Performance: Use the new ignoreValues features of sqlc to speed up filters with value '%' (feature request 9549)
      • Performance: Ignore the value '%' in select which use like (feature request 9545)
    • The following enhancements are available in the pi code repository and will be released as part of 2.50 MP3:
      • Performance: optimize Product selector on oracle to be faster with many products (feature request 9562)
      • Performance: remove extra request for the MessagesJS in generated pages (feature request 9727)
  • Defects
    • A total of 130 defects have been resolved, with the following breakdown by severity:
      • 12 critical
      • 87 major
      • 27 minor
      • 4 trivial
  • Infrastructure

In addition, the Openbravo Community released the following new 2.50 compatible modules:

  • Fiscal Calendar Enhancements
  • Fiscal Calendar Enhancements Template
  • Report Sales by Month
  • Romanian chart of accounts
  • Romanian language for Romania
  • Romania Administrative Departments
  • Translation: Arabic_Saudi Arabia (ar_SA) for Openbravo 2.50

Finally, we made progress in the following areas that have not yet been released:

  • We also advanced in the definition of the next version of the Spanish Localization pack intended for the Community as well as the Spanish Professional Localization intended for Professional Subscription customer.

Sunday, June 7, 2009

Status Update for May 2009

This is the second installment of the status update series, which refers to the month of May 2009. You can see the consolidated updates on our wiki in a status update page.

Status Update for May 2009
  • Product releases
    • Openbravo POS 2.30 Community Edition was released in production status on May 20, 2009. This is a significant new release with extensive new capabilities; please see the Release Notes for details.
  • Maintenance packs
    • Openbravo ERP 2.50 MP1 was released to the community on May 15, 2009
    • Openbravo ERP 2.40 MP4 was released to professional subscription customers on May 26, 2009
    • Note: Openbravo SMB Network One appliance for 2.40 MP4 introduces an improved approach to apply maintenance packs.
  • New sample modules:
  • Core enhancements:
    • The following enhancements have been completed and released as part of 2.50 MP1:
      • Accounting templates (feature request 8467)
      • Ability to run jUnit test cases from the command line (feature request 8562)
      • Goods receipts Copy Lines window has 'order' label column instead of 'order line' (feature request 8529)
    • The following enhancements are available in the pi code repository and will be released as part of 2.50 MP2:
      • Ability to filter and group by bank account in the Payment Report (feature request 8162)
      • Display business partner in the Payment Report even when not grouping by business partner (feature request 8777)
      • Added PDF capabilities to the following reports (feature request 8474):
        • Bank Report
        • Cash Report
        • Invoice Tax Report
        • Expiration Date Report
        • Stock Report
        • Daily Work Requirements Report
        • Standard Costs Report
        • Production Run Status Report
        • Pending Work Requirements
      • Sqlc-generated data classes are not public unless explicitely declared as public (feature request 9230)
  • Defects
    • A total of 149 defects have been resolved, with the following breakdown by severity:
      • 19 critical
      • 79 major
      • 45 minor
      • 6 trivial
  • Infrastructure
    • Added the following automated tests to the Continuous Integration framework at builds.openbravo.com
      • Database consistency check
      • Module installation test
      • Module integrity test
  • Documentation
In addition, the Openbravo Community released the following new 2.50 compatible modules:
Finally, we made progress in the following areas that have not yet been released:

Tuesday, May 19, 2009

Extend, don't customize

Introduction
Every business is different and every company adopts different business practices. As a consequence, every ERP implementation is different and being able to adapt the software to the customer's specificities is a key characteristic of a good ERP package.

In this area, open source has a clear advantage compared to proprietary solutions thanks not only to the access to source code but also to the free availability of knowledge about the inner logic of the product.

Openbravo ERP, in particular, has been designed and managed considering the need for adaptation as a key requirement:
  • Its model driven architecture enables users to change standard functionality in a very easy manner.
  • The free flow of information available on the project wiki and forums have empowered the community to make changes to even the innermost core part of the product.
This openness and flexibility provides great power to implementor and allows them to customize the product to meet practically any business requirement.

However, as Ben Parker said, "with great power comes great responsibility" and in this case the great responsibility is to your own implementation as it is very easy to abuse this great customization power and change the product to the point where you can no longer afford to upgrade it nor benefit from the improvements made by the rest of the community.
If you are not very careful, it is very easy to find yourself in a position where you are either stuck on a product version that will eventually age and become obsolete or you have to bear the burden of maintaining and evolving the full solution instead of only your code. In both cases, you started with a packaged off the shelf application and you ended up with a bespoke piece of software.

It is important to notice that this danger exists both in open and closed source software. Countless literature has been published on the topic of customizations and upgrade nightmares and the prevailing wisdom is that, for mature ERP packages, customizations should be avoided at all costs. Many experienced implementors also recommend that a vanilla solution should be adopted for the first few months of production usage as many users discover that customizations initially thought essential are in fact not needed.

The problem of excessive customization, however, is much more pervasive in open source software because:
  • Often the open source solutions are more recent and innovative than their proprietary counter parts but not as mature; therefore, the pressure to customize is stronger.
  • Most importantly, the "hacking ethos" of the open source movement makes open source implementors culturally more disposed to implement invasive customizations.
Openbravo to the rescue!
Openbravo ERP 2.50 introduces a set of very powerful adaptation techniques that will allow you to personalize the software to meet your unique needs without the need to customize it.

This approach guarantees that you will be able to apply patches and maintenance packs without impacting your adaptations and minimizes the cost of upgrading from one major version to the next.

The core principle of these techniques is summarized in the title of this blog: as you design your adaptations, you should think of them as extensions and not customizations.

Openbravo ERP 2.50, in fact, introduces support for modularity, which allows you to define and package additional functionality and configurations as extension modules, independently from the core product.

One of the key features of modularity is that your configurations are also defined as extension modules. Because of that, they are kept separate from core and you can safely update the latter without touching you configuration.

Let's see how it works. To start your adaptation process you need to set the system in Configuration mode. You can do that by navigating, in System Administrator role, to General Setup / System Info and checking the box Customization Allowed (see picture below).


This will automatically create a template called "System Customization" and any configuration change or addition that you make will be added to this template.

In this template, you can safely modify the Openbravo core model (the meta-data that describes Openbravo ERP out of the box) and you can add your own model elements, such as new windows and processes. You can also add code-based artifacts such as new PL/SQL functions or Java classes and add or modify their corresponding calling points within the application.

What you can do is practically unlimited and, most imporantly, it is always update safe.

Examples
To illustrate this principle, I would like to consider a few concrete examples.

Let's start with something simple. Let's assume that your adaptation consists in modifying the default screen layout for the Sales Order screen, to change the order of the fields, hide some that are not needed in your business, and rename some labels. After having set the system in customization mode, you would proceed by performing your configuration changes in the same way you did in 2.35 and 2.40 by navigating to the Application Dictionary / Windows, Tabs and Fields, querying the definition of the Sales Order window and modifying its configuration there. Similar to previous releases, to see and test your changes you then need to rebuild the system.
What is new with 2.50 is that, once you are satisfied with your application, you can export your configuration script using the export.config.script ant task and that at this point your configuration is saved in the file system.
If at a later point in time you need to apply a core maintenance pack, you can do so safely because the pack will override the files for the core module but it will not touch your configuration file. Once the system is rebuilt, you configuration is considered once again and automatically applied to the updated core.

This is a much easier process compared to previous versions when your configuration changes were saved in the same files as core definitions and you therefore had to go through a very difficult and error prone merge process to apply the maintenance pack without loosing your adaptation.

Let's consider another example. In this case, we will assume that your adaptation consists in creating a new window. After having set your system in customization mode, you can define and declare the new tables that are needed to support your window and proceed by defining the window in the Application Dictionary. If you are not an experienced Openbravo developer, you can follow the tutorial in the Developer's Guide for step by step instructions.
The concept is very similar to the previous example and in this case as well your adaptation will be saved in a separate set of files and preserved when you apply maintenance.

A difference worth mentioning in this case is that you need to create your new window as a module separate from the "System Customization" template; this will give you more flexibility, including the option of reusing this window in another Openbravo instance without having to share the full configuration; you can even publishing it to the whole community leveraging the Central Repository.

This process can be extended directly to very similar examples, like adding a new report or a new background process. In both cases, you can adapt your system through extensions that are both easy to implement and maintenance safe.

At this point, you are probably thinking that this is quite good for configuration changes and additive extensions but you might be wondering how can you modify core. For example, does Openbravo let you add a column to a core table and a new field to the corresponding window?
Of course, it does. Just create a module and follow this tutorial. The concept is the same and, in this case as well, your changes will be preserved when you apply the next maintenance pack.

You want something very sophisticated? How about adapting the accounting logic to account asset depreciation in a different manner?
This is easy is easy to do. You need to define a module that contains a small Java class that implement your asset depreciation accounting logic and declares it in Openbravo as an accounting template

Impressive, isn't it?

So, what are some of the limitations? Well... they are not many, but the major limit is caused by the fact that the lowest level of modularization is the finest grained software artifact.
This is very fine grained in the case of Application Dictionary elements where you can adapt the software at the level of the individual property. However, it is less fine grained in the case of code, where you can adapt by logical file replacement: you add a module with the new file that substitutes the core one and you repoint the calling points as a configuration.

For example if you need to modify a core report, the lowest level of granularity is the jrxml file and you will need to create a module that includes a copy of the original core report and make your changes there; you then need to change the report declaration in Application Dictionary to invoke your modified report in lieu of the core one.
The consequence of this is that you have now taken the ownership of the whole report and that maintenance changes to the core file will not impact your adapted one.
Depending on the extent of your changes, this might or might not be an undesirable effect.

Similarly, if you want to modify a selector to, for example, add a column in the results table, you will need to create a copy of the full selector and maintaining it from that moment onwards.

In most cases, this is a small limitation and does not cause any significant issue. In a few circumstances, however, it does result in adaptation modules that are too large compared to the desired change in functionality. A typical example is a small change required in a database procedure like C_INVOICE_POST that will require you to own and maintain the whole procedure.

We have been uncovering cases like this working with the early adopters of release 2.50 and our experience has been that we can easily refactor these components so that there is always a clean way of plugging-in your extended logic without the need to take over the full component.

Because of that, I encourage you to contact us if you find yourself in that situation; in most cases, we can work with you and make small changes to core that will make your extension much easier. If you are a partner or a professional subscriber, you can request our assistance through your Channel Business Manager or through the Openbravo Support portal respectively; if you are a community user, you can contact us using the Help forum.

Do you still want to customize?
We strongly believe, and our experience so far confirms it, that using the techniques that I illustrated above, you can meet any adaptation need through extensions rather than customizations.

That said, since Openbravo ERP is an open source solution, you continue to be free to change its code. You need, however, to be conscious that this freedom comes with a high price tag in terms of maintainability.

The only legitimate reason to customize core is when you find a defect and rather than relying on a fix provided by Openbravo, you decide to implement the change yourself. In that situation, we would strongly encourage you to report the defect in the issue tracker and - if possible - to contribute your fix as a patch attachment to the issue. By contributing your fix, you can be sure that it will be included in the subsequent maintenance pack as well as in the next release; this way, when you update or upgrade your system, you will not have to neither merge the code nor reimplement the fix.

In the unlikely event that your really cannot meet your requirements through an extension, first I would encourage you to contact us before considering a customization: we might be able to make a small core change that makes your extension possible.
You should also consider whether the benefits of the customization are really worth the maintenance problems that they are going to cause. Perhaps, you can trade off that functionality in favor of ease of maintenance.

In the end, if you decide that you still want to customize core, you can follow the same approach that was recommended with 2.35 and 2.40: manage your customizations as source code and merge your development stream with our maintenance stream using a source control tool like Mercurial. Even in this case, Openbravo ERP 2.50 will result in a better experience than previous releases and the effort of resolving conflicts will be much lower because:
  1. A good portion of the customizations should be done through modules anyway
  2. The only core customizations are likely to be for code and not for meta-data and merging code is a lot easier than merging the XML based meta-data.
Above all, remember: extend, don't customize!

Friday, May 8, 2009

Status Update for April 2009

Starting from this month, I would like to introduce a practice of sending regular updates to the Community on the recent achievements and progress of the product development organization.
My goal is that, before the 10th of a month, I would publish a quick update of what was done in the previous month.

This is the first installment, which refers to the month of April 2009. I will also consolidate these updates on our wiki in a status update page. The idea is to provide a complementary page to the product road map, where the product road map gives a forward looking view of where the product is going and the Status Update a historical record of the road we traveled.
In this context, I also just updated the product road map to make sure it is current.

Status Update for April 2009