Fernando Labastida: Technology, Austin, Latin America

I received a very nice email from Michael Kuhbock, the Founder and Chairman Emeritus of the Integration Consortium. He provided me with another definition of integration that he said allowed end-users at corpoorations to provide a better understanding, from a business-point-of-view,  to their business counterparts:

“Being a business man rather than a technical professional or engineer,” Kuhbock said,  ”I have found that one of the main reasons the industry  has alignment problems is due to innefective communication.”

Here’s the definition of integration from the Integration Consortium, courtesy of Michael Kuhbock:

Integration (in-te-gra-tion) - a combination of parts or objects that work well together.
Communication (com-mu-ni-ca-tion) - the exchange of information. Integration is the successful communication between data, applications, processes, people and enterprises.       

I really like this definition because it’s functional as opposed to technical.  Yes, technology has to be involved to make parts of  this happen, but at the end of the day integration is more about a way of operating a company or an organization rather than a cool technology that everybody has to have. A company can have excellent integration between applications, but if the people are not communicating  with each other or using the data effectively, then there’s no business value there.

Thanks, Michael!

The word “integration” is bandied about so frequently and in such a cavalier manner, it is assumed that everybody knows what integration is.  However, integration means a lot of things to a lot of people.

The Merriam-Webster online dictionary definition of integration says:

1: the act or process or an instance of integrating: as a: incorporation as equals into society or an organization of individuals of different groups (as races) b: coordination of mental processes into a normal effective personality or with the individual’s environment2 a: the operation of finding a function whose differential is known b: the operation of solving a differential equation

The first definition describes one of the great socio-political accomplishments of the latter part of the twentieth century, and the latter definition refers to a complex mathematical equation.  However, this doesn’t really work for our purposes.  A more useful definition from TechDictionary.com:

Putting diverse hardware and/or software components together to work as a system.

That’s a little bit better.  However, when we narrow the term down to data integration, and we get my favorite definition, one provided by Georgetown University on a glossary page on Data Warehousing:

Data Integration The movement of data between two co-existing systems. The interfacing of this data may occur once every hour, once a day, etc.

That’s it, it’s as simple as moving data between two co-existing systems. This can be done in four different ways:

Migration: moving data from a legacy application, with data stored in a legacy format such as Cobol or Isam, to a modern CRM or ERP application. This is usually a one-time movement of massive amounts of historical data.

ETL: Extract, Transform and Load of data. Extracting data from operational applications, such as CRM, ERP, accounting, manufacturing, or other systems into a database used for the sole purpose of manipulating the data and reporting on the data, usually for business analysis purposes. This is usally the movement of large amounts of data in a batch process, durng certain intervals such as hourly, nightly, weekly, etc.

Application Integration: The movement of data in small chunks between two business systems, such as from a CRM application to an ERP system, or from a manufacturing system to an accounting system.  This is usually done at an API level, as opposed to through the database or text interface as is more typical of the first two scenarios, and is usually event-based and real-time, or near real-time.

B2B: Integration of data between trading partners, such as wholesalers, manufacturers, retailers, transportation companies, etc.  This also is usually real-time or near real-time, and event based, and entails transforming EDI documents or documents in any other format, transported over the internet, ftp, EDI VANS, etc.

There are many tools out there that do one or the other of the above scenarios, and some even do all of them.  The best integration tools for a company usually depends on what their primary focus is, how much data is being moved, the required frequency and speed of data integration, among others. Integration tools are usually not a one-size fits all solution; however, there are some tools that are very flexible, or rather agile, that can be used for almost all data integration scenarios. Most companies, if they’ve been around for any length of time, usually have diverse integration needs, and so sometimes it makes more sense for them to acquire a tool that can span the breadth of integration scenarios so as to capitalize on a one-time investment in software as well as knowledge.

In a recent blog post on Intelligence Enterprise, Cindi Howson cites an article on USA Today that mentions some of the horrible consequences of lack of data quality, such as voters being kept off the voter roles.  All these national and global heartaches could be solved by a simple, cost-effective data profiling tool that could easily fit in the budget of some of our bloated government agencies!!!

Ok, I promise this will be my last post on the outlook for 2008!  David Lithicum of Intelligent Insight has listed his 5 predictions for Saas in 2008. Lo and behold he mentioned integration as # 2 on the list!  He said that integration between SaaS CRM solutions and back-end systems will be an afterthought for most companies and they will “…end up figuring things out on the fly.”

However, Peter Coffee over at Salesforce.com was kind enough to refer back to one of my original blog posts where I argued SaaS CRM applications are becoming de-facto data or SOA hubs (with a little help from data integration vendors).

Read on Salesforcetimes.com, CNN Money posted an article, originally in Investor’s Business Daily, where Mark Benioff, Salesforce.com’s illustrious CEO, in his typically modest fashion, claimed that 2008 will be the year for SaaS.

In the article they mention the “end of software” mantra that Salesforce.com has used since it’s inception; I’ve been hearing that from the Salesforce.com camp since I was working for a new defunct internet start-up back in 1999.  However, now I really believe it’s coming true. A large percentage of my integration business is from customers wanting to connect SaaS applications to traditional on-premise software.

Read the Benioff article here.

CIO Insight, in a recent posting in their “Future of IT” series, announced that 60 percent of those surveyed for their “Future of IT” survey have said they’ve adopted SOA and web services, and three quarters say it will be “the bedrock of their IT architecture.”  However, they are concerned that this last figure is 5%  below the survey results of January of last year.  One of the potential reasons they cite is the failure to meet the expectation that SOA will save companies money, among other things.

I believe that, to a certain extent, SOA has been over-hyped.  As a sales person I recognize that what I sell is not for everyone; SOA is not for everyone either.  Architecting an overarching SOA architecture, with an Enterprise Service Bus (ESB), and all enterprise applications connected to the ESB so that ad-hoc applications can be built “on-the-fly” from the functionality of these enterprise apps, is for large enterprises. And it’s not for every large enterprise either.  There are large companies with dozens of different geographical locations or departments where it doesn’t make sense for all applications to be connected under a SOA umbrella.

And gone are the days when large, 12-18 month enterprise projects are the way to go; the success of Salesforce.com vis-a-vis Siebel is ample evidence of that.  In my world, companies large and small contact me for tactical point-to-point integrations.  If their project works out well with my product, then they’ll roll it out as part of a larger product and buy more of my product.  And it works.  They didn’t have to marshal dozens of people and put together new teams and come up with new strategic IT and Business initiatives and get everybody aligned with the business and IT goals and disrupt what they were currently doing so that their companies could embark on these huge monolithic projects. A small team of go-getters just “got it done.”

So I’ll get off my soap box now, but I think that companies are starting to see that SOA is not for everyone.

It’s been too long since I’ve posted to this blog; end of year craziness is my excuse. And why, you ask, am I posting on Christmas eve? You’re probably thinking: what a bad family man, he’s working when he should be spending time with his family in harmonious communion. Well, I’ll have you know I was up real early this morning baking and I’m taking my rest time to write a little relaxing blog entry!

On to the top IT priorities for 2008. According to CIO Insight magazine, the number 1 and number 3 top priorities in terms of technology that will improve a company’s business strategy are business intelligence and data and application integration, at 44% and 29% respectively. Since business intelligence requires data integration tools to build the data warehouses that BI tools sit on top of, then two of the top five priorities regarding technology for business improvement for 2008 will require application integration tools.

According to SearchCIO.com, the top two priorities are enterprise software and business intelligence (again), respectively. Enterprise software implementations achieve minimal value if not integrated seamlessly with other enterprise application. For example, in order to provide a 360 degree view of the customer and to encourage usage of new Saas or on-demand CRM applications, these have to be integrated with order entry and accounting apps, again, requiring a data and application integration tools.

Ok, I’ll stop tooting the data integration horn and get back to baking and preparing brisket. And no, there’s not enough food for everybody!

“Get on the bus!”

I wanted to follow-up on my post about legacy migration from Nov. 30th to comment on some blog posts about ESBs (Enterprise Service Bus) and SOA (Service Oriented Architecture).

For a definition of SOA, I saw a great one on a recent post “SOA Essentials, Part 1,” by Jeff Davis,

“A Service Oriented Architecture, or SOA, represents an approach towards software development that emphasizes the creation of reusable software services that are based upon discrete units of business functionality.”

In other words, certain functions from applications like a CRM or ERP application are “componentized,” and can be put together to make distinct services that serve a particular function such as “create account,” and these services can then be put together to form an “orchestrated business process.”  These new business processes are essentially composite applications that can be created on-the-fly with the new web services technology available today.

An ESB is supposed to the be messaging layer in a SOA environment.  Messages from different applications are communicated, using WSDL and SOAP protocols, and transported in the ESB, or service bus.  However, Blogger Loraine Lawson from IT Business Edge, today references an article by fellow blogger Robin Bloor which essentially expands the mission of an ESB from that of being a SOA-enabler to being an “integration-on-demand” enabler.

ESBs, according to Bloor, after interviewing executives at ESB provider CapeClear Software, “…actually herald from the Enterprise Application Integration days. So, he contends, it’s really more than “just” a messaging software for SOA – it’s an integration engine or mediation engine…”

That still does not address the issue of making the functionality from legacy applications available as a service in a SOA environment. ESB vendors essentially need other integration software, such as Pervasive’s Data Integrator, to provide the “on-ramp to the bus,” or “last-mile connectivity,” on to the ESB. Our experience at Pervasive, with ESB provider Sonic Software, has been that older versions of ERP, accounting or CRM packages, or home-grown applications, don’t have web services APIs.  They need a tool that can connect natively to these applications or to the underlying databases and expose their data or functionality as a service first, before they can actually connect to the ESB.

So in the beautiful brave new world of web services and SOA, where every application uses XML and can communicate with every other application out there, legacy applications do not fit into this perfect picture.  However, it will rear it’s ugly head again and again, and as in my previous post, the majority of data in companies out there is still locked away in legacy applications.

With all the buzz and sexiness of SOA and web services, vendors have created an environment where customers feel inferior if they don’t have the latest modern application.  However, these applications are nothing without data. Data is the one constant at a company, whereas software has become a disposable item. At our company, we have gone through about four different CRM applications, but we have kept the data on our customers.

The majority of data, at companies large and small, exists in old hard-to-get-to file formats or technologies. If a company has been around for at least 20 years, they may have applications on mainframes, or built on older databases that don’t adhere to modern DBMS standards, or data that exists essentially as flat files, such as in COBOL formats.  But getting that data out of older systems and into new, modern systems that utilize these latest technologies is a very hard problem which software vendors typically gloss over.

If an ERP or CRM vendor tells you “migrating your data to our new app. is a cinch, no problem at all,” then you know somebody’s lying.

Most applications these days are built on modern databases that are self-contained wonders. They have things like tables, fields, relationships between these tables and fields, passwords, programming, indexes, metadata, all wrapped in one nice package.

Most legacy systems use older data storage technologies such as VSAM, ISAM or COBOL which have no indexes, metadata, nor relationships, just long rows of data, with rudimentary attempts to indicate what each section of data means, when they were written, etc.

Converting legacy data to data that is neatly placed in rows and columns and in a format that can be utilized by one of the new applications is not something one of the modern SOA vendors can do, and requires a traditional data integration or ETL (Extract Transform and Load) tool to perform.  It’s not sexy or modern, but it’s something countless organizations face every day.

So before going out and buying the latest SOA-compliant whiz-bang application, you need to really consider how you’re going to get all your valuable customer and product data from those old hard-to-get-to systems into your beautiful, shiny new application.

By Fernando Labastida

DM Review is a great source for articles on data and application integration. However, I have to take issue with a recent article on Application Integration for Midmarket Companies. The article starts out on the right foot, comparing enterprise-level ETL and EAI tools and custom coding, and poking holes in both. Enterprise-level ETL and EAI tools are like “using a chainsaw to open a letter,” and custom code is a very risky proposition: it monopolizes the time of the most skilled developers, lacks scalability and flexibility, and is not reusable, among other things.

However, the prescribed solution is all wrong. The article recommends an “integration appliance” as the answer to the midmarket integration challenge, asserting positive attributes such as low-TCO, no need for programming skills, fast delivery, and simpler operations. An integration appliance is a self-contained hardware/software combination that delivers “integration-in-a-box.”

However, customers and system integrators who have had experience with integration appliances point out the following drawbacks:

1. Simple integrations, such as direct mapping of data from one source to another, can be done without programming; however, when complex business rules have to be implemented, a more common scenario, appliances still require programming, often taking more time to implement than customers expect.
2. Appliances tend to be very costly. Up front costs are high and are not justified when the cost of their attendant software and hardware components are factored in; a proven integration application, coupled with a state-of-the-art server, properly configured by a system administrator, is more economical and provides more flexibility. Connecting additional end-points to the appliance, when complex business rules have to be implemented, cannot be done simply through drag-and-drop configuration, but requires programming, increasing the ongoing costs of ownership.
3. Meeting increasing performance demands with an appliance is complicated. Appliances are delivered with a fixed hardware configuration; optimizing the hardware for increasing data volume requires additional intervention by the appliance vendor. The latest application integration software, on the other hand, comes already equipped to utilize additional CPUs or cores if and when the client company decides to add horsepower, which it can do very cost effectively.

The best solution for midmarket application integration is a robust, agile, lightweight, low-TCO, integration software platform, that can either be delivered as a toolset with an easy-to-use yet powerful visual integration interface and extensive connectivity, or as a fully delivered, fixed-price integration solution.

Click here for a free trial of Pervasive’s new Data Integrator V.9, with the revolutionary new Data Mediation Services for creating your own connectors and adapters. After downloading the software, contact me so I can issue you a license key.