Re-Invigorating Old Data

How do DevOps teams lever maximum value from their aging data assets?

The American statistician W. Edwards Deming is quoted to have said ‘Without data, you’re just another person with an opinion’.  Data matters and it’s central to digital transformation.  The problem is no matter how much companies invest in their data, and however they capture it, the quality will always be suspect.  Without data, any ideas of digital transformation is likely to be a pipe dream.

Most companies hoard gigabytes of data on their finances, their products, their customers and markets.  The difficulty is that almost no enterprise has their data organized in a structure that makes it easy to access.  As soon as business leaders come up with ideas for business model reinvention, probably the next thought in the minds of DevOps leaders is ‘Where is the data coming from?’

Digital transformation projects have a habit of either generating new data (as in the case of sensor network centric projects) or reusing old data (such as plotting assets or customers on a map and gaining value from location-centric perspectives), or a blend of the two.  Re-using data found within the enterprise can be challenging because of data quality issues, the variations of data structures and field formats between applications and issues getting data out of systems.  This means DevOps teams need to have very good data management skills.

So how do DevOps teams approach their data challenges?  Here are some examples of how DevOps teams are reinventing their old data with Encanvas.

1. Mashing data

Old data may be held in various applications and formats.  It’s not uncommon for Encanvas to gather information from spreadsheets, big back-office systems databases like SAP R3, IBM DB2, Microsoft Dynamics and SQL – all at the same time.  Encanvas is a plug-and-play multi-threaded and multi-sourcing platform which means designers can create concurrent live data feeds from multiple systems or endpoints at the same time.  This capability is used extensively by designers when creating applications that reuse data from existing and new systems together, creating new data structures on the fly for the specific canvases they author as part of applications under development.

2. Special filters

Your old data may require filtering to select only the records relevant to your project.  A powerful feature built into Encanvas’ mash-up environment is our special filter which allows designers to employ drag and drop controls to instantly create very powerful data filtering on inbound data from third party sources.  Any number of filters can be applied to tables at the same time.  For example, if a designer wants to only ingest data from a customer table of a specific type, and that relates to a specific region, they can create special filters for ‘types’ and ‘regions’ selecting only the records that apply to those conditions.  All of this rich configuration is done without any coding and doesn’t influence the integrity of the ingested table, or its potential for reuse in its native form by other applications (or canvases).

 3. Enriching or validating data with third-source data

If your old data can benefit from being enriched by other sources of data, Encanvas’ mash-up capabilities can really bring value by making the internal and external data accessible to applications designers without having to use coding or API to build new integrations.  For example, back in 2002, we helped a client to create an Encanvas application that would integrate Lotus 123 customer data with a third party industry database to enrich customer records so that the client company could produce refined searches of their customers using fine-grained drop-down filters – using data that didn’t exist in their database!

4. Cleansing and transforming data

Sometimes old data requires cleansing at the point of transfer from its original location using a machine to machine cleansing and transforming process to shed unwanted data and apply transformation rules to re-order, de-dupe and relocate data to new data structures.  Encanvas Information Flow Designer (IFD) is the machine-to-machine software module built into our architecture that equips designers with the means to configure these ETL actions and normalize data before it gets ingested into applications.  IFD also automates the generation of notices to alert designers (and users too if necessary) that transformations have worked – or not.  Transformations can be triggered by events, scheduled times, watch folder changes and a variety of other means.

 5. Quarantining data

A powerful (and pretty unique) feature of Encanvas lies in its ability to create quarantining protocols for old data that fails to live up to your expectations for data integrity.  There’s little point uploading records that are unfit for purpose.  If you are gathering customer records for example and would determine that records that fail to have any contact email, telephone or mobile numbers included are not suitable for use, then designers can create quarantining rules that filter this data out for special treatment.  In such cases, the data remains ‘in the system’ but is no longer visible to users until it has been manually or machine cleaned.

6. Applying voting systems to ingested data sources and end-points

It may be that old data is being ingested from multiple systems or endpoints and you need to create a new data mart that has to prioritize the best likely source of good quality data over others.  This can get really complicated because different systems may create new data at different speeds and this can create latency issues but, nevertheless, Encanvas has the code less tooling to enable designers to author voting systems to vote on which source is most trusted. Voting systems can use algorithms to automatically test data integrity and then automatically augment the voting structure, or they can be manual, where the data owner or manager uses a sliding scale of trust levels to determine which source is proving to generate the best results (or both!).

7. Creating new data

When there are gaps in your old data, there are many ways that Encanvas can create new data as part of its application design.  For example, the numeric controls of Encanvas allow designers to create formulas and calculations on data to total columns, sum value, source averages etc. that may be required for your new dashboards and reports but do not exist in the ingested data. Encanvas also has the ability to ingest SQL script and DLLs to make it easy for DevOps teams to reuse existing code blocks or create new APIs and transformations.

 8. Location-centricity of data

Another way to create new data is by using Encanvas’ mapping capabilities to apply location-data to existing addresses and locations.  Encanvas has an integrated – and codeless – mapping engine (sometimes referred to as Geo Spatial Intelligence, or ‘GIS’).  It allows designers to plot and pin records on maps.  The geo-data of records is added to the data-set (companies like Google and Microsoft charge lots of money to do this!).

9.  IoT API

Parachute-in a high profile technology-centric team with a strong leader into an organization with an existing IT department it’s hardly surprising that you’re going to have to put out some fires and smooth over a few ruffles.

Balancing two-speed IT means having an internal IT teams focused on reducing costs and improving process efficiencies through Business Transformation (BX) and a DevOps team re-inventing business models through Digital Transformation (DX) in tandem.  Recognizing each team for its own skills and contributions to business outcomes and balancing praise is going to be important for a healthy culture.

10. Building a wholly new data structure

We’ve saved the most dramatic way of fixing old data quality issues until last – because it’s no small project to build a new data warehouse to gather and reorganize data into new structures but sometimes it’s the most sustainable way to ensure that data integrity is preserved for the life of your application.  For mission-critical processes, it’s probably the best quality outcome although the time and investment needed to create a data warehouse or enterprise data-hub is definitely ‘none trivial’.  Encanvas includes all of the codeless tooling needed to fast-track the creation of new data warehouses and data marts using the data repository of your choice – whether you are moving towards a big data solution like Hadoop or are seeking a more traditional data structure like SQL or DB2.

So there you have it – ten new ways to turn your old data into useful, data for your next digital transformation. To find out more about the capabilities of Encanvas DX, please contact our team.

10 Big Challenges

A summary of the top challenges leaders face when implementing DX projects.

Encanvas and its partners around the world have been implementing digital transformation projects for over a decade.  In that time we’ve seen as many projects successes as failures. It isn’t easy to make transformation projects a success – and rarely is a technology the problem, though its shortcomings often show through.  In this article, we re-visit the learning lessons of a decade of real-world experiences in this summary of the 10 most painful learning lessons.

  1. Measuring success before the first line of code is produced – It’s a tricky issue, but how do you determine the return-on-investment (RoI) on software developments when you haven’t created them yet?  Without a clear RoI it can be an uphill struggle for business managers to gain support for inventive projects so being able to quickly form an appreciation on the likely return can be a make or break issue.  Many projects will stall even before they get started because business managers with big ideas will lack the evidence to prove their worth.
  2. Balancing the rewards of contributors/beneficiaries – When projects are set to change business models the likelihood is that more than one business area will be involved.  An awkward dynamic occurs when one business area needs to put proportionately more of the time, effort and investment into a business transformation initiative when another part of the business is set to reap proportionately more of the rewards. This can create, at best, a need for very sound project management and strong leadership and, at worst, can compromise the success of a project.
  3. Overcoming the corrupting impact of cultural/data silos – The corrupting influence of silos of operation and silos of data is never too far beneath the surface when it comes to change projects. The impact of silos is best described by Dr Eli Goldratt’s Theory of Constraints which states the core constraint of virtually every organization is that organizations are structured, measured and managed in parts, rather than as a whole. This results in lower-than-expected performance, difficulties securing or maintaining a strategic advantage in the marketplace, financial hardships, seemingly constant fire-fighting, customer service expectations were rarely met, the constraint constantly shifting from one place to another and chronic conflicts between people representing different parts of the organization.  When leaders put their energies into creatively re-inventing business models, tackling data silos is something that needs to be dealt with head-on.
  4. Data integrity–when data is reused, it’s often sub-optimal – You may think your financial accountingcustomer relationship management or enterprise resource planning software is brimming with useful data that’s ready to be reused for the new uses you’ve in mind for it. But experience says you’ll probably be disappointed.  When digital transformation teams get to work building new ways of making data useful, more often than not the quality and integrity of data comes up short.  Overcoming this problem could be as simple as a transform activity, or referencing a third party source of ‘good quality data’ to enrich the existing data entities, though the worst case can be nothing less than life threatening to projects.  Sometimes, digital transformation projects are halted in their tracks by poor quality data that puts the desired project outcome out of reach because of time, the investment required or technical complexity.
  5. Skills–DX ideology is at odds with conventional IT mindset – List out the technical skills-sets required for your digital transformations such as project management, business intelligence, programming, mapping, analytics, data modelling (etc.) and it might sound like a familiar list.  You’d maybe think, with a little stretching, your existing teams have the necessary skills needed to meet these new demands.  Okay, the tools and technologies are new, but how different can it be… right? Wrong!  Digital transformation projects are characteristically at odds with traditional IT methods, tools and approaches. You need people that ‘think and act differently’; that want to take risks and try things out; that have more of a knowledge and interest in how business, organizations and processes work; that are outcomes focused; people that want to talk to users and stakeholders because they find it rewarding.  Almost all the activities that behind-the-desk coders hate doing will be cat-nip to digital transformers.  The good news is that you probably will have much of the talent in your business that you need; it’s just not where you’d expect it to be.  Bear in mind that you will need fewer coders and more soft skills, planners and creative thinkers for what you’re about to do and you might well find these competencies in project management and analyst teams, in finance, marketing, R&D or HR departments that are more accustomed to change programmes and cross-organizational projects focused towards business outcomes.
  6. Adopting tools/methods to make change affordable – Organizations that try to use their existing development approaches and tools to fulfil digital transformation projects are going to come unstuck with the economic challenges they face.   Simply put, the need for rapid time to market, fast integration and early stage quick-win results means that the old ways just don’t cut it.  Even the activity of programming has to be questioned in a rapid, agile development environment.   It creates a divide between technology people that ‘get it’ and the mass of users and stakeholders that are instantly turned off when they see a line of code.  Digital transformation projects are an awkward mix of needing to get things done quickly whilst not compromising on the end result. The resulting applications must still meet in full the enterprise IT hygiene factors that businesses expect.
  7. Discovery/displacement of incumbent shadow systems – Sometimes developments of innovative solutions can uncover those systems that small teams and individuals have built themselves by necessity to overcome the business challenges they have encountered. Many of these systems are not on the radar of IT or those individuals driving new spotlight digital transformation projects.  It’s not surprising that those people that have invested time in these shadow systems feel somewhat aggrieved when their work is discounted for a trophy project with not even a thank you. Project teams can trample through these shadow systems unknowingly building up a storm of resentment and positive disengagement.
  8. Over-engineering of project scope and applications – It’s easy to try to deliver the best possible solution when ‘good’ will do.  Sometimes, delivering a simple, adequate system is the best first outcome.  Then you can develop more features and capabilities to bring more value to stakeholders.  As projects progress it’s not uncommon for project teams, users and stakeholders to learn from first-build model solutions and come up with better ways of solving problems, new features and capabilities.  This type of scope-creep can lead to short-term objectives being missed eventually resulting in a loss of faith and project failure.
  9. Inter-departmental bias and BX versus DX conflicts – Parachute-in a high profile technology-centric team with a strong leader into an organization with an existing IT department it’s hardly surprising that you’re going to have to put out some fires and smooth over a few ruffles.  Balancing two-speed IT means having an internal IT teams focused on reducing costs and improving process efficiencies through Business Transformation (BX) and a DevOps team re-inventing business models through Digital Transformation (DX) in tandem.  Recognizing each team for its own skills and contributions to business outcomes and balancing praise is going to be important for a healthy culture.
  10. Denial of the need for a bimodal approach – Many digital resistors will have internal forces arguing the case for the status quo, normally with the argument, ‘We’ve worked like this for years and we’ve never needed another team’. Denial of the need for a bimodal approach can often be supplemented by a lack of understanding in management tiers of the possibilities of digital technologies and how they are changing our world. This makes it all too easy for incumbent disruptors to underplay the impact of digital technologies and the need for change.

And that’s our top 10! They aren’t in order. The level of potential impact will vary according to the type of project.  On publishing this list we do hope it doesn’t put you off. There are far more project successes than failures; and you are ever more likely to be succeed if you manage these challenges.

Are you a digital transformer?

According to McKinsey&Co, by 2025, digitization is expected to contribute $2 trillion to US GDP—and those on the digital frontier have 2-3X faster profit margin growth. Digital leaders maintain an enormous lead over everyone else. IDC predicts that by the end of 2016, two-thirds of CEOs of the largest 500 European enterprises will have digital transformation (DX) at the heart of their corporate strategy.

It would be easy to view ‘digital transformation’ as just another hype curve promoted by the IT industry to sell more tech, but that would be a miss-reading of the current dynamics of the business world.

It may be that key technologies such as 4G, cloud computing, the Internet of Things and big data have only reached a point of maturity that makes them useful but that’s only one side of the page that makes digital transformation a big story.

The other side of the page is left to topics like the shortfall in talent, growth of e-Commerce, social networks, consumerization of IT, globalization of markets, the broadening of markets like Europe, the rapid growth of BRICS economies and the increasingly robust regulatory environment that organizations find themselves having to deal with.

All of these factors are contributing to the need of organizations to make smarter decisions, adapt to change faster, harness new ideas more often and be prepared to shift between traditional market segments – just as competitors and rivals are doing.  You could argue that digital technologies are a timely agent of change servicing the challenges of a maturing digital economy rather than the instigator of change or unwanted guest forcing him or herself into the room without invitation.

What is the digital maturity of your organization?  

Market analyst firm IDC breaks down the transformation journey into five levels of maturity:

  1. The Digital Resister – These organizations are responding to ad hoc requests but are yet to grow a positive bias towards digital transformation business projects.  It is unlikely that such enterprises have built the organizational capability to repeatedly embed new digital technologies into processes to engender new business models and achieve step change transformations.  Neither is such an enterprise likely to have appointed dedicated management to oversee digital projects.
  2. The Digital Explorer – This category of organizations will be seeking opportunities where technologies and business models can bring competitive advantage as and when business units bring forward suggestions they want to champion.  Inevitably such demands have to compete against all of the other IT budget and resourcing pressures which make these initiatives difficult to progress.  A lack of organizational modelling towards a sustainable capability will mean that many of the critical capabilities needed to underpin digital transformation projects – like strong leadership, a digital culture committed to ideas innovation, a cross-organizational development team and simple things like a thoughtfully designed master data management and data integrity agenda – are not in place and are likely to compromise results.  This phase of transition is perhaps the most worrying for business leaders who will be depending on quick-wins and satisfactory results to build momentum towards a digital transformation agenda and yet the poor talent capability, departmental conflicts of interest, unfit for purpose IT architectures and tooling together with a blend of other factors will be unwittingly working against the successful fulfilment of these projects.
  1. The Digital Player – These organizations will be on the road to success; repeatedly using digital transformation methods and tools to fulfil projects that are making a real difference to the efficiency and effectiveness of their business models.  Digital Players will be making progress in establishing the skill-sets and tools required for sustainable business transformation and are most likely to have leadership in place.
  1. The Digital Transformer – Is an organization that is transforming: Applying digital transformation projects as an embedded part of day-to-day operating behaviour.  All of the necessary hygiene factors for digital transformation will be established; a strong and dedicated leadership, a development operations team, a digital culture that is always looking to lever advantage through the application of digital technologies, key IT services such as a coherent data management and integrity plan, well-designed cloud technology architecture etc.
  1. The Digital Adoptor – These organizations have transformed both their business models and their ability to apply them through the thoughtful application of digital technologies.  They have embedded IT and digital processes into key aspects of business models. According to McKinsey, these firms will be ahead of the game in their respective markets. They suggest those companies on the digital frontier have 2-3X faster profit margin growth.

So – Are you a digital transformer?


Encanvas Guide for Executives

It’s never stopped surprising me how much knowledge there is for executives to learn and apply when it comes to technology these days.  It’s difficult to keep up with all of the new tech – big data, cloud computer, the Internet of Things – even when it’s your day job. Making sense of where Encanvas

We know making sense of where Encanvas fits in this sea of technology can be confusing.  So we’ve authored this easy to read guide to shed some light on our technology and why it exists.

Read it here: Executive Guide

What’s in the box?

You’re probably wondering why we’ve spent over a decade perfecting a piece of software that lets you create apps without coding so you can afford to throw them away.  Sounds ridiculous when you write it down doesn’t it?

And yet…

We started on the Encanvas journey after being frustrated time and again as consultants by the slow-pace of IT development in organizations.  Businesses cannot reasonably hope to be agile when they have inflexible IT systems.

We realised that the concept of developing software in back-offices and building in errors with manual coding would never be good enough for the future of enterprise computing.  What was needed to create best-fit applications quickly was an approach that meant Users and stakeholders could work with designers to author solutions in near real-time.

Translating a vision like that into reality ‘that works’ isn’t so easy.  The truth of the matter is that data is a fiddly creature and it’s difficult to tame.  Even gathering data from legacy systems can be a nightmare.  Lots of things have changed since we began with Encanvas development in 2002.  For one thing, we started designing Encanvas on a desktop because that’s what you did back then.  The new idea of ASP.NET was just being introduced by Microsoft.  And nobody had heard of cloud computing, big data or the Internet of Things!

Well here we are – 15-years later and somewhat older and wiser.   Encanvas mow supports big data repositories, JSON integration, IoT device monitoring capabilities – and is one of the most advanced cloud platforms on the planet.

So what’s in the box?  It’s a code-less way of creating apps without coding so you can afford to throw them away.

Some things never change.