Fracking the Data Lake

Turning information byproduct into jet fuel

By Richard Leader, 2 March 2017

Glance across a trading floor and you’ll spot there’s no shortage of data – it’s a sea of screens.

Risk-takers and revenue-generators, the rationale goes, need at-a-glance access to data to make informed decisions, fast – hence screens are (still) a proxy for importance, at least as far as hands-on producers are concerned. Credit salespeople often have 4; traders 6 or sometimes 9.

Yet there’s a snag: the human brain is only capable of processing and filtering so much data – which nowadays is generated in ever greater quantities. There are only so many Bloomberg chats and Launchpad views an individual can effectively monitor before the human operator is overloaded.

Those with the sharpest IT skills get around this by developing their own sophisticated Excel workbooks – fed by APIs – to automate some of the eyeballing and use formulas to generate alerts. Yet while this is practical for structured data (e.g. prices), for unstructured data (such as chats, or regulatory filings) the learning curve steepens sharply and those with the requisite ninja skills are few and far between.

Furthermore, the most valuable (usually proprietary) data is usually hidden away in inaccessible data lakes. Liberating this data via innovation-friendly APIs should be a top priority.

Yet even with a motivated cadre of front office developers, many firms still (unwittingly) throw barriers in the way of data innovation. In many cases the IT environment itself – desktop especially – is hostile to innovation, with machines excessively locked down (unthinkable at the likes of Google) and the approval process for new tools (even open source) slow and cumbersome.

Overcoming this – and unleashing the innovative potential of front office technologists – requires effort at every level of management. Risks must be balanced with the do-or-die need for innovative agility. Firms that can successfully achieve this difficult balancing act will be rewarded. Those that cannot will see their competitive advantage erode.

Finally – waste not want not: information byproduct can be extremely valuable. Equity IOIs (indications of interest), for example, contain real time indication of intent – which, if analyzed and cross-referenced against house data (such as holdings, and historical trades/IOIs) can yield actionable insight. The same applies to bond axes and FX RFQs (requests for quote).

By doing so, market makers can establish relationships and correlations – which can in turn be fed back into processes (automated and manual) such as pricing or sales – and the precious insight can be monetized.

In this sense, the data lakes really do contain untapped reserves, ready for fracking – and information byproduct can become jet fuel. Yet to achieve this, firms first need to embrace the potential of predictive analytics – and liberate the energies of those in the front office with the skills and wherewithal to make innovation of this sort happen.

And (usually) that starts with liberating the IT environment – thus allowing creative problem-solvers to experiment.