Processing...

 Real-Time Data Processing, Enrichment and Analysis

Your data in as it happens, enriched and analysed, and out to your systems automatically

Our clients' systems send us their data as it happens: sales from an online shop, activity on a website, leads, orders and customer records. Automated processes on our data platform enrich it, cross-reference it and analyse it, then pass the results straight on to the systems that need them, or build them up into reports and analysis that arrive on a schedule.

It runs on our Serenity data cluster, in our own UK data centres, including the machine learning systems tied into it. Your data stays in the UK and in our hands throughout, and we act as your processor under the Framework Agreement.

  • "Our sales should reach the reports already enriched"
  • "New leads need scoring and routing the moment they arrive"
  • "Orders have to reach the fulfilment partner without anyone touching them"
  • "We want a monthly view of who is buying and what is trending"
Real timeData in from many clients and systems, as it happens
AutomatedProcesses run and deliver without anybody handling a file
4 minFor a 38 million record batch job, when batch is what you need
UKProcessed in our own data centres, never offshore

What the Platform Does

This began as batch work: a client supplied a file, we matched and enriched it, and a file went back. It still does that, but most of what runs on the platform now is continuous. Data arrives from many clients and many systems, automated processes act on it as it lands, and the results flow on to other automated systems or accumulate into periodic reports and analysis. Each client's processes are built around what they need to know or do.

Sales and insight

Sales data enriched with who bought, where and in what circumstances, and turned into insight on customers, products and trends.

Leads and outreach

Leads processed as they arrive, enriched, checked and routed, and outreach lists selected and cleaned before they are used.

Fulfilment and provisioning

Orders and requests passed on to fulfilment and distribution partners, or used to provision services, without anyone re-keying them.

Web usage and traffic

Website and service usage data collected, analysed and fed back, including into traffic shaping decisions.

Enrichment and cross-referencing

Your records matched against large datasets and against each other, so every system works from the same, fuller picture.

Archiving and analysis

Data archived for as long as you need it, and analysed, from simple aggregation to complex modelling with machine learning systems tied into the platform.

How It Works

1. Data arrives

Your systems push data to the platform as it is created, by API or automated feed, or drop files by SFTP or WebDAV where that is all a system can do. Many sources can feed the same process.

2. Automation acts on it

Processes built for you run as each record lands: matching, enrichment, cross-referencing, rules that decide what happens next, and the models that score or classify it.

3. Results flow on

Enriched data is available to your own systems by API straight away, or is pushed on automatically to the next system, whether that is a CRM, a reporting tool, a fulfilment partner or a provisioning platform.

4. It accumulates

Everything processed can be kept, archived and analysed over time, producing reports and analysis on whatever schedule you need, from daily figures to periodic deep analysis.

Two Ways It Is Used

Continuous

An online retailer pushes every sale to the platform as it happens. Each one is enriched with the demographic profile of the customer's area and cross-referenced against the retailer's own history, and the trends across all of them are worked out as they build. The retailer's reporting systems read the enriched data back by API, so the reports show not just what sold, but to whom and what is rising, without anybody exporting or importing a thing.

Batch

A client wanted to understand who was buying from them and how satisfied those customers were. We processed 38 million sales transaction records against 50 million UK census records and linked in 33 thousand customer satisfaction surveys, so each purchase could be read against the profile of the area it came from and against what customers had actually said. The results came back in four minutes.

Your Data, Protected

Customer data is personal data, and sending it to somebody else for processing is a decision your data protection officer will want to see properly documented. You remain the controller. We process your data as a processor, on your instructions, under the processing schedule of our Framework Agreement, which sets out the subject matter, nature and purpose of the processing, how the data is secured, and what happens to it when the work is done.

The processing happens on our own hardware, in our own UK data centres, run by our own engineers. Your data stays in the UK, which keeps the question of where it went, and who could see it, simple to answer.

What that means in practice

  • You stay the controller, GEN act as your processor
  • Processing terms set out in the Framework Agreement
  • Data held and processed only in the UK
  • GEN's own hardware and in-house UK engineers
  • Retention governed by our published Data Retention Policy

What the Enrichment Draws On

Processing can draw on any combination of the sources below, alongside any further data you supply yourself, such as your own survey results or another system's records.

The same sources are available to developers as APIs, authorised with keys and with full request tracking, for use directly in your own applications. See Software Backend for those. For the background to the platform, read What Is Big Data?

Data sources

  • Census: income, population, ethnicity, employment and more
  • House Price Index: average and median prices by postcode
  • Affluence: income brackets and median income
  • Postcode: location, ward and local authority
  • Broadband availability from Openreach, by address
  • Mobile coverage by network, indoor and outdoor
  • Telephone number provider lookups
  • Validation of phone numbers, postcodes and email addresses

Data Platform Questions

Does our data have to arrive as a file?

No. Most data now reaches the platform continuously, pushed from our clients' own systems as it is created: sales from an ecommerce platform, events from a website, leads from a form, records from a CRM. Files still work, by SFTP or WebDAV, for batch jobs and for systems that can only export.

How do we get the results?

However your systems need them. Enriched records can be read back by API as soon as they are processed, pushed on automatically to another system such as a CRM, a fulfilment partner or a provisioning platform, or accumulated into periodic reports and analysis.

What kind of processing can the platform do?

Enrichment against large datasets, cross-referencing between your own sources, rules that route or act on each record, archiving, and analysis ranging from simple aggregation to complex modelling with machine learning systems tied into the platform. Each client's processes are built around what they need to know or do.

Where is our data processed?

On GEN's Serenity data cluster, in GEN's own UK data centres, run by GEN's own engineers, including the machine learning systems. Your data stays in the UK throughout.

Who is responsible for the data under data protection law?

You remain the controller of your data. GEN process it on your instructions as a processor, under the processing schedule of the GEN Framework Agreement, which sets out the subject matter, nature and purpose of the processing, how the data is secured, and what happens to it when the work ends.

Can you still run a one-off job?

Yes. Batch work is still part of the platform: one job processed 38 million sales records against 50 million census records and 33 thousand customer surveys, and returned the results in four minutes.

Can we use the data sources directly in our own application?

Yes. The same sources used for enrichment, including census, house price, affluence, postcode, broadband availability, mobile coverage, telephone number and validation data, are available to developers as APIs, authorised with keys and with full request tracking.

Start With What You Need to Know

The useful projects begin with something you want to know or do, not with a dataset: reports that arrive already enriched, leads handled the moment they land, orders that reach a partner without anyone touching them. Tell us what that is and which systems hold the data, and we will tell you what the platform can do with it and how the data would be handled.