Event Data Modeling at Scale

As the big data train at your company picks up steam you suddenly find yourself needing to process more and more data - and everyone wants it in real time. This talk describes the architecture of a generic system that can be used in a variety of applications. It assumes an input of generic events and the output is a model of those event generators (e.g. customer’s purchase behavior, media engagement, transportation patterns).

We will review the different technologies used in such a system, and how they fit together including: NoSQL databases, caching, queuing, and distributed computing. The talk will also touch upon risks and scaling issues when running this type of system.

Educated at MIT (with degrees in physics, EE/CS, and a masters in cryptography) Mark has spent his career growing and fixing new ventures at startups, Fortune 500s, and academia. Past products have included a wireless application platform, online advertising, lead generation, digital media, OLAP, and a new software language. Mark was instrumental in launching ServiceLive.com, Sears' online home services labor market; he also helped fix NBC's online video marketplace (now Hulu.com). In academia Mark spent a year at Harvard Business School working with two finance professors to create the upTick system now used to teach finance at many of the top business schools. At MIT Mark helped to start the Undergraduate Practice Opportunities Program at which he teaches annually along with his teaching at SUNY’s Levin Institute. In his spare time Mark works with numerous non-profits and is on the advisory board of the Museum of Mathematics. He is also a former US ballroom champion.