Big Data System Development: An Embedded Case Study with a Global Outsourcing Firm

Big data system development is dramatically different from small (traditional, structured) data system development. At the end of 2014, big data deployment is still scarce and failures abound. Outsourcing has become a main strategy for many enterprises. We therefore selected an outsourcing company who has successfully deployed big data projects for our study. Our research results from analyzing 10 outsourced big data projects provide a glimpse into early adopters of big data, illuminates the challenges for system development that stem from the 5Vs of big data and crystallizes the importance of architecture design choices and technology selection. We followed a collaborative practice research (CPR) method to develop and validate a new method, called BDD. BDD is the first attempt to systematically combine architecture design with data modeling approaches to address big data system development challenges. The use of reference architectures and a technology catalog are advancements to architecture design methods and are proving to be well-suited for big data system architecture design and system development.