As we leave the Petascale milestone behind us, the computing industry is changing rapidly to address the next challenges. Energy, dependability, cost pressure and economy of scale are pushing IT consolidation into large "cloud" datacenters where new workloads and legacy applications coexist. Data is growing at a higher exponential rate than computing, and novel data-centric architectures are starting to emerge. Heterogeneity and specialization, within and across instruction sets, are reemerging to address energy efficiency. In light of these secular shifts in the IT industry, the benchmarking and simulation techniques are lagging behind and need to deeply transform to address the upcoming challenges. Like the ancient Hindu parable, current practices in the architecture community suffer from an excessive focus on narrow metrics, none of which is either completely correct or totally wrong, but often risk missing the big picture. This talk will discuss how computer architecture simulation and benchmarking must evolve to provide better quality decision support data for datacenter-level computing. Speed, full-system, validation and modularity are some of the fundamental characteristics of a scalable simulator. Dynamically trading off speed and accuracy, running unmodified software, and the flexibility to interface with multiple tools are other key aspects that should drive the development of the next generation simulators. As a case study, the talk will cover some of the design considerations behind COTSon, an open-source scalable full-system simulation infrastructure targeting fast and accurate evaluation of current and future computing systems.