Video game graphics have become hyperrealistic over the past four decades, with companies pouring money into the technology. But in recent years, the upgrades have had diminishing returns. The industry is now wondering whether a longtime marketing tool is instead a financial liability.