@akkartik excellent! this reminds me a bit of macroresilience on the at-first-blush sensible practice of having humans as emergency backstops to automated systems. a similar kind of creep from a simple but systemically illiterate narrative to pathological outcomes. https://www.macroresilience.com/2011/12/29/people-make-poor-monitors-for-computers/ https://www.macroresilience.com/2013/05/09/deskilling-and-the-cul-de-sac-of-near-perfect-automation/