Discussion about this post

User's avatar
jrchips's avatar

"The people aboard do not know what all of it does. They cannot. The quantity of inherited machinery in any system that advanced exceeds what any individual or any team can hold, and this is not a deficiency in them, it is the definition of the thing they are operating. To be advanced is to be carrying more history than you can audit.

And given enough operations, the encounter happens. Some old assumption meets a condition it was never written for. Not because anyone was careless. Because the number of operations kept increasing and the assumption kept being carried forward and eventually the two intersect. That is not a risk. Over a long enough run it is a schedule."

Sounds like a description of every large scale AI application. Will anyone know, in detail, what the app does or how it does it? If not, it's an automatic recipe for human, legal and financial disaster.

Todd Humiston's avatar

There’s never enough time to do it right, but there’s always enough time to do it over.

10 more comments...

No posts

Ready for more?