When to Automate and When to Keep Humans in the Loop