I think the most irritating part of being a mathematician is when you are asked to do an arithmetic calculation. If you get it wrong or take too long, everyone starts questioning your career choice.
For God's sake, there are a thousand reasons why any mathematician could quit the profession and being slow at calculations is the least one of them!
@Breakfastisready perhaps that's due to the limitations and narrow scope of mathematical education that lay people have, which wakes them believe that mathematics is only about doing fast mental calculation...