I've been on and off Vyvanse and Ritalin for a decade. Work with your doctor on side effects, I found Concerta to have the least side effects for me (long-lasting, tapers slowly at end of day, and fewer anxious feelings for me than Vyvanse.) I've also taken breaks from the medication for a year or more. It takes somewhere between 3 days and 3 weeks to feel like I'm not missing something when I stop the medication, but overall it's not that bothersome or unpleasant unless work/life is very busy at that time. Quitting coffee cold-turkey feels much worse, for comparison.
I had horrible physicals side effects with Adderall, I was switch to Desoxyn and I get no physical side effects from it. I just get clear headed with no speedy feel in the body or cramps. Adderall always made me clear headed, but also jittery in the body and gave me the worst cramps but the biggest one that I was not going to put up with is Adderall killed the marital bed performance, but funny enough not the desire, that was a deal breaker for me.