The downside is you have no one to recount the experience with later on, you might tell people about it but no one will ever understand what it was like unless they were there with you.
This is true... but you do end up making different kinds of memory on your solo trip. It may not be with your loved ones, but it may be with type of strangers that you may not have met.
I went on a solo trip to Japan couple of years ago. Even though I had planned out the trip tightly, I ended up veering off course. For example, I was just strolling aroud one day, and saw a fellow solo tourist hop into a bike rental shop. So I followed him, and rented a bike myself too. Whopping $9/day. And we kept on crossing path over the course of day, which was hilarious.
Around 9pm, I needed to return the bike, but I was completely lost. With all businesses shut down in deserted part of kyoto, I was helpless. But I found this one-man police station, so I walked in, asking for help. This chubby jolly patrolman was so delighted to help despite communication barrier, he went out of his way to walk me towards the rental shop.
Another day, I was just observing people on the street. I was surprised how many young men and women make living hawking on behalf of clubs and restaurants. One young lad caught my attention. He was shouting loud and he looked thirsty. So I ran to a nearby vending machine and handed him a can of ice coffee. Initially he was completely baffled, but then he got it after my lame attempt at communicating. And thanked me and gave me a big smile. So he and I have a nice shared memory.
Anyway, both types of trips are enjoyable. People who've never done solo travel definitely should, when luxury allows.
Dervla Murphy (Irish solo travel writer, in her eighties now) has often expounded the value of travelling alone. Basically she says that if you travel with others it's a safe cocoon. But if you travel alone you end up observing much more and interacting much more with the culture and people you visit.
And this is a woman who went out of her way to avoid a 'safe cocoon'. Such as traveling up the Indus valley on donkey-back in the 1970s. She did it when she was a single mother and took her daughter, aged less than 10, along with her. Just the two of them.