Seriously, try not to give it a second thought.
It's not personal, it's social. Although he's a regular nice guy to me, he's an accomplished Actor and Celebrity with career ambitions and the desire to utilize his social media in the very best possible way.
That being said.. Yes of course the fans are absolutely important! Connecting with fans and supporters of your work should be a paramount social media goal. And so should having a strategy that builds and retains those fans and supporters, while maximizing social media enjoyment and effectiveness for YOURSELF. That means having to let go of noisy feeds from time to time, which logically means unfollowing people. Agreed? And they shouldn't have to first unfollow real life friends and family or industry connections to achieve a more enjoyable Twitter feed should they?
My advice? Follow celebrities that you're interested in following and not because you have the expectation that they should follow you back. It's nice, yeah. But it shouldn't require immediate unfollow retaliation if you don't get it.
Plus, I happen to know for a fact that Johnny Whitworth loves all of his fans regardless of follow status, including YOU!

Thank you from the bottom of our hearts for your continued support of Johnny and his recent exciting projects. You'll be able to catch him this fall on NBC's hot new show BLINDSPOT, premiering Monday, September 21st at 10pm!