Hey Greg,
Ya, there can be many things like temp, hiding space, movement around the cage, vibrations from loud music, etc. Try to make him as calm as possible. He could have a parasite.
Did you scent the pinkies first? Were they dead and cold? Mine take thawed pinkies when they have been scented with fish. I will take a fresh water minnow freeze it, then and cut it into pieces, and use 1/2 or 1/3 of it thawed, smashed in some cool water.
If that doesn't work, try sliced beef heart. Mine go crazy on that and you can cut the pieces into 1/4 x 1/4 inch by one to three inches long and it goes down nicely and quickly. It also freezes well and is cheap, 2-3 dollars a pound.