The tendon seemed "kind of tender" after an early morning gallop and veterinary scan, according to Dennis O'Neill, brother of trainer Doug O'Neill.
"The horse is sound and happy, but it's not worth it," O'Neill told The Associated Press.
It was the first time since 1936 that the Derby and Preakness winner didn't run in the Belmont.
"It's not tragic, but it's a huge disappointment," Doug O'Neill said on "The Dan Patrick Show."
I'll Have Another had shin problems in September and took the rest of the year off. He returned to racing in February and won the Robert Lewis Stakes at Santa Anita, putting the colt firmly on the Triple Crown trail.
He won the Kentucky Derby on May 5 and the Preakness two weeks later to set up a highly anticipated bid to become racing's first Triple Crown winner in 34 years.
But the Belmont Stakes isn't called the "Test of the Champion" for nothing, and its longer distance has stopped 11 previous horses' attempts at winning the Triple Crown. Given the slightest hint of a problem, the colt's connections withdrew him rather than risk further injury for a shot at making history.
"It's devastating. I thought this was going to be one of the greatest races in history, and I wanted the opportunity to be part of it," said Dale Romans, trainer of second favorite Dullahan. "But this is bigger than that. This is terrible news."
From the day after I'll Have Another's thrilling win in the Preakness, not much has gone right for Team O'Neill. First, O'Neill was suspended 45 days and fined $15,000 by the California Horse Racing Board for a medication violation. His suspension is to begin after the Belmont.
Then, racing stewards said that for the Belmont, I'll Have Another could have to go without the nasal strip he wore in races this year. Exercise rider Jonny Garcia also had visa problems and had to be replaced for several days.
Also, the New York State Racing and Wagering Board ordered a detention barn to be set up for all the Belmont horses to strengthen security, a decision that angered trainers, who said moving their horses might have an effect on their performance.
"With a horse like I'll Have Another, you're under a really strong microscope," O'Neill said earlier.
Copyright 2016 Nexstar Broadcasting, Inc. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.