If you double the radius of the balloon, you increase the volume eightfold (since volume is proportional to the radius of the cube). But what about the material outside the balloon? Let's say I want to make everything fair and double the thickness of the material for the larger balloon. Since this material only covers the surface area of the balloon, its area will increase four times. If you include double the thickness, the mass of the contents of the larger balloon is also eight times that of the smaller balloon.
But at some point, you don't need to keep making thicker and thicker balloon skins. I can find some material (let's say rubber) that is very strong while being only a millimeter thick. This means that if I increase the radius of the balloon by 10 times, the volume will increase by 1,000, but the mass of the shell may only increase by 100. Volume is important because that's where I get my buoyancy force.
Now let's go the other way. Let's make a balloon for ants. If I reduce the radius of a regular party balloon by 100 times (actually it should be even smaller than that), the thickness of the shell will also be reduced by 100. These balloons are already quite thin. Subtract too much and you won't have a structure capable of holding the balloon together. Increase the thickness a little and the mass becomes too thick to float. Sorry, no parade balloons for the ants.
bigger balloons are more difficult
Wow! I have a giant balloon and it floats. What could be more wonderful than this? Sure, I'd need a group of people (along with some vehicles) to hold it, but it's still a giant balloon. But wait. There are still problems with giant balloons. Making things bigger may make it easier to swim but it also adds other problems.
The first problem is air. Sure, the gust of wind on the little balloon you're holding in your hand is annoying. But what happens when you increase the size of the balloon? This force acting on the balloon is proportional to the area of the cross section. If you double the radius of your balloon, you quadruple the area, which quadruples the wind force.