"Balance" is fighting words in some of the trenches of the Edition Warz.
Some hold it up on a pedestal as the sovereign goal of game design.
Others disparage it, call it unnecessary, and claim it is all an illusion.
Both of those camps are wrong. Balance is a tool. It is an essential tool to ensure a psychological sense of fairness in the players, and thus to keep them coming back to play.
Life Ain't Fair, Kid.
Okay, the cold physics and/or omniscient creator-god of the world really does not make the world a fair place. Bad things happen to good people. Wrong things happen that shouldn't. Some people get hit harder than others. Sometimes you're born to a wealthy and talented Caucasian family in America, sometimes you're born to garbage-pickers on the Subcontinent who melt discareded electronics for pennies.
But that's the world.
A game, an artificial construct, should endeavor to feel as if each player can generally equally contribute. This makes people happy, and thus more inclined to play the game, giving them positive reinforcement and the sense that, if they just keep at it, they can succeed. The fix is not in against them.
Fairness and balance are actually deeply primordial in RPG DNA, in the DNA of every game. We'll see how, below:
Why Roll Dice? Why have a DM?
So, you and your friends are sitting around room and decide to play a game of make-believe and imagination. You each take up an imaginary persona, and interact in an imaginary world, and talk to each other as if you were these imaginary beings.
Right there is the genesis of the Role Playing Game (not to mention the genesis of theater and probably a lot of religion and a significant amount of sex, too, but I digress). You don't need anything more than your imagination. Everyone's a player in this game. It is make-believe in the purest sense.
In this context, you may reach a point where the players in this primordial RPG start testing the boundaries: they start to kill other players' characters, or they start to change the world dramatically, or they decide to do their own thing. This might ruin the fun for others ("You killed my imaginary character!"), so you start inventing rules to preserve the fun. You might say, "Okay, there's a chance that you don't succeed in stabbing my character. Flip a coin." Or you might say, "Okay, any player can veto something that another player's character does." Or you might say, "Since Billy came up with this game, Billy gets to say what really happens and what does it."
So for tabletop RPG's, which are a particular type of make-believe, you have two failsafes: the first is the GM, who makes a judgement call about the thing in question. The second is the funny-sided dice, which help determine if you successfully do things that require some chance and skill.
Those two things, the GM and the dice, are what you might call in design-speak, "Resolution Mechanisms." They help you decide if your character does what your character wants to do. They are the mechanics of the game.
In D&D (and most tabletop games), the GM is the ultimate authority, the primary resolution mechanism. You could play a hypothetical RPG with NOTHING but GM judgement calls. I'm willing to bet a good chunk of the D&D audience have already played entire sessions like this: not one die roll necessary.
So you roll dice to resolve actions mostly to give the resolution to a neutral third party that you can modify with other game rules, rather than to put it all in the hands of a GM to simply judge as they see fit. The dice are impartial, and they're modifiable.
They're also exciting and unpredictable and tension-generating, and mystery-resolving…those are psychological reasons, and they're also important.
Resolution and Balance
So now that we know how and why your character can accomplish what they want to do (either because the DM allows it or because they make a successful die roll), we've got a metric for balance:
Balance is succeeding roughly as often as any other character.
A DM can unbalance a game by favoring a certain character or a certain player above the others. Think of the worst "DMPC's" or the worst "DM's Girlfriend" characters you've experienced: this was DM-based imbalance, helping them to succeed where others would fail. It also manifests if the DM has a certain bias. If a particular DM doesn't like gnomes (but allows them anyway), and then spends the adventure ruling against everything the gnome tries, that's imbalance.
The dice can also unbalance a game, either by simple luck ("I haven't rolled above a seven all night!"), or by the numbers you apply to the roll. Think of min-maxing: putting penalties in things you never roll for, and putting big bonuses in things you always roll for. Think also of skill bonus items or saving throws in 3e, how two characters might have wildly different scores so that one character might ALWAYS succeed on a check that another character has NO chance of succeeding on.
Either way, you wind up with a definition of balance that measures successes. Roughly speaking (since D&D always takes specializations into account to a certain degree), you should have vaguely the same chance as the next person to accomplish a given deed.
Balance and Specialization
A strict form of balance could be something like a coin toss: every character always has a 50% chance of accomplishing a given task that they try to accomplish.
This is very balanced. However, it is not necessarily a fun play experience: it stretches the believability of that imaginary world to imagine that the trained knight and the local street tough have the same chance to hit someone with the lance on a charging steed. It stretches credulity to think that the knight would be as skilled at picking a pocket as that street tough. In a tabletop RPG, the believability of the world is very fragile, so it's not always ideal to stretch credulity so very thin. Furthermore, imbalance creates a variety and chaos that is fun in and of itself. It creates a sense of the unexpected and a feeling of anticipation that a strict equal balance doesn't.
Specialization, then, comes into play. It can come into play vertically (the street urchin has a higher chance to pick a pocket, though both can attempt it), and it can come into play horizontally (the street urchin can pick pockets as a special ability, and the knight cannot), and it can rely on different resolution mechanics (the DM decides that the knight can't pick a pocket, but the thief can, based on what they know about the characters).
Specialization necessarily leads to some degree of imbalance. This is desirable, and inevitable. If the adventure consists only of picking pockets, the knight is out of luck, and if the adventure consists only of charging folks on horseback, the thief cannot contribute. Both of these are problems, clearly unbalanced scenarios, so if the game is to include specialization, generally speaking it needs to feature each area of specialization.
It's always possible to skew the results — to include a night of gameplay set at the faire where the knight dominates — but these are risky propositions. You might want to have the thief player roll a new knight for this evening's game, or you might want to include a subplot featuring picked pockets, or if you're in a regular group you may want to have a night next week where the thief dominates. You can do entire campaigns in this way, but the longer you skew the odds, the more you should probably telegraph that in advance, and simply adjust character selection so that no player is left out of the participation — so that everyone remains basically balanced.
It's in this way that specialization plays into variety of gameplay experiences.
Balance and Pacing
We can see from the examples above that it is difficult to get perfect balance that also has variety and specializations, so in order to keep things "balanced" overall, we need to keep a variety of actions bubbling up for the players to thwart: each specialization should somehow be useful to be balanced, so we need to change scenes over the course of a session so that each specialization comes to light. This also allows us to accept weakness: the pocket-picker won't ALWAYS be the star of the show, and neither will the horse-rider. They will each fail some times, and succeed some times.
The fact is, we always remember our failures more prominently than we note our successes. It is a very common cognitive bias. The knight player is going to notice the session that was all about picking pockets more than she'll notice the session that was all about jousting, and she won't be happy about it. Likewise, the thief's player will notice that joust-fest he didn't contribute to and won't notice as much the crime spree he got away with.
However, people are sensitive to different levels of this. Some folks will feel quickly slighted (or at least bored) if they don't contribute anything for ten minutes of play. Others might be fine with never really using their specialization in an entire two year game. It's a highly subjective thing.
As a guideline, the game session is a useful metric. The timescale is long — often at least a few hours — and the time between sessions is also long — often at least a few days. You can easily imagine someone new to the game getting quickly disenchanted if, in their first try at the game, they couldn't contribute because of some quirk of DM adventure selection. So we generally want balance to occur over the course of a single game session, ensuring that each character's abilities are highlighted before days pass, and letting each player feel like they're contributing at least something to the meeting. If you're cutting a character's specialization out of the night's gameplay, you might want to let them make a character whose specialization IS relevant for that night.
Within a session, we must have variety. It is less key how this variety is achieved than it is that the players trust that the variety will exist. The DM might select, or you might roll dice for random adventures, or the party might select from a menu of things they might do, or whatever. The important part is that each session include each specialization. A session of Lancers & Larceny should probably have both jousting AND pocket-picking. Half and half in a perfectly balanced game, but perfect balance is always artificial.
That, notably, is why we shouldn't balance based on the combat round or some similar, smaller unit (such as the encounter). It makes sense that sometimes a character won't do well in a given encounter, that sometimes a character will fail against a particular enemy, and that sometimes a character will dominate. The smaller the timescale, the less that becomes true, returning us ultimately to that coin-flip scenario where there is no true mechanical difference between characters. That isn't satisfying, but then, neither is repeated and frequent failure (or repeated and frequent success).
The Big Takeaway
So we see that balance is a game psychology thing. It is essential so that the game remain appealing to the players, so that the players feel like their contributions matter, and on a timescale that lets them contribute close by areas where they will likely fail.
Balance can be taken away by a DM, or by the mechanics, either intentionally or not. Strict, specific balance will feel artificial, but a broad balance keeps folks returning.
D&D specifically has had some trouble with a particular kind of balance related to the magic system of the game, but that, I'm afraid, is a different musing.