How to balance not only the gameplay, but the systems of two very distinct but believably meshable games? Start with a faithful recreation of both, and alter and add subsystems to each to allow them to interact appropriately.
Let's start with the rundown of each, separately.
First off: Items omitted. There is no way I can think of to get those to mesh. With the exceptions of item-spawning moves, of course; those would get their own inputs unique to them and how each one is used.
For PSASBR characters:
Controls for Nintendo*:
*Let's be honest here, I'm not messing with the controls for the original systems. That said, this is how I would map the controls to make them more accessible and natural to those with experience playing the other game. They would be customizable anyways, so you do you.
Standard controller (Wii U Gamepad, Wii Classic Controller, Wii U Pro Controller)
Left stick: Move
A: Attack 1
Smash input or A+B or right stick(option)* (or either X or Y): Attack 2
B: Attack 3
ZL(+Left stick) or right stick(option): Throw
Plus Control Pad**: Taunt
*Option refers to one of the input methods, not the movement it inputs. This basically is a customization option; suit it to your own playstyle. "(or this)" is much of the same; make a substitution if you so desire.
**Henceforth known as PCP.
GameCube controller (many will play with this out of nostalgia, habit, or experience)
Control stick: Move
A: Attack 1
Smash input or A+B or C stick(option) (or either X or Y): Attack 2
B: Attack 3
Z(+Left stick) or C stick(option): Throw
3DS (may as well take it on the go)
Circle pad: Move
A: Attack 1
Smash input or A+B or right stick(option) (or either X or Y): Attack 2
B: Attack 3
L(+Left stick) or PCP(option): Throw
Select or PCP(option): Taunt
Wii Remote and Nunchuk (common accessories; easy to hold)
Control stick: Move
A: Attack 1
Smash input or A+B or Shake-smash(option) (or Z): Attack 2
B: Attack 3
1 button or PCP in direction not being faced(option) (or Z): Block
2 button or Shake-smash(option; I recommend this to require a button being depressed while shaking to perform the move): Super
Wii Remote sideways (I'm crazy)
Tap up twice: Jump
2: Attack 1
1: Attack 2
1+2: Attack 3
A+no input from PCP: Taunt
Minus or Shake-smash(option; again, optional requirement of a depressed button while shaking): Super
Pokkén vets (For Classic/Hori. One last setup, because Pokken controls and PSASBR are so similar.)
Control Stick or Plus Control Pad: Move
Y: Attack 1
X: Attack 2
A: Attack 3
B+Y+CS/PCP(of course) or ZL+CS/PCP: Throw
L+R or ZR: Super
All characters can block attacks from all directions by holding a single button. This prevents all damage and thus, meter gain. Counterable with throws, of course.
While blocking, roll by inputting a move left or right. Intangibility frames, but punishable at the end.
Airdodges are performed with the block input while in midair, of course.
All characters have a double jump; except for Kat, who has an aimable airdash with its own strengths and weaknesses.
Combos are very possible with these movesets, however after being comboed for a specific maximum amount of AP the victim is blasted away invincibly; that state only wears off after attacking or landing on solid ground. There is an AP reward for triggering this Infinite Prevention System.
Offensively, each character has three types of attack. And they have up to (and usually all) four of each attack type on each of the ground and midair: Neutral, Side, Up, and Down, selected by the held direction of the Move input. This means:
N1 S1 U1 D1 N2 S2 U2 D2 N3 S3 U3 D3
aN1 aS1 aU1 aD1 aN2 aS2 aU2 aD2 aN3 aS3 aU3 aD3
is your basic moveset. Some characters don't have all 24 moves covered by unique attacks, and that's disregarding the fact that many aerials are just alternative versions of the same input on the ground; but there are more than enough unique ones to warrant listing all 24 here. Add followups on some moves (mostly N1 aka jab) and there are more than enough attacks for every character who needs them.
Attacks with 1 are often simple melee attacks. Attacks with 2 are either more ranged or slower, more powerful attacks. It may help Smash players to consider these moves "semi-specials". Attacks with 3 are more often than not equivalent to Smash Bros. special moves, and can vary greatly in effect.
Each character has three throws: Side, Up, and Down;
St Ut Dt
These not only put your foe somewhere else, but they force the opponent to eject AP orbs they have collected. You can then pick those up.
By hitting opponents with attacks, you gain AP; All-Star Power. This fills up a meter on the character portrait on the UI. Each character has different amounts of AP it takes to get to each of three levels. Once it reaches Level 1, the character can perform a Super, which drains the bar of however much it took to get to Level 1. However, it's not done filling yet; it continues going up to Level 2, which means the character can activate a Level 2 Super (and only Level 2, not Level 1, sorry but this isn't Street fighter with EX attacks). Due to 0>1 being less than 1>2, it is possible to preform two Level 1 Supers in a row. And due to similar being true for 1>2 and 2>3, a Level 2 followed by a Level 1 is feasible as well.
Each character has three levels of Super, and the bar stops filling after you reach level 3. Powerful, yes; but be sure you know how to use that one properly. Keep in mind that it may be more efficient to just use 1s and 2s.
Connect with a Super to score a kill, and temporarily remove the victim from the playing field. They arrive again with an invincibility period. Death also occurs by falling out of stages with gaps, holes in the floors, or without solid walls in all places.
In timed play, most score wins; score by killing and lose score (or not) by dying.
In score play, get the target amount of points by killing the other players.
In stock play, kill other players to make them lose a point from a preset amount. When you have 0 points, you don't respawn. Last man left wins.
For Smash Bros. characters:
Controls for Playstation:
Dualshock (Let's say...4.)
Left stick or dpad: Move
Triangle or R1(option) (or Square+Smash input): Smash attack
Right stick or R1(option): Grab
All characters can pull up a shield by holding a single button. This prevents all damage. However, the shield shrinks over time and when hit by attacks, unless perfect shielded (pressing the Shield button at the last moment before getting hit). If the shield is not large enough to block an attack, it's as if the shield was never there at all. If the shield instead is allowed to shrink to nothing...it breaks, the shielding character is tossed into the air for a second, and they will be stunned on landing until either enough time has passed or they get hit. Counterable with throws, of course.
While blocking, roll by inputting a move left or right. Intangibility frames, but punishable at the end. They can also spotdodge by hitting down, jump out of shield with jump button or tapping up (only with tap jump on of course), or grab by hitting attack.
Airdodges are performed with the block input while in midair, of course.
All characters have at least a double jump; some have more.
All characters can dash, and enter fast fall. Not much more to say on those.
Combos are not as easy for these characters. They can do it, sure, but most of their attack are for knocking the opponent away, especially the more powerful ones. Any combos are often looser than in other games and require much more specific stick inputs (and watching for DI), but instead of rewarding with just damage, they often provide a positional advantage as well. Befitting the playstyle.
All Smashers have a jab, with various properties; holdable, tap-to-combo, infinite with finisher, etc.. Three tilts, side up and down. Five aerial attacks; neutral, front, back, up, and down. These all (well, most of them) come out quick. Throw them out whenever you like.
Jab (and Jab Combo) Ftilt Utilt Dtilt Nair Fair Bair Uair Dair
No reason to list moves with the same words, right? Preempts some forms of confusion.
Anyways, there are also four special moves; these are always the same (or mostly similar) on both the ground and in the air.
Neutral special, Side special, Up special, Down special
Up special is usually a recovery move that aids in getting back on top of a stage after having fallen below it. This is important to know for a point I will make later.
And there are three smash attacks for each character:
Fsmash Usmash Dsmash
These each can be charged, and often send the opponent flying a good long distance, if not past the blast zone entirely. Charging increases damage and knockback, but can leave you vulnerable if you're not careful, so be careful. Though that should go without saying.
Oh, and there's a dash attack.
It's good at starting combos and winning in the neutral.
After grabbing an opponent, they aren't thrown immediately. You get to pummel them with the attack button! And then you get to throw them in any of four ways, just by inputting the direction!
Forward throw, Back throw, Up throw, Down throw
Be sure to either spam pummeling for max damage or throw them before they break free.
By hitting opponents with attacks, their %, or damage percentage, increases. This is shown on the character portrait on the UI. When hit, a character will get launched farther if they have a higher amount of damage. Launch a foe offscreen beyond an invisible "blast zone" to score a KO on them.
After being KO'd, a character is briefly removed from the stage. They arrive again with an invincibility period. Death also occurs by falling out of stages with gaps, holes in the floors, or without solid walls in all places, i.e. most Smash stages.
In timed play, most score wins; score by KOing and lose score (or not) by falling.
In score play, get the target amount of points by KOing the other players.
In stock play, KO the other players to make them lose a stock from a preset amount. When you have no stock left, you don't respawn. Last man left wins.
Okay, so that's the recap...it seems simple enough. Two platform fighters.
But where's the rub?
The two systems don't actually mesh well at all. Sure, the main movesets could be slotted into each other with a bit of fuss, but the scoring systems don't play that well together. I believe the issue originates from how damage is tracked: For PSASBR, it's a power bar on you. In Smash, the damage percent is applied to your opponent. And Smash doesn't need super attacks to kill, but the All-Stars outright require it. Therefore, something has to give; or, if you'll forgive my joke, switch.
In this game, Smash-style fighters can use the original method of KOing and damage tracking, but only if they are fighting other Smashers. Otherwise, or by selectable option, by attacking an opponent a smasher's own launch % will increase. This increases the distance a foe is launched as you hit them, as if they were using the original rule and the percent applied to them. This preserves most Smash-style combos, and tracks damage done on an All-Star without foisting a damage percent on them or their UI. It's not like they need two super-important stats to be tracked on them and none on the smasher.
Which brings me to my second point: Smashers now have a power meter. It fills by collecting dropped AP. They cannot directly produce AP Orbs; they don't really need them. They're building up their damage to kill just the same as the other guys. But it's not fair if they're just keeping a powerful move in reserve that can potentially KO you twice, now is it? Let's even the score by allowing Smashers to knock AP out of their foes the same ways they do to themselves. Pummels and throws are a good source of this, but a fully charged Smash or other specific attacks should be just as capable. You can't knock AP out of a Smasher, either; All-Stars can still gain it by hitting them and it scatters when they're thrown, but Smashers can not lose their acquired AP. (Oh, and the meter for this is much more subtle than for the All-Stars. Uh...let's say a meter that appears only as AP is being picked up, like Cloud's Limit meter only appears while charging.) Meter should take somewhere between a Level 1 or Level 2 to get to, due to the indirect methods required to gain any of it.
At full meter, a Smasher will be in a state as if they had picked up a Smash Ball. They lose access to their neutral special, but get to input it anyways to perform a Final Smash. This often, but doesn't always, kill. But there's something important to note before touching on the Final Smashes any more...
All that damage you're building up has to go away some point in time. On KOing a foe, your launch % stops going up on hit, and a timer based on your launch % begins to decrease, and when it gets to 0 so does your %. In Battle Royale, one Super can kill many players at once. Especially higher-leveled ones. So if you've been building up % for a while, it hardly seems fair to take it all away from you at once just because you gimped that one guy. Gimps can be cheap, in the cost sense; if you kill from low percent you don't really need to keep it all that long. The more damage you've done, the longer you get to keep your launch %. This system gives you a chance to KO any other players on the field (if any) or, if you've got more than enough for the size of the stage (i.e. they die hard) you can possibly KO the same guy twice. Oh, and any damage you manage to do before it hits 0 outright is stored, and you get it back when your % resets.
Now, about those Final Smashes. Transformative final Smashes will apply the same rule on kill. With - for example - Giga Mac, Samus' Gunship, and Super Dragon, you continue doing damage throughout so you'll get something to show for it after it ends. One-off attacks, however, will either drain a portion of your launch % per kill (somewhere from 10-25%, i.e. 1/10 to 1/4) or a set amount of it (50%, 75%, 37%), which is removed after the FS ends. The harder to hit with the move, or the less powerful it is if it's easy to hit with, the less that is taken away. No need to take away all that hard work because of a sudden instakill, right? Especially if you earned it either way.
It's worth noting that some stages have walls but no ceiling. This is okay. Smash moves can cause a wallbounce; ASBR seems to lack this. If you can't kill of the side, you can combo off them. Up smashes or other moves that launch upwards become the better killing options on these stages.
Oh, and on stages in open air, are All-Stars at a disadvantage? I think not, not really. Their double-jump system is pretty much the same as in Smash Bros., and due to their combo system I'm confident that every character has at least one aerial option to continue a juggle. This also means at least one option to recover.
They also have no "blinky state", or special fall. Not that I know of, at any rate.
Or have you forgotten that Battle Royale has its own fair share of stages in the air?
Basically, any stage that has at least one completely/mostly open wall is fair enough for both Smashers and All-Stars. When Smashers have to work harder, the system does them a favor. When the stage favors them, the system (slightly) works against them.
This is how I believe a game could utilize both Super Smash Bros. and All-Stars Battle Royale gameplay at the same time, keeping both true to their origins while also keeping them balanced.
Are there other ways to do this? Sure. But this is what makes sense to me. Every other thing I've seen (mostly in fanfiction) overpowers various aspects, and I'm not willing to step one toe off that end. I did the best I could, mostly disregarding everything but what appears in the original games themselves.
Is there room for other fighters from other games to keep their own systems and mechanics while getting in on the fray? Sure. Pokken could fit rather snugly, but that's something I'd have to think about the details of later. Rivals of Aether could be the same way...though I'd have to think about how AP would help them, especially when they can't grab. Eh, they have other benefits, and are objectively more powerful than Smashers. Add the system things for Smash that apply to RoA and it's mostly balanced.
Is there room for items? Items in Smash are usually either single use or versatile. Multiple attack types for battering weapons, and even for those that can only be thrown there's more than one useful way to toss it. Battle Royale items are not nearly as useful nor versatile, I don't think. Something would have be tweaked on all sides to make those fair, and that's a lot of work and explanation nobody wants to do. So, no, with some minor exceptions that would not go any farther than being throwable. That said, I wouldn't be against a Smash Ball that gives about a Level 2's worth to All-Stars and does...well, you know, for Smashers.
Anything I did wrong? I dunno. Am I underestimating or overestimating either side? I have logic, and I have experience with Smash, but I don't have personal experience with Battle Royale. I do have research, though, and a designer's mindset and imagination. I think I did a pretty good job, but if you disagree, please let me know.