Question As Written – Multiple Problems
My bat familiar (from the find familiar spell) has a Strength score of 2, and can therefore carry 7 lbs. (Per the “Lifting and Carrying” rules (PHB, p. 176), its carrying capacity is its Str score times 15, divided by 4 for being Tiny.)
I cast enlarge/reduce on a rock that weighs 56 lbs. to Reduce it, making it weigh 7 lbs.
So far, so good.
I have my familiar fly over a target. Then, as an action, I recast enlarge/reduce on the rock to Enlarge it,
No. Per the description of Enlarge / Reduce, it targets an “object that is neither worn nor carried”. If your bat is carrying the rock, you cannot cast Enlarge/Reduce on it.
cancelling out the first enlarge/reduce spell and enlarging the rock x8 to make it weigh 448 lbs.
Maybe not. When you begin to cast Enlarge / Reduce, that immediately ends your concentration on the previous spell and the rock returns to 56 pounds:
As soon as you start casting a spell or using a special ability that requires concentration, your concentration on another effect ends instantly.
This optional rule from XGtE (p.5) was officialized in the Sage Advice Compendium 2.5 and more recently repeated in TCoE (p. 5)
“If you’re concentrating on a spell, your concentration on it ends immediately when you start casting another spell that requires concentration.”
Although your previous Englarge/Reduce ends immediately, your new cast does not take effect until your new spell is complete. Thus, there will be a slight delay between ending the concentration on the first spell and completing the second spell so that the rock grows to 448lb. That delay is the casting time of the spell, which is one action.
One action does not map out to any amount of time more specific than “less than about 6 seconds”, but we can estimate the actual amount of time using real world falling speeds. The range on your Enlarge/Reduce is only 30 feet. With this maximum range being the 30 foot hypotenuse from you to the rock, you are standing 20 feet away from your opponent, and rock is just 21.25 feet over them. To fall 21.25 feet it will take the rock just 1.15 seconds. Can you begin and end your spell in just over a second? DM’s call, but possibly not.
Although they are written for falling creatures, if you apply the optional falling rules from XGtE (p.77) to objects, the rock actually falls the first 500 feet instantly, meaning you have no time to cast another spell on it as it falls.
This is too much for the familiar to hold and it drops the rock. Not as an attack, but because it has no choice.
It does have a choice. It can choose to hold on to the rock, and fall with it, rather than dropping it.
Question as Intended – Not an attack for either of you
Setting aside the Rube Goldberg (or, perhaps Heath Robinson, since you are Canadian) set up of your problem, what is it that you are really asking? Looking at your comments to others:
A familiar cannot attack. So it matters in the sense that if it is considered an attack, it will not work if the familiar is the one considering to have attacked. But if it is considered my attack then it would be legal
All that rule states is that IF there is an attack roll it is definitely an attack, it does not say that if there is no attack roll it is not an attack,
You clearly understand that a familiar by definition cannot attack, so dropping the rock cannot be the familiar’s attack. I think you are trying to ask something like, “Can I make an attack if I haven’t taken the attack action?”, or “Can the effects of my spell constitute an attack if it is not actually a spell attack?” or “If my decisions cause damage to happen, is that an attack?”
If something like this is your question, then I would answer No, this is not an attack, for design principle reasons.
Let’s clear away the superfluous familiar and the double-casting of Enlarge / Reduce, and posit this: Your enemy is standing under a precariously balanced rock at the base of a small cliff. With a single casting of Enlarge/Reduce, you could dislodge the rock, causing it to fall on them. Is this an attack?
No. NOT because you don’t make an attack roll specifically, BUT rather because you don’t have an available Attack action to make. Your action this turn was to Cast a Spell. You did not choose the Attack action, therefore you cannot make an attack. You can certainly use your spell to set up a potentially damaging environmental situation, and you may end up hurting your opponent, but it was still not an attack.
My perspective here is informed by my understanding of action economy, agency, and game balance. This is not an articulated game feature, but it underlies nearly every discussion on this site about what is permitted, and what is balanced. Other DMs might rule differently, but at least for me:
As a PC, you receive a limited number of actions. However, when you take those actions, you have a great amount of control over them. If you choose to Cast a Spell on a legitimate target, most times the spell will work and do what you wanted it to do – you will Enlarge that rock and cause it to tumble off the cliff. If instead you choose to Attack (perhaps run up that cliff and push the rock down on your opponent as an improvised weapon), you have some control over that – you get to make the attack roll, perhaps drawing on your relevant ability bonuses, Inspiration, or even your familiar giving you Help. That attack is mostly about you and your rolls, and the opponent is just a passive AC score for you to hit.
In either of these cases, you have paid for your agency in the currency of actions. Allowing you to both cast a spell that significantly changes the world, and then also have an ancillary result of that change be an attack, your attack, would be going beyond your allotted actions. Rather than this, your spell action has now created an environmental situation that could damage your opponent. But the resolution of that is now not under your control or influence – it is the DM’s concern. Perhaps they will allow a saving throw, but they get to decide the ability score targeted, assign the DC, and assign advantage or disadvantage – you are not an agent in the process. Or, perhaps for narrative reasons, they will simply decide that your opponent steps out of the way or gets flattened by the rock. The point is that this is the DM’s action, not yours.
From this framework, let’s consider your follow-up questions:
An alternative but similar thought for comparison: If I use an action to dismiss the familiar, does it drop the rock? (ending concentration on the enlarge/reduce spell and still dropping a 56 lbs. rock)
You can end concentration simply by choosing to, without an action – but you originally paid for the spell with an action. You can dismiss the familiar as an action. For me, you have paid for this to be an attack by spending your dismissal action on it. I would be fine with, in that case, allowing you to time the dismissal to correspond with dropping the rock on the opponent, and allow you to make an attack roll – I might allow you to include proficiency, and I might allow you to use your spellcasting ability as the to-hit modifier. Even if I didn’t allow an attack roll against passive AC, I might have you make an Arcana or Animal Handling check contested by the Athletics or Acrobatics of the opponent. In any event, the important thing is that you would get to make the roll. You have purchased that by spending the action; I would consider it an attack.
In either case, does this count as an attack from the familiar, or because I used an action to cause it, is it my attack?
It would never be an attack from the familiar, by definition. A familiar can certainly cause damaging things to happen, but these are not attacks, because their resolution is squarely under DM, not player, control.
If you used an action whose primary function was cause damage, that to me is an attack, even if it also results in the dismissal of your familiar, and whether or not I called for an attack roll. If you used an action whose principle result was the casting of a spell and a subsequent change in the nature of reality (growing or shrinking a rock), that is not an Attack, that is Casting a Spell.