In a large saucepan, add the fruit, sugar, jelly and brandy, and cook until the fruit is starting to release some liquid and beginning to burst, but still holds its shape, about 5 minutes.
Line a large 6-cup bowl with plastic wrap, allowing lots of overhang.
Remove the crusts from 10 to 12 slices of bread. Press the bread slices briefly in the juices at the top of the berries in the pot then lay them into the bowl, juice-side down, ensuring there are no spaces. Use smaller pieces to overlap or close gaps in spaces, as needed.
Remove the berries from the heat and cool slightly. Using a slotted spoon, spoon the fruit into the bread-lined bowl and top with a final layer of bread. Spoon a bit more of the juices over the top to color the top layer of bread evenly. Reserve the remaining cooking juices for serving. Cover with the plastic wrap and place a plate over the top. Add a large heavy can or a 2-pound weight to the plate and let sit at least 8 hours, or overnight, in the fridge.
Gently invert the bowl onto a large serving dish, shaking it lightly to loosen. Peel the plastic wrap off and spoon the reserved juice all over to soak any bits of bread that still look white. Serve with a bowl of whipped cream on the side, or cut individual wedges, and add a dollop of cream and berries for garnish.
For the whipped cream: Add the cream to a bowl fitted with a whisk attachment and whip until soft peaks just begin to form. Add the sugar and vanilla, and continue whipping until almost stiff and reserve for garnishing summer pudding.