You don't understand what vacuum energy is.
For one, it's not just "energy" (a scalar value). It's stress-energy tensor (a 4x4 matrix). If you want to work in a simple flat Minkowski space, you can do away by looking at four-momentum, but it's still a 4-dimensional vector, not a scalar.
More importantly, vacuum energy, in order to be *vacuum* energy, has to have some rather unusual properties:
(1) It's a local minimum of energy. Any fluctuations on top of it (such as adding any particles to vacuum) _increase_ energy. [this answers your question]
(2) It should be Lorentz-invariant: boosts should not change it ("moving relative to vacuum is undetectable"). This means that (in flat Minkowski space), it should have form (p, -p, -p, -p) - note the negative pressure. This is rather unusual compared to most other forms of energy you know. For example, "dust" has four-momentum (ro, 0, 0, 0), "fluid" (for example, radiation-dominated early Universe) has four-momentum (ro, p, p, p).