It sounds like a chainsaw, and I've thought about and studied those fairly extensively
I am not going to go into vibration due to the mechanical output of the engine, but describe how I'd look at breaking down the linear vibration in 2 axes due to the weight of the rotating parts.
I'd look at it in the 4 quadrants of rotation, 0, 90, 180, and 270 degrees from TDC.
Near TDC, the piston is essentially stopped,, but the 'big end' of the connecting rod is moving, and on the opposite side of the main bearings will be the counterweights moving in the opposite direction, you'll have an acceleration (vibration) of the main body due to the differences in weights of these two sides of the crank.
Near 90*, the piston mass will be moving, as well as the connecting rod, the ratio of accelerating masses on either side of the crank will be different than when it was at TDC.
When balancing a single cylinder engine, the counterweights will probably be designed to weigh somewhere between the two extremes, too much weight and you'll have a lot of (horizontal) vibration when near TDC and BDC, too little weight and you'll get (vertical) vibration near the 90* points
Hope this makes sense.. I can't help much with the formulas though.