The BFMI way is to break a polygon into rectangles and triangles. In theory, this is generalizable to 3d, but my visualization skills are probably not sufficient for that not to be an error prone path. It seems like that there should be some geometry that provides a simpler path for certain classes of polyhedra.
In general and for lower dimensions, volume can be derived from the divergence theorem.
The general formula is
## {\displaystyle {\frac {1}{3}}\left|\sum _{F}(Q_{F}\cdot N_{F})\operatorname {area} (F)\right|,} ##
... where the sum is over faces F of the polyhedron, QF is an arbitrary point on face F, NF is the unit vector perpendicular to F pointing outside the solid, and the multiplication dot is the dot product Do not trust my latex here. Use this:
See:
Goldman, Ronald N. (1991), "Chapter IV.1: Area of planar polygons and volume of polyhedra", in Arvo, James (ed.), Graphic Gems Package: Graphics Gems II, Academic Press, pp. 170–171
See for higher dimensions: Büeler, B.; Enge, A.; Fukuda, K. (2000), "Exact Volume Computation for Polytopes: A Practical Study", Polytopes — Combinatorics and Computation, p. 131, CiteSeerX 10.1.1.39.7700, doi:10.1007/978-3-0348-8438-9_6, ISBN 978-3-7643-6351-2