The solution is too complicated. But, yes, it can be done and you also described the solution. VertexMaps for the simulation, Fields for the vertex maps, vertex maps for the materials (use the Vertex Map Effect as a Mask under a Layer).
Chris would love that kind of question yesterday in RocketLasso.