you're probably looking at the TV (torsional vibration) damper. There's an outer ring, with a carefully selected inertia, connected to an inner hub via a ring of rubber. If the rubber deteriorates or tears, the outer ring (and sometimes the pulley surfaces, if they're integrated into the outer ring) will start to come off. Yes, it's important to replace that vibration damper if it's coming apart.