There are many caves near Mt. Adams - Cheese Caves, Dead Horse Caves, Dynamite, Ice, Pond II Cave, Slime Cave, Christmas Tree Cave, just to name a few.
There is a local, Portland area, caving group that regularly visits these caves to clean up trash and explore the caves.
This cave is easy to get to and fairly interesting: http://www.a2zgorge.info/area/GulerIceCaves.htm