On the other side of the hotel, there were a few colorful buildings. I did not get a chance to check out what they were. Further back, I saw a large statue and apparently a monument of some sort. After reading the sign in this photo, I am guessing it was a monument for Wen Tian Xiang 文天祥.
From what I understood, a few years after Kuomingtang lost mainland China and retreated to Taiwan, they decided that they need to have an east-west road through the high mountains in the middle of Taiwan. After the road was built, this particularly scenic spot was named Tian Xiang Taroko and was slated to be a tourism spot. Tian Xiang was referring to a heroic figure in the Chinese history.
In the photo below, you can see that the Chinese words which roughly mean "Wen Tian Xiang, hero of the Chinese people" was defaced. I can only guess that it's the work of Taiwan independence supporters. During the ruling of Chen Shui Bian, the government tried hard to rid Chinese names and Chinese concepts, replaced by Taiwan or Taiwanese ones.
I took the second picture trying to show how large the rocks were - the road and the tunnel entrance was two laned, and the rocks in the river were easily many times the size of the tunnel entrance!