We spent the whole morning there and left at about 12:00 noon.
There were a lot of local tour to Nami Island from Seoul. These tours will depart from city of Seoul so they will arrive at about 11:00am. Hence if you visit there late, you will have to line up with thousands of people to enter the island.
I took this picture when I left the island, some other people were going to leave to, but the people waiting to go to the island were so many that they almost filled up the whole carpark.