I've been a coach and an AD and here's my input:
At the beginning of each season most coaches hold a meeting for families explaining the schedule, rules and expectations, etc. At that meeting parents can ask for clarity regarding the selection process for Varisty, State qualifying, etc. Coaches have a responsibility to explain their process, and ADs have a responsibility to ensure they do. Anyone who believes it's simply a high school coach's prerogative to make decisions without any explanation is giving the coach way too much room to be arbitrary.
As an AD I used to ask every coach to develop a clear and transparent system for making such decisions. That was the only way I could back them up when parents came in to complain. By the way, posters who say ADs laugh at parents who complain have never been ADs - interacting with parents, athletes and coaches is the job.
While it's incredibly difficult to objectively determine playing time for sports like basketball, track and XC have the huge benefit of time and distance. There are still many factors (leadership ability, performance under pressure, hand-off ability, attendance at practice, etc) but it is much easier for track/XC coaches to define the expectations, at least a decent/reasonable/fair coach.
As a coach, we wrote into our policies that coaches sometimes have to make hard decisions at the end of a season - one year we did put a senior on the team as our #7 instead of a freshman who beat him at Districts, but we looked at the entire season to make that decision. And we were open about it.
I don't have children, so I can't speak to how to parent. It seems a bit late in the game to complain now, although your son has every right to ask for clarification. But he's young and may not feel comfortable doing that, and from your description his coach doesn't sound too open. At this point, it's fine to go directly to the AD, let him or her know that there was confusion about the selection process for state, and that next year athletes and parents would appreciate clear policies.
We rail on this message board about bad coaches at the top levels (US 4x1 right now), but posters here are defending this high school coach. Based on my experience and how I define good coaching, he hasn't don't his job because he hasn't been transparent with the youth in his care. But it's a bit late to realize you actually don't know what his criteria are for decision making.