In light of the recent deja vu discussion on the Tcl newsgroup about Tcl/Tk's popularity etc., I thought I would abstract things out a level. I'm not trying to solve any problem, suggest a new way things should be done, or influence or convince anyone of anything. But I wanted to try to give a bit of what I see as an explanation for the current situation. This isn't intended to be judgemental. And yes Steve, I really am procrastinating on finishing up those test cases.
Many people are familiar with these ideas already of course (and David Welton has also been trying to communicate a number of ideas from general marketing wisdom). But many others aren't. They may well disagree with the ideas of course, but at least it brings to bear a perspective that is not as dominant among the more technical community.
I'll draw a contrast first with the current open source darling. DHH describes Ruby on Rails as "opinionated software". By that he means that it's designed for one particular thing (quickly building mid-sized MVC database-backed websites), and it does that very well. It has its own ideas how that should be done. Keep within those ideas, and you get a lot of things done for you automatically - a big incentive. You can stray from that path, but it's more work.
Tcl is exactly the opposite of opinionated software - let's call it "unopinionated software". For the developers, flexibility is paramount. It's used in a ridiculously wide variety of environments, for a ridiculously wide set of tasks. Believe me, I'm not at all saying that's a bad thing. This flexibility also explains why for example there's been no standard OO system, and instead there are a hundred kicking around; the key is that the core system should never preclude new or different ways of doing things. The role of the core is to stay out of making decisions or having opinions. Perhaps this is no surprise coming out of Tcl's heritage and guiding principles: to build a core language that is easy to extend to suit specific needs.
Now, as to the "popularity" or "marketing" issue. There's a general dichotomy between designing something that is very flexible and can do many things (this should appeal to more people, so it'd be more popular) and something that is very finely focused (this may not appeal to too many people, but those who it does appeal to it really appeals to). General wisdom from the marketing world says that paradoxically, the second approach has a higher degree of making you popular (making you money, or whatever). Not guaranteed, and of course it's possible to take this too far. But if you consider flexibility vs. focus a spectrum, you'd place your bets more towards the focus side.
Why is this? After all, doesn't it seem appealing to more users would give you a better chance? There are all kinds of reasons relating to network effects, barriers to entry, adoption rates, etc. It's about having a niche and targeting it well. Jeffrey Moore talks a lot about this in the technical marketing classic "Crossing the Chasm". And you'll find some of the best wisdom in this area in Kathy Sierra's Creating Passionate Users blog.
Bottom line: if your goal is popularity (etc.) then doing one thing amazingly well will generally get you further than doing a lot of things ok.
That's where Rails succeeds so well. If it tried to be a general framework for all web development (or building any kind of application) it would make building the particular type of application it does so well now more difficult. It's just way too hard to do both. It very definitely has an opinion, and if you don't agree with it, you're probably better off elsewhere.
It might surprise people that Tcl was particularly opinionated at one point in its history as well, and that was in the early-mid 1990's. The subject was how to build GUI's for Unix/X11, which was the dominant "real" computing platform of the day. The received wisdom of the day was that C/C++ was the correct way to build GUI's, with its explicit event loop, long list of required parameters to widgets, explicit callback functions with many parameters, repaint handlers, etc. It was very flexible. Tk came along and said "no way"; scripting is the right way to do things, and we're going to have the system manage a lot of housekeeping for you (repainting a canvas, hit detection, event dispatch).
Tk was a solution to the problem of building X11 GUI's that absolutely delighted users, and with Tcl/Tk they had everything they needed right out of the box to do it. With a Hello World that was one line instead of 100, it was easy to learn and accessible. Is it a coincidence that period happened to be when Tcl enjoyed its greatest popularity?
So, turning to Tcl today, perhaps one can understand its popularity in terms of it not being an excellent solution to any one problem, but being quite good for a lot of different things. Notice I say "understand" its popularity. I'm not saying this is a bad thing. I'm only saying its a bad thing if the goal was to very popular. If so, it's evident that achieving that goal has been a failure. I would suggest that popularity has not been a goal, or at least a very secondary one. Maintaining flexibility and the wide range of uses the current system is being used for is a more important goal today for the core team. This has been I think a very conscious decision. Again, I put this out there as a fact, not a judgement one way or the other. These attributes are very important in terms of serving Tcl's existing user base.
Is it possible to both be flexible and really good at one thing? Yes, but it's really hard, and takes a ton of effort, and you're more likely to screw up both if you try than succeed at achieving both. So for the core group to focus two much on making Tcl amazingly good for one particular thing (which could help with popularity and attracting new users) might end up coming at the expense of the existing user base. There's no right answer, but I think the main thing is that this should be carefully considered. Again, I think in this case, it has been a conscious decision.
Is it possible, or even a good idea to change this? Probably not. Let's look at Ruby on Rails again. Ruby has been around for a while; it's a nice general purpose language, good for a lot of things, but was never particularly popular. Rails came along and used Ruby to focus on a particular problem, and that led to the current success. But it wasn't Matz and a bunch of core Ruby people who said "let's go come up with Rails as a killer app for Ruby". It was an outside group (DHH and 37Signals) who wrote that killer app, and for their own needs, not to boost Ruby's popularity. That did end up happening, as a positive side effect, but was not their goal. Killer apps come from outside core computing platforms. They're enabled by that good foundation the platform provides (which may well be very flexible), but they're outside users of the platform. It's hard for the core platform and team to "force" a killer app to appear, other than by making a platform that is as attractive as possible.
I think the main point is that Tcl is where it is as a result of fairly conscious choices, even though all the results may not have been clear at the time. Popularity is one attribute of the language that it may be possible to optimize for (to varying degrees), but as with most optimizations, it comes with tradeoffs and opportunity costs. The chosen path should reflect particular goals and priorities, and that may well come at the cost of lesser priorities. In the end, there's no absolute right or wrong; there's just how well you've met your goals.
