Random insight of the night: every couple years, someone stands up and bemoans the fact that programming is still primarily done through the medium of text. And surely with all the power of modern graphical systems there must be a better way. But consider:

* the most powerful tool we have as humans for handling abstract concepts is language
* our brains have several hundred millenia of optimizations for processing language
* we have about 5 millenia of experimenting with ways to represent language outside our heads, using media (paper, parchment, clay, cave walls) that don't prejudice any particular form of representation at least in two dimensions
* the most wildly successful and enduring scheme we have stuck with over all that time is linear strings of symbols. Which is text.

So it is no great surprise that text is well adapted to our latest adventure in encoding and manipulating abstract concepts.

@mdhughes vis-a-vis terrible languages, I mean Sturgeon's law predicts they are gonna be there. But if it's easy to make languages, I'd suggest you'll get more good ones too.

@rafial Even if 90% of languages are crap, which seems lowball, it's peculiar that the popular ones are all crap. Programmers are supposed to be "smart" but it looks a lot more like doubling down on whatever you get by random chance.

@mdhughes local minima are a challenge, and network effects swamp anything short of order of magnitude improvements. Also, most people are just trying to get shit done, and global variables and hack til it works gets things done just fine at the scale most day to day problems actually exist at.

@mdhughes if your need is to hire reams of fungible low paid devs to crank out "good enough" one-off business websites, PHP is a cracking success.

@rafial Sure, and a few million security holes that let h4xx0rz take credit card numbers and passwords is just a bonus!

@mdhughes @rafial this situation arises because of competing standards for what's "good". For people learning to program, good means cheap/free, accessible, easy to set up, has lots of examples freely available, has discoverable 3rd-party libraries and frameworks, and offers a responsive and helpful community

For businesses, good means "I can easily and inexpensively find programmers to maintain it"

Those criteria tend to be overlooked when comp scientists and expert engineers design what they consider "good languages". So many good designs have been killed by gatekeeping or snobbery.

@mdhughes @rafial Consider that the most popular cars are not the "best" cars; they're the cars that best meet the needs of the masses, which include affordability and ease of repair

Same basic idea: a language that's easy to hack stuff together with serves more people's needs than an elegant and expressive language, even if the latter is better in every technical manner

@calcifer @rafial Scheme, to pick my weapon of choice, is both fast, elegant, and hackable, including C FFI, better than anything I've had since Objective-C. Neither are "popular", and in fact are feared by low-grade codemonkeys.

PHP is… "designed" is the wrong word, but has randomly evolved to be an existential risk every time you paste some URL argument into a SQL query.

Popularity follows inverse quality in almost every case. It's as if management wants the apocalypse.

@calcifer @rafial Phrases like "gatekeeping" and "snobbery" are "sus af" as the kids say. We do want gates to keep out the incompetent. We do want snobs using taste to choose the better solutions.

@mdhughes @rafial we don't want gates to "keep out" anyone. No one should stand at the gate of capability or knowledge and say "you shouldn't learn this". We want to provide mentorship and education, not tell fledgling developers "you're not worth talking to until you learn a 'good' language, and oh by the way if you can't learn it without accessible manuals and guides then you aren't worth teaching". THAT is gatekeeping.

@mdhughes @rafial We don't want snobbery. We don't want people who look down on others because they disagree with or don't understand our choice of tools. We want people who have taste driven by experience, yes. But not people who will say "oh this language doesn't fit the way you think? Then you're not a real programmer". That's snobbery.

And incidentally, it's my experience that snobs are, the majority of the time, on the short end of Dunning-Kreuger. So pattern suggests that if someone is language/paradigm/toolchain snob, they probably shouldn't be listened to.

@mdhughes @rafial you want good languages to get popular? You have to understand "good" isn't good enough. They need to be made accessible. The toolchains have to make sense to younger programmers. The community needs to be welcoming and supportive. The entire ecosystem has to solve problems you can articulate to people who are not masters of the art. The system has to play well with the problems people want to solve.

The "bad" languages are successful because they addressed the people part of the equation better than the "superior" languages.

@calcifer @mdhughes @rafial
I can't see earlier posts in this thread, but I have years of stories about the Haskell community that boil down to exactly that.
1. welcoming community
2. solves real world problems
3. plays well with other languages
4. decent tool chain

@calcifer "We"? You have a mouse in your pocket? Your opinions are yours alone, don't misrepresent them as some mob behind you.

Education has been tried, and the demands of hordes of unqualified children have driven it from reasonable languages like LOGO (for small children), Pascal, C, and Scheme, to Scratch, Java, and Python so they'll "be ready for enterprise" rather than learn anything hard.

The world is better with *less* bad software written by the unskilled.

@mdhughes I do in fact have a mouse in my pocket. His name is Euclid.

I also know when someone is has an opinion they're not willing to reexamine because they are convinced they must know better than everyone. There's nothing but a silly fight to be had like that, so I'll leave you to your tower and you can leave me to my little research dungeon.

@rafial Both accurate and also misses the fact that Excel is REGULARLY misused for scientific calculations and near-programming level things since its GUI is so intuitive for doing math on things.

Like, GUI programming is HERE, we just don't want to admit it due to how embarrassing it is.

@Canageek very good point. Excel is actually the most widely used programming environment by far.

@rafial Now what we need to do is make a cheap, easy to use version of it that is designed for what scientists are using it for it. Column labels, semantic labels, faster calculations, better dealing with mid-sized data (tens of thousands of data point range), etc

@Canageek I'm wondering, given your professional leanings if you can comment on the use of "notebook" style programming systems such as Jupyter and of course Mathematica. Do you have experience with those? And if so how do they address those needs?

@rafial I have not, I've done math and such in Excel (making a molecular weight calculator, sheets to automatically work out student marks by letting me see which step of a calculation they got wrong, etc) and I've done actual programming (A little python, C, C++, heck, QBASIC back in high school and a tiny bit of FORTRAN90 one summer). but not anything in between.

I can't figure out the use of them? They seem like the worst of both worlds. You have to debug python AND you have to deal with a slow loading GUI program

@Canageek I'm very interested in that space, in so far as it seems to intersect with the ideas of Don Knuth's literate programming. But I also admit to be slightly unclear as to what domains it is best for. I think it's big with the data science crowd?

@rafial Yeah, I'm a chemist who works with very small data, no statistical analysis or anything like that. About the most I have to do is "Adjust this data to account for the lamp response on that day" or "Normalize and scale these two spectra against one another"

@Canageek if I had to guess, I would say if your domain involved exploration of data sets, with visualization as a key component of that, the notebook things might well be a killer app.

@rafial @Canageek I've done a fair amount with Python notebooks when that first happened, a little with Julia notebooks.

It's hard to make a coherent program in them, it's a long series of fragments that get run in arbitrary (creation) order.

But as a mix of documentation and calculation, it's super useful. I'd rather work in a REPL (and Julia has a fantastic REPL), but notebooks are better for persistent math and proving your results.

@rafial @Canageek You absolutely need a beefy server, fast connection (local if possible), and fast front-end CPU to make notebooks usable. They're a massive increase in resources needed, but they're one of the few "this is better than what we did before" technologies I've seen in 40 years.

@mdhughes @Canageek interesting to hear. And Julia is definitely on my stack of "stuff I need to learn more about."

@rafial @Canageek There's annoyances with #Julia: Startup time is many (5-30) seconds, indexing starts at 1 unless you do crazy things to "fix it", docs can be confusing/nonexistent.

But it has a great method dispatch/object model which is not just "static Smalltalk" like most, it's super fast at runtime, and like I say the REPL is amazing.

@mdhughes I mean for those who, unlike us, didn't come up through C and "indexing is really pointer multiplication", index base 1 kinda makes sense, nah?

"Should array indices start at 0 or 1? My compromise of 0.5 was rejected without, I thought, proper consideration."
—Stan Kelly-Bootle

@urusan @rafial Whomever wrote this is an idiot that has never worked in actual physical sciences where you have actual physical experiments, just FYI. Like 10% of my labs works involves computers at all. But sure, science now moves at the speed of software.

@urusan @rafial It totally ignores the advantages of PDF though, like the fact there are a stack of independent implementations that can view it, which means we will still be able to read these files in 50 years, unlike whatever format they are using, and we can even print them out on paper to edit them (for example, how my boss and I do it, as he doesn't know LaTeX, which is what I write in).

Or the fact that like, 90% of scientists don't know how to program and are unlikely to learn.

@urusan @rafial In my lab right now... I can program a little, one post doc has spent some time doing python tutorials, and I think that is it, out of ten people.

Organic labs might have less then that, as there is a reputation at least that organic chemists and programming are like oil and water: if you are good at one, you are likely terrible at the other.

@urusan @rafial You public the source files as ESI the way we do with input files in crystallography?

@urusan @rafial (I might be giving opinions different then normal due to spending two days trying to get this figure to compile and dealing with moving to lualatex from pdflatex as a result)

@urusan @rafial Right but then you are stuck with a system 0 people understand instead of 1 person.

If I wanted to use GNUplot as it was built I could use the script a former grad student passed on, but I want it in LaTeX so the fonts and such match.

(Honestly, if I could go into campus it would be done, but the software we have that makes these graphs is trapped on one computer, and not worth me going into campus during a pandemic to get, at least not yet)

Honestly, if you are going to try and replicate any of my work you aren't going to go to my data: You are going to synthesize the compounds yourself and take the measurements on your equipment against your standards so that it doesn't turn out to be some dumb different between how I set up my experiments and how you do, or some defect in my hardware, etc etc.

@Canageek @urusan @rafial > 90% of scientists don't know how to program and are unlikely to learn

Do you think that's stable, or does it really mean "90% of old scientists"?

I probably listen to biased sources, but my impression is that basically any field doing any amount of statistics these days does it in Jupyter.

If that counts as "know how to program", of course.

@clacke @urusan @rafial Well yeah, but most scientists don't do statistics. Most chemists, most biologists, geologists, etc.

Like, there is a reason computational is a subfeild of every discipline.

I think it is going up, due to more stats and computations being used, but I also think we are way to reliant on stats these days and use them instead of getting good data.

@urusan @clacke @rafial Right, I've been frustrated with this in science for decades. We should do half as many studies and put twice or more as much funding into each one so we have actually decent stats.

For example, lately you have to justify the minimum number of rats for ethics committees for any experiment. Fuck that, use 4 times as many so we can be confident in our work instead of justifying it to heck and back.

@urusan @clacke @rafial (This is part of why I picked a field where all the stats were baked into software and validated before my Dad was born, and the sample sizes for everything else is around 10^26 (ie the number of atoms in solution)

@urusan @clacke @rafial Nope, its counted as bad, but justifiable so you have to minimize the number you use, at least as I understand it.

Likewise, academic human studies are typically very underfunded, which is why there is such a bias towards small sample sizes and all the participants being undergrads found on campus.

@Canageek @urusan @rafial A Swedish stand-up comedy musician has a routine "Ain't it Weird" about modern life. The very first segment goes:
> Listen, my grandpa was a mason, from Borlänge, Gösta was his name, construction worker, he used to say this:
> "In my day, we made an honor out of building houses as strong as possible, so they would last as long as possible. But now, they have computers that calculate how weak they can build a joint without it falling in on itself."
> *ding*
> Ain't it weird?
Peter Carlsson & Blå Grodorna: "Än'te könstut?" (Dalecarlian Swedish)

He's wrong, but he's not wrong.…

@clacke @urusan @rafial I've heard from engineers that anyone can build a bridge. And engineer can build a bridge that only JUST carries the required load to make it not cost a fortune

Thanks @urusan, I found the article interesting, and it touched on the issue how to balance the coherence of a centrally designed tool with the need for something open, inspectable, non-gatekept, and universally accessible.

PDF started its life tied to what was once a very expensive, proprietary tool set. The outside implementations that @Canageek refers to were crucial in it becoming a universally accepted format.

I think the core idea of the computational notebook is a strong one. The question for me remains if we can arrive at a point where a notebook created 5, 10, 20 or more years ago can still be read and executed without resorting to software archeology. Even old PDFs sometimes break when viewed through new apps.

@rafial @urusan Aim for longer then that. I can compile TeX documents from the 80s, and I could run ShelX files from the 60s if I wantd to.

@Canageek @urusan oh hey, I'm all for longer, eventually we'll need centuries. But I've also seen the hair pulling it can take to revive archived source code from even a couple years back, and realizing we've got to start by getting to the point where even a decade is a reliable and expected thing.

@rafial @urusan Fair, though I'd say source code is pointless and what we need is more focus on good, easy access to the raw data.

If you can't reproduce what was done from what is in the paper, you haven't described what you've done well enough, and redoing it is better then just rerunning code as a bug might have been removed between software versions, you might notice something not seen in the original, etc.

@urusan @rafial That would be my favoured approach. Raw data plus a good set of standard statistical tools. More basic analysis = less care if the EXACT toolchain is lost.

If you can't redo the analysis in alternatives then it isn't good science in the first place.

Sign in to participate in the conversation

A bunch of technomancers in the fediverse. This arcology is for all who wash up upon it's digital shore.