Computer scientists tend to reach first for usability testing and clinical evaluation when assessing hypertexts and hypertext systems. Other methods and approaches may yield better judgments
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Criticism
1. Tis hard to say, if greater want of skill
Appear in writing or in judging ill.
But, of the two, less dang’rous is th’
offence,
To tire our patience, than mislead our
sense.
Criticism
Mark Bernstein ❧ Eastgate Systems Inc.
Hypertext 2010
2. The Paper
“Our methods for
accumulating and testing
• 93 references
evidence of a hypertext’s
successes and shortcomings
• [Aarseth 97]-[Zaid 03]
are numerous but poorly
understood. This paper
• From 1711 to today
surveys the most influential
approaches to evaluating • From science to art
hypertexts and considers
their impact on crafting a • We should read more
hypertext criticism
new literary economy.”
5. Poets are partial to their wit, ’tis
true,
But are not critics to their
judgment too?
6. Lewis W. Hine.
Group of
newsboys on a
stoop at 4th &
Market Sts,
Wilmington
Delaware. "Take
our mugs,
mister?"
7. Photo: Lee Russell
An Economy of Judgment
The disappearance of newspaper and magazine review
Mark Bernstein And Diane Greco, “Designing A New Media Economy” Genre XLI 3/4, 2010
8. Lewis Wickes Hine. A "Reader" in
cigar factory, Tampa, Fla. He reads
books and newspapers at top of his
voice all day long. This is all the
education many of these workers
receive. He is paid by them and they
select what he shall read.
There is a lot to read.
The most salient economic fact about books: books are numerous.
9. How do we
discover what we
need to read?
✤ teachers, friends,
colleagues
✤ booksellers
✤ reviews in
newspapers,
magazines,
journals
✤ weblogs
Roosevelt reading in front of his tent in hunting
camp, 1910.
10. But there are
lots of
problems
For example, small literary magazines
receive thousands of submissions, but
have hundreds of readers.
Lewis Hicks Wine, Newsgirl, Park Row,
New York.
11. Whoever thinks a faultless piece to see
Thinks what ne’re was, nor is, nor ne’er
shall be.
12. What Is
Writing the paper: 80
Professor hours @ 100
$8,000
Presenting: 50 professors
x 1 hour
$5,000
Travel: $1000/
conference ÷ 18 papers/
conf x 50
$2,700
14. William McGregor Paxton, Leaving The
More Costs
Practitioners learn to despise all
research
Readers assume our work is
“academic” and uninteresting
The trade press fills with corrupt
pseudo-research.
Students leave the field
16. Trust not yourself; but your defects to
know,
Make use of ev’ry friend – and ev’ry foe.
17. Miall & Dobson 2006
✤ Take “The Demon
Lover”
✤ Make it into a
hypertext (how?)
✤ Ask a bunch of
undergraduates to
read it
✤ Run the statistics
✤ Write the paper
18. Moulthrop 1991
To require hypertext
to function like a book
is a bit like expecting
a jetliner to behave
like a locomotive: yes,
it’s very fast but the
blasted thing won’t
stay on the rails.
19. Evaluation and
its discontents
Usability privileges the
first encounter
Statistics wash out the
exceptional case (when
not discarded as an
outlier)
Unrealistic definition
of “work”
20. “While the results
were not
statistically
significant, we
note nontheless
that the trend was
strongly…”
24. Who to a friend his faults can freely show,
And gladly praise the merit of a foe.
Real
riticism criti
Re al c of
cism
of
r eal real
xts! hype
hyp erte rtex
ts!
25. Who to a friend his faults can freely
show,
And gladly praise the merit of a foe
26.
27. Who to a friend his faults can freely show,
And gladly praise the merit of a foe.
✤ Strength
✤ Flexible, concrete
✤ Weakness
✤ Partiality can be masked
✤ Small samples, hard work
28. William McGregor Paxton, The Breakfast, Metropolitan n Museum
Most critics, fond of some subservient art,
Still make the whole depend upon a part
Essentialism
29. Most critics, fond of some
subservient art,
Still make the whole depend upon
a part.
33. Murray
“When we stop thinking of the
computer as a multimedia
telephone link, we can identify its
four principal properties, which
separately and collectively make it
a powerful vehicle for literary
creation. Digital environments
are procedural, participatory,
spatial, and encyclopedic.”
34. claptrap
✤ Essentialism run amok: the
essence of computational
media is that they are
pernicious (and so we need
not — should not — read
them)
✤ Believing hypertext to be
pernicious or harmful, these
critics read little (Miller,
Kakutani) or almost none
(Birkerts)
35. William McGregor Paxton, The Breakfast, Metropolitan n Museum
Strengths
you don’t need to work very
hard
Weaknesses
you don’t learn much
Essentialism
37. Our sons their fathers’ failing
language see,
And such that Chaucer is, shall
Dryden be.
38. It was Kael's therapeutic advice to the
overcultivated that if they just
concentrated on responding to the
stimulus, the aesthetics would take care
of themselves. What good is form if the
content leaves you cold?
The academic term for the kind of
antiformalism Kael promoted is
“postmodernism.”
Louis Menand, 1995. Finding It At The Movies. The New
Yorker. 42 (5)
45. In ev’ry work regard the writer’s end,
Since none can compass more then
they intend; And if the means be just,
the conduct true, Applause, in spite of
trivial faults, is due.
54. thank you
✤ Pope On Criticism (1711)
✤ Stacey Mason, Samantha Panepinto
✤ Fonts: Neutraface 2, Epic, Kane,
Tungsten
✤ Images: Library of Congress,
iStockPhoto, MFA, Metropolitan Museum
www.eastgate.com serious
of Art.
hypertext since 1982
Notas del editor
My name is Mark Bernstein. I build hypertext systems, and I’m a hypertext publisher. This paper reviews approaches to hypertext criticism. The talk is challenging; the paper is easier.
The paper takes a sincere shot at broadly surveying hypertext criticism and the many approaches people have taken to figuring out what is good and what is not. I believe this to be the first review of the area.
A big part of what we do in hypertext research is, simply, figuring out whether the stuff we’ve done is any good.
Those 93 references aren’t padding. Lots of people have reviewed hypertext. I disagree with some, but read and form your own opinions. The strength of this conference had been its scholarship, and that scholarship is not what it was. You need to know the literature.
If the idea of literary hypertext seems strange to you, Diane Greco and I recently pulled together a volume of readings about Reading Hypertext.
We make judgments when building our systems, and when writing hypertexts, and when designing Web sites, and when submitting papers to this conference.
Literary machines inhabit a literary economy. I outline the shape of that economy in the paper. You know the essence: Writing, publishing, bookselling.
Everyone thinks their own work deserves to be rewarded. There is nothing outside the economy; open source is not a stairway to heaven. And we claim to be a science; we want evidence and we’d like proof that we know what we know.
I won’t recapitulate the whole discussion of the economy. The most salient fact about books is that books are numerous. And their plenitude is precisely what we want to preserve and expand in hypertext; we want to build things that are better than books.
How do we know what we know? Most of you are computer scientists, and computer science has an answer. Some of you may be writers, or scientists (who after all are writers) and Literature has its own answers.
I want to suggest today that it’s all wrong. We’re rejecting work that might be good, and accepting work that might be wrong.
Choosing unwisely is an expensive proposition.
All of us have sometimes reviewed conference papers that were, frankly, hopeless from the outset. Sometimes, we’ve heard them presented. Maybe this is one of them.
A bad conference paper costs taxpayers and benefactors fifteen or twenty thousand dollars — and that assumes it’s not pernicious or misleading.
Of course, the costs can be much greater. We take a paper that’s not QUITE right, and people rely on it, and cite it, and pretty soon we’ve got entire research projects that are motivated by a mistake.
This has happened several times at this conference, where (after all) we don’t build bridges.
When we misjudge research and writing — in either direction — we tempt practitioners to despise all research. Readers assume that we can’t advise them. Worse: that we don’t want to. The trade press fills up with corrupt pseudo-analysis. And students stop doing the work. THIS SHOULD SOUND FAMILIAR TO YOU.
We need to know what it to read, and we need to know how to write hypertexts worth reading. How can we tell? The favorite tool of computer science has been empirical evaluation.
We implement a system of some sort, and then we test that system on actual test subjects.
We can, for example, take a print story and turn it into a hypertext. We ask a bunch of our students whether they like the jumbled story or the real story. They give us the answer we want; and we get a publication.
Drag races between media are almost always a bad idea. You can’t measure the average productivity difference between an opera and a bronze statue. WE HAVE UNDERSTOOD THIS SINCE 1991, and yet we see these papers every year.
Evaluation is sometimes useful, but often leaves us adrift. First, it privileges first and casual encounters, which are not really our first interest. Second, the really important outcomes —changing the world — are by definition rare, especially with undergraduates! Finally, we keep defining “work” as “finding facts” or “filling out forms”; this is easy to measure but not in fact what knowledge workers do.
We ourselves don’t do evaluation very well. I review a paper with this sentence almost every year. So do you. I’ve always thought this was self-evident grounds for instantly rejecting the work. Not all of you agree, though I’ve never heard a convincing defense.
Let’s face it: we insist the papers have evaluation sections, but we don’t love reading them. When we meet, we don’t rush to talk about our latest tricks for recruiting test subjects. It shows in the papers: the statistical sections are often ill-written, stilted, and artificial. And it’s equally hard to get referees to pay close attention.
We insist that it be done, but it’s obviously not what we want to do. This is not determinative, but it is (I think) a hint.
Summarizing quickly and oversimplifying: evaluation keeps us from deceiving ourselves, but it threatens to turn us into a conclave of pettifoggers, refining details that nobody cares for.
A variation that designers sometimes hold out: we’ll ask the users what they want.
Readers dont know what they want. They haven’t found it yet; that’s why they’re reading!
So, perhaps there might be an alternative to applying Student’s t-test to a room full of our students.
Others, instead, have turned to careful examination of actual hypertexts.
There’s quite a lot of excellent work along these lines, examining actual hypertexts and exploring what they do and how they do it.
Notable among this group was a cadre of young European and Oceanic scholars, many of whom, interestingly, now write much less than they did ten years ago. This is a problem.
Criticism can potentially answer any question we care to ask it. We are inclined to distrust it, but often find it gives the right answers. But our distrust is not unsound; partiality can be masked, and induction from a single sample is hard to trust.
Another approach — one that has been very popular amongst our literary colleagues — has been to explore the essential properties of digital media.
Discussing specific hypertexts, one by one, is hard work. Can we draw any lessons, beyond respect for isolated genius? It seems more attractive to identify general tendencies in the nature of hypertext.
Often, these take the form I’ve called “Kids Today”, deploring the ever-declining abilities of the latest technology-addled teens.
This never changes and has no content: its essence is that it is – precisely – essentially irrelevant.
There is better work essentialism, but it has proven surprisingly difficult.
In his “Golden Age” speech. for example, Coover retreats from a modernist examination of actual hypertexts to argue from the Web’s apparent propensity for superficial imagery. He has no particular evidence for this, and I think it was wrong, but since Coover says it can’t be done, a generation hasn’t tried very hard to do it.
Or take Hamlet On The Holodeck. Murray says digital environments are procedural, participatory, spatial, and encyclopedic. But this isn’t an empirical observation; it’s an argument from the supposed essence of computation. This is fine when the essence is necessarily true, as is is for Joyce in “Nonce Upon Some Times”. But here we can easily quibble: is “Lust” encyclopedic? Is “afternoon” procedural? Is Prolog?
Finally, an important thread of essentialism begins from the conclusion that new media are pernicious. Since hypertext is harmful (especially to kids), these critics tend to avoid reading much of it.
I’m being hard on essentialism here. In fact some of my essentialist colleagues have done some hard work, and we have learned something. But I think essentialism can take a vacation.
Essentialism is popular in those parts of the digital humanities that like to talk about the essence of the digital. It’s largely discredited, and computer science it is especially unpopular.
The reaction against essentialist modernism in the arts is called postmodernism, and it is with this reaction that hypertext is chiefly identified in the humanities.
It was Kael's therapeutic advice to the overcultivated that if they just concentrated on responding to the stimulus, the aesthetics would take care of themselves. What good is form if the content leaves you cold?
The academic term for the kind of antiformalism Kael promoted is "postmodernism."
Formality considered harmful. Spatial Hypertext. LITERARY MACHINES. All postmodern arguments.
But, absent form and power, it hasn’t been clear how Theory (with a capital T) can be wrong on the merits, not just the politics.
A number of critics have embarked on a reflective practice, essentially auto-ethnography. They read and, reflecting on their reading, try to figure out what’s happening in front of them.
We can record our reading of a hypertext, our reaction to it, our theories about what is happening. Jane Douglas pioneered this method, Jill Walker won the first Nelson award for it; we don’t do it anymore.
The same thread runs through much of the best new media criticism: looking both at the work and at its experience. Think for example of Marshall’s thoughtful studies of annotation, Efimova’s reflections on blogging about writing a dissertation on weblogs.
Autoethnography — which is to say, intelligent and critical reflection on your own system — can yield intelligent reflection that would probably be beyond the capacity of a bunch of undergraduates working for a pizza. The potential for bias is clear. More important, I think we can sometimes find self-criticism too convincing. Randy Trigg talked us out of link types in 1989, and we believed him
Though I’ve been hard today on the Essentialists and the Evaluators, all four threads contribute to help us know what hypertext to read and to explain how it works. We can practice all four to good effect.
Let me conclude with some guidelines about how this new economy of judgment could be most helpful.
I return again to artistic and political practices of the late 19th and early 20th century radicals from Ruskin to the early Bauhaus, practices I call (without regard to nationality) NeoVictorian. The key concept that runs throughout these disparate movements is the integrity of the artist and the work. As a field, we have affected to honor these ideas and have not always succeeded as well as we ought.
First, we need to get our hands dirty. We need to read hypertexts and we need to read about them. We need to use hypertext tools in our daily work. Here, for example, are some notes on my recent reading from Tinderbox, a hypertextual tool I designed for making and analyzing notes. We should writing new media. We should demand excellent new media from students. You can (and must) understand computers now.
In my view, a great fault of digital humanities today is that there are few consequences (and many rewards) for being wrong. Indeed, being spectacularly wrong gets you press coverage and party invitations. Further, it is one thing to be mistaken and another, entirely, to be wrong and know it. In the sciences, we call this fraud; until you get this right, many will consider your discipline contemptible.
We do not wish to return Art to the patron, the prince, and the priest. It is easier for a critic to pin a fault on a student than to censure a colleague. It is also wrong. It is easy to assign nefarious motives to corporations and to sneer at tradesmen. Absent evidence, it is also wrong. Capitalism may be foul, but it is not our fault.
Convincing critical appreciations approach the hypertext in detail, yet draw on muriad sources and ideas. Sympathetic reporting of what was missed — what, as Harpold suggests, the hypertext turns away from — helps establish the critic’s position, and candor.
Failure can be as interesting as success. Many of the best readings of hypertexts begin from incomprehension and confusion. I hated afternoon the first time I read it; I thought hypertext fiction preoposterous.
The critic may be wrong. Take Lust: I edited the story, and I know it fairly well. I’ve read it dozens of times. Yet one of Rich Higasson’s students’ comments, reported in his dissertation, reveals my reading was wrong.
We should expect to write seriously about actual new media throughout our careers, not only as as the outset.
Special thanks to Eastgate editor Stacey Mason, whose survey of criticism formed the core of this review, and of course to all of you who build hypertext systems and write hypertexts. Where, by the way, are the hypertexts at this conference? Thank you.
![The Paper
“Our methods for
accumulating and testing
• 93 references
evidence of a hypertext’s
successes and shortcomings
• [Aarseth 97]-[Zaid 03]
are numerous but poorly
understood. This paper
• From 1711 to today
surveys the most influential
approaches to evaluating • From science to art
hypertexts and considers
their impact on crafting a • We should read more
hypertext criticism
new literary economy.”](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)