On establishing a web archiving program

Earlier this week, I drove to Davenport, Iowa to attend — and speak at — the Upper Midwest Digital Collections Conference. Trevor Owens of the Institute of Museum and Library Services delivered a wonderful keynote, and others followed his remarks by discussing digital projects — audio digitization! community archiving! — happening throughout the upper Midwest.

I attended to discuss the web archiving work I’ve been doing on behalf of the Madison Public Library (and, as the conversation evolved, I also spoke a bit about the work I’ve been doing at the University of Wisconsin-Madison Archives to document #therealuw). The talk was called You’ve Decided to Establish a Web Archiving Program. Now What? with hopes that I could demystify that odd space that comes between deciding to start a web archiving program and actually crawling sites.

The talk was broken into chunks: an introduction; a brief overview of available programs; and seven questions to ask of your organization / your employees as you start your program. And because my slides — as always — are quite bare, I thought I’d expand here, in writing.


I used the introduction to urge professionals to give up on perfection. (My actual words were: the perfect web archive does not exist.) The web, as it stands, moves too fast and contains too much content to ever build the perfect web collection.

I should say: I’m not urging you to half-ass (sorry) the development of your collection(s), but rather to think critically and be creative as you build. When an institution assembles a physical, analog collection, there’s no expectation that — at its beginning — the collection is going to contain everything related to its subject, right? And yet, we think about web archives differently — I think — because it seems like just about everything is right there, ready for the taking.

Strategic, adaptive collecting is one hundred times better than mindless, let’s-grab-it-all collecting — and that’s true for any format.


As far as platforms go, I suggested two: Archive-It and webrecorder.io. There are other options, certainly, but these are two that I have used before and trust.


The rest of the presentation focused on these seven questions I’ve developed, which, quite honestly, are things I asked of my own organization (and myself) after we were knee-deep in creating our web collection. I like the collection we’ve established, and I think it’s going to serve Madison’s community well, but I do wish I had tried to answer these questions before we started because it would have streamlined the appraisal process and, right off the bat, made our collection a bit more focused.

But, live and learn, right?


1. Who is the archive’s target audience?

This question comes easy to a lot of organizations, and that’s great. Establishing the Library’s collection threw me for a loop, though, because the Library serves such a diverse user group. There are traditional library patrons, who use the Library’s books and media and computers; there are the researchers, who use the Library’s reference services and research material and local history room; there are artists, who use the Library’s makerspace and exhibit spaces; there are community members who use the Library’s physical meeting rooms; and on, and on. This is not to say that this doesn’t happen at historical societies and universities, but I do think this happens on a larger scale at public libraries.

In order to focus the Library’s collection, we really had to think about who might use it and why. Researchers seemed to be the obvious answer — much of archive is a sum of sites that reflect digital versions of material previously in print and local reference collections. But what about more traditional library patrons: would they use it? And employees and residents of the Library’s makerspace: could they incorporate it into their lessons and workshops?

Our collection caters heavily to researchers, and that’s fine, and we’ll see if it works. We could have, just as easily, directed the focus elsewhere. But thinking about this question allowed us to narrow our focus — if a site we wanted to collect had little to no (perceived) research value, was it worth collecting? (And I know, I know: who knows what we’ll end up using for research. But, like I said earlier: perfection doesn’t exist and, honestly, we’re guessing.)


2. How will people access the archive?

When I brought this question up, it threw my supervisors for a loop, but: some web collections — like at the Library of Congress — contain material that is restricted, and can be accessed only on site. Some don’t. And that changes things a bit, I think, because it adds another level of thought: if, say, the Library wanted to include restricted sites, how would we make that work? Who on staff would monitor use of the archive? Who would patrons call to set up time with the archive? Do we have the equipment (a spare computer, let’s say) to create this kind of setup?

You see what I mean. It’s not a huge question, but it’s something to consider.


3. How will your organization promote the archive?

Don’t do a ton of work establishing a web collection and then let it go unused. Collections of this nature are not (yet) intuitive. Think of how often you tell someone you’re an archivist and they say, “What’s that?” and you jump right into your elevator pitch. Now think about how you’ll answer when you talk about web archiving and someone says, “Wait, what?”

People don’t get it. And that’s okay. But there’s a huge literacy gap between We’re creating a web archive and But stuff on the Internet lasts forever! So, promote the archive. Don’t bury the link to it on your institution’s webpage. Make fliers. Hell, offer workshops and lectures on web archives. (I’ve done the latter twice as part of a larger digital preservation workshop. In both sessions, I’ve instructed users how to download their data from Facebook / Twitter / Gmail and done a walkthrough of the Internet Archive’s Wayback Machine. Remember: just because everyone in your circle of coworkers and friends understands the need to archive the web doesn’t mean your users do.)

This is about getting the word out. Do it. Get people to use the collection you’ve created.


4. Has this work already been done?

This is one of my favorite questions. Madison Public Library is within walking distance of the Wisconsin Historical Society, the University of Wisconsin-Madison Archives, the Wisconsin Veterans Museum, the Madison Museum of Contemporary Art, the Chazen Art Museum, the Wisconsin Historical Society Museum, and on. And I know for certain that two of these places — the University of Wisconsin and the Wisconsin Historical Society — actively practice web archiving.

If the Wisconsin Historical Society is already crawling the blog that belongs to the Governor of Wisconsin (and their collection is publicly accessible and discoverable), why should the Library do it, too? And if the University is already crawling everything associated with wisc.edu, there’s no need for the Library to grab that domain, too.

This is both an ethical and practical thing for me. Practical because it’s going to save you money and time and needless maintenance. These collections are available online. And if a user of the Library’s web archive walks in and says, Hey, do you have the wisc.edu archive? it’s remarkably easy to say, No, actually, but here’s the webpage for that collection.

Ethically: the Society of American Archivists has a code of ethics. In that code of ethics, there is this statement: “[Archivists] collaborate with external partners for the benefit of users and public needs.” And this one: “Archivists cooperate and collaborate with other archivists, and respect them and their institutions’ missions and collecting policies.

We’re all so much better if we collaborate.


5. Can your organization afford the archive?

Listen: web archives cost money, whether you sign a contract with an external provider, or do it on your own. Sit down and create a realistic budget for your web archive and make sure that you can fund it not just this year, but next year, and the year after, and even the year after. A web archive deserves to be sustained. And that requires money.


6. Can your organization sustain the archive?

Listen again: web archives require labor. Labor done by a human. Labor done by a human who works as an employee at your institution. Labor done by a human who works as an employee at your institution on a frequent basis.

You cannot set and forget a web archive. It’s tempting, I know, because who has room for one more thing on their plate, or in their job description? But web archives require maintenance, and someone to monitor crawls, and someone to continually evaluate the sites that you are — and are not — collecting, and someone to promote the collection, and someone to teach patrons to use the collection. Just as physical collections require similar maintenance.

It doesn’t have to be a forty-hour a week commitment, but it should be, you know, more than a zero-hour a week commitment. Find a balance, and adapt. But do it. A good web archive is a cared for web archive.


7. How will you assure that the web archive remains inclusive?

This is the most important question. And it probably deserves a post of its own, but:

I was the sole curator — what a terrible word — of the Library’s collection. I am also white, well-off financially, college-educated, and on. My perspective — as hard as I try otherwise — is reflected in what I choose to collect and leave out. And for a collection that needs to serve the entire Madison community, this is not okay. So, recognize this. And then do something about it.

What I did? Went outside of the Library.

I did speak with Library employees. And the Library’s board. And the Friends of the Library. And then I kept going. I spoke with teachers and faculty at the University, and I spoke with classmates of mine, and students I had never met. I spoke with community leaders, program managers, business professionals, and workers. I asked other archivists, at other public libraries, for their opinions. And I’m still asking people these questions: what represents Madison to you? Where do you work? What do you do for fun in Madison? What are you involved in within the Madison community? Where do you get your news? What neighborhood do you live in? Where do you post your creative work, and your words?  What sites did you consult before deciding where to live? And on. These aren’t difficult questions, but they — I hope — get at the heart of what makes the community.

I also consulted a lot of other resources: online neighborhood lists, city-sponsored community event calendars, newspapers and alt-weekly newspapers, fliers taped up around the city, radio ads, protest signs, advertisements on television, etc. If I heard or read something in Madison that led me to a website, I looked at the website.

It’s not about having the right answers, but about seeking them. And so often — when you assemble any type of collection, when you embark upon any project — these answers come from outside of yourself. And that’s good, and that’s important.

Do not shirk this step. And, even when you think you’re finished with it, keep going. This one is constant and never stops.

Let your collection evolve and grow, and do it by asking for help.




On personal archiving labs

Hi, WordPress. It’s been a while.


So, the Personal Archiving Lab at the Madison Public Library has been up and running for about two months now — and the response has been many things: wonderful, certainly, and overwhelming. (I’ll explain.)


I staff the Lab, alone, on Wednesdays, for five hours at a time. The Lab is also accessible throughout the week by appointment. Right now, today, our appointments on Wednesdays and throughout the rest of the week are booked solid for the foreseeable future. To meet current demand, the Library would honest-to-God have to hire a full-time librarian and pay them to do nothing but help people use the Lab. And that is, for many reasons, not realistic.

I will expand more upon this later, but: for now, the Lab has no permanent home within the Library. Fortunately, the Madison Public Library — and its new building — are big and open, with plenty of spaces that we can borrow while people use the Lab. Someday soon, the Lab will find its permanent home in the Local History room, but that room is currently in use. Hence the Lab’s by-appointment status. Before the Lab officially launched, I spent a bunch of time creating documentation for each piece of equipment available for use — and included in the documentation are step-by-step manuals that should allow users to work on digitization projects without constant guidance. Each manual contains pictures and text that correspond with every step, and they begin with ‘First, turn the equipment on,’ and end with, ‘Now, turn the equipment off.’ Once the Lab docks permanently in the Local History room, its hours will expand and the Library should be able to accommodate more people over the course of a week, a month, etc.


I should say: the Lab is tiny. As a professional, I hear the word Lab and think big — and I don’t know why. Media labs, makerspaces, computer labs. This Lab fits on a cart, like this one, and is fully mobile. (In theory.) The only piece of equipment bolted semi-permanently to the actual cart is the Lab’s computer, which is also the Lab’s most expensive piece of equipment.

In addition to a Macbook Air, the Lab contains a flatbed scanner (this one, which can handle poster-sized documents as well as slides and film negatives), a Sony HandyCam (which are reasonably-priced on ebay), a portable miniDV player (here’s one), a tape deck, a combination VHS / DVD player (get ’em while they last), and a floppy disk drive.

Oh, and. The Lab uses Elgato Video capture for everything except the scanner, tape deck, and floppy disk drive.

In a perfect world — or, someday, with the help of a grant — I’d love to see the Lab add a second (or third) computer (more on this later). Maybe something to handle Betamax tapes. A record player, for sure. But, as is, the Lab can currently convert: photographs, film negatives, slides, Hi8 tapes, miniDV tapes, cassette tapes, VHS tapes, and floppy disks. (We do get a lot of questions about 8mm film, which we just can’t do. It’s far too cost prohibitive and the risk of ruining someone’s media is much higher.

Speaking of: each and every time someone uses the Lab to digitize something from home, we absolutely require that they sign a waiver indicating that the Library cannot, and will not, be held responsible for any damage that occurs during the digitization process. Old media is, well, old. And there’s always the risk.)


In all honesty: the size of the Lab is — often — limiting. Sometimes, when we’re busy, I’ll run to the reference desk and check out a second library-owned computer in an effort to help two patrons at once. This sometimes works, and sometimes it doesn’t.

The Lab’s Air has all the necessary digitization software installed already, while other library-owned computers do not. Library-owned computers work great for patrons only interested in, say, using the scanner, but when two individuals are interested in converting Hi8 tapes, we’re out of luck. It’s one of the many reasons that the Lab is by-appointment only, in small chunks, and one of the many reasons that the Lab has pretty limited hours.


The Lab came about in January, by the way. And it officially launched, as a service to be used by the public, in June.

In October of last year, I organized a Personal Archiving Day, and held it at Library. A few months later, the Library’s public services manager approached me and asked if I could help turn the personal archiving concept into a more permanent fixture of the Library. We worked backwards, I think: we started with a small pot of money, and prioritized. Essentially: What were patrons already asking the Library for? And how could we meet those needs? The scanner was high priority, as was the ability to handle Hi8 tapes. Third was VHS.

I’m not full-time at the Library, so the project took shape in fragments: ten hours one week, twenty hours the next. The first few months were technical in nature — I spent hours testing equipment, reading (and then writing) manuals, and handling little things, like tracking down extra cords and cables. We were lucky, I think, because all of the equipment we bought worked, problem-free. That’s not always the case.

After the Lab was ready to go, I used it at Library-sponsored events. A few digitization days, which went over well. These events functioned, really, as a preview for the Lab: Oh, if you like this, come to the Central Library, where this equipment lives! I took the Lab to several branch libraries around the community, and, as I did this, we started advertising. Facebook and Nextdoor have been great, for the record. Free, easy advertising. And when I follow up with people and ask how they heard about the Lab, the answer is always ‘online.’


The day the Lab launched, not a single person showed up. I sat alone, for five hours, in a study room with a bunch of equipment, a charged-up MacBook, and waited. The next week, ten people came — and we had to turn several others away because we were operating at capacity. Since the second week, the Lab has been booked solid, and is almost always in use. And the interest just doesn’t seem to be dwindling, which is great.


I expect that interest in the Lab will ebb and flow.

I also expect that interest in the Lab depends on how well I do my job as an archivist: if I can continue to develop successful outreach strategies, then people will continue to seek out the Lab and its services. What I’ve found throughout this process is that Library patrons — and potential Library patrons — know that they should be doing something with the media they have at home, they just don’t know what exactly. (Or where. Or how.) And this stuff is so easy to forget about because, you know, where does it end up? The basement, the attic. Out of sight, out of mind.

Part of my job moving forward, then, will be to integrate the Lab in archival outreach efforts. To see it as a partnership, and not a stand alone service. To bring the Lab to people, instead of waiting for people to come to the Lab. Because once the shine wears off — and the shine wears off of all new things — it’s going to be about education.


Now go forth and digitize.

briefly, on love:

i have known, and been in love with, brian for about six years now. we got married, the two of us, under a big tree: the ceremony took place in missouri, and it was hot, even for september. brian and i love each other both in public and in private: i kiss him, certainly, before i leave the house and i tell him i love him before bed, when it is just the two of us, content and alone and drifting. but i have kissed him, too, at the grocery store, and i have kissed him on campus, and i have placed my hand on top of his at almost every restaurant in madison, and i do these things absentmindedly, perhaps as easily as i breathe in and then breathe out. and nobody objects. when brian and i got married beneath that tree, our loved ones looked on, and smiled. nobody protested. and when i grabbed brian’s hand on the sidewalk in madison yesterday, nobody stared. because people do not object — have never objected — to this kind of love.

there is courage in this kind of love, certainly. i practice no religion, but: to love brian is to have faith — that our commitment to each other lasts, that we remain healthy and happy, that the two of us continue to bring each other joy. after all, who knows?

i am awed, each day, by those who love in the face of objection and rejection and protest. i am awed by their courage and their determination. i am awed by their sacrifice. what happened in orlando was, without question, an attempt to silence them. to punish them. to terrorize them. to keep them hidden. and the vast majority of us, i imagine, cannot — could not — possibly imagine.

but if our society — if our communities and our laughter and our children and the things that bring us joy — is great, it is because these individuals continue on. have you noticed? can you imagine the strength it takes? to make art. to write. to live, and to bask, in the sun. to teach. to dance, to dance. they are alive and they cannot — and they will not — be made afraid.

to my friends who are reeling, and to my friends who are afraid, and to my friends who hurt: i see you, and i stand with you. i see you, and you matter. i see you, and i care. everything within me that is good and alive and creative, i owe, in part, to you. i do not learn — we do not learn — from those who attempt to terrorize and silence. i learn — we learn — from the courageous. from those who, day after day after day after day, wake up and still feel something — curiosity and determination and hope and everything in between — within themselves.

i see you, and i love you.

On hashtags

Up front:

A lot of this work was sparked by a discussion I had with Ed Summers earlier this week, and, you know, he’s also the one that built the tool I’ve been using to do this work, so. I deserve minimal credit for anything written here.

And, I’ll give additional credit to a conversation we had in class last week — about having questions to ask of data, versus having the skills to use tools to manipulate and look at data. It’s been fun for me, this semester, to tinker with things like twarc (and Programming Historian), but I’ve also realized that I’m not necessarily doing this work with a research question — or a thesis, or grant-funded project in mind. And I wasn’t quite sure, then, where I fit. But in class, we talked about how there is value — in its own right — in knowing how to use the tools. And so I’ve been tinkering.


Wisconsin is currently — and finally — grappling with the issue of racism on campus among its students and faculty, and, in response, student activists brought the conversation online, at #therealuw. (Disclaimer: it looks like this hashtag was used for the first time on March 17, but, due to Twitter’s API, I’ve only been able to work with tweets from the past seven days. Start as soon as you can.)

Alongside #therealuw are other similar hashtags: #reclaimOSU, concerning the creation of a “just, transparent and democratic food system,” the halting of “the Comprehensive Energy Management Plan which would further privatize Ohio State,” and demands that Ohio State administration divest from “companies that are complicit in Israeli apartheid.”

Another is #dismantledukeplantation, started by “a group of nine students” who are “currently occupying the main administrative building at Duke university to demand accountability and justice for Black & Brown workers.”

There are more, of course. And they’ll keep coming. But these are the three I decided to take a closer look at.


Twarc requires a bit of setup: you need to have Python installed, for one, and you need to register an application at Twitter (which will give you four codes you’ll need to run a search / crawl). Also, I encountered quite a few bugs while working with Twarc the first time, but these problems were on my end, and had to do with upgrading to El Capitan.

Anyway, this is all to say: I thought it would be interested to take a look at #therealuw, #dismantledukeplantation, and #reclaimosu as a group. So, first, I executed the crawls. (Another disclaimer: both #reclaimosu and #dismantledukeplantation are very active right now, while #therealuw was much more active earlier in the week, and that skewed the numbers a bit. Also, while crawling #reclaimosu, I hit Twitter’s rate limit, and stopped the crawl. So I wouldn’t call what I’ve collected a complete, or scholarly acceptable, set of data. It’s pretty haphazard, but it’s okay for a blog post.)

From #therealuw, I collected 1,269 tweets; from #dismantledukeplantation, 5,241; and from #reclaimosu, 5,290. One of the first things I did was create a word cloud for each.

Students, as exemplified, was the word that appeared the most across all three. (I ran ‘grep -c ‘students’ against each json files, and found that the word appears 371 times in #therealuw, 1612 times in #reclaimosu, and 1435 times in #dismantledukeplantation.) This isn’t exactly surprising, because these are student-led protests and demonstrations. But, the same concept can be used for less obvious words. Silenced, for instance, is not used in #therealuw or #dismantledukeplantation, but it appears 40 times in #reclaimosu. Community appears 71 times in #therealuw. Isolate appears 35 times in #dismantledukeplantation. Heard appears 85 times in #reclaimosu. And on. If you know what you’re looking for — a word, a hashtag-wide sentiment — this tool comes in pretty handy. (And I bet people a lot smarter than me can use this to consider trends and changes in the movements just by looking at a word or two.)


I attempted, too, to get an idea of how many original tweets were being generated within each hashtag. I think the original crawl I ran counted tweets en masse: so one tweet that was retweeted fifty times counted as fifty-one tweets. But by removing retweets, the number of different and unique tweets in each hashtag became a smaller group. #reclaimosu had 657 original tweets, #dismantledukeplantation had 897, and #therealuw had 325. 

You might — ? — be able to use this in an attempt to think about the number of ideas and thoughts being contributed to the hashtag. A collection of fifty tweets that, when stripped of all retweets, ends up being one tweet probably has less than a collection with six-thousand tweets that, when stripped of all retweets, ends up containing four thousand tweets. Right? It’s probably a faulty method, but I think it could at least help narrow the set of data you’re looking at, if you’re interested in something like the most active tweeters, or the first time an oft-retweeted sentiment appeared.


The last thing I did was look at links embedded within the tweets, which is something Ed Summers did for #therealuw about a week ago. By using a utility script installed alongside twarc, you can find out which link appeared most often in a certain hashtag. (I think what I’ve done here is slightly incorrect. The first step in this process is unshortening urls, which requires unshrtn, and I could not, for the life of me, figure out how to get that to work. Again: this is a user error.)

Interestingly, running this command turned up links that provide a whole lot of context to the origin and intention behind each hashtag. For instance, the most-tweeted link (90 times) within #therealuw hashtag was a link to a YouTube video created by Patrick Sims, Wisconsin’s Chief Diversity Officer and Vice Provost. (An aside: beware the link, friends. When I viewed it when it was posted, on March 31, the comments section was not open. But between now and then, it seems that the comments section was opened up, and what fills that space now are racist, vile, sickening comments. Hateful, inexcusable sentiments that are just gutting. And I don’t know what to say here, other than I can’t imagine the strength it takes to stand up to hatred like this, and that I support my fellow classmates in their brave, and seemingly unending, activism. We are better for their bravery and, in so many instances, undeserving of their efforts that push on in the face of this oppression.)

In #dismantledukeoppression, a Guardian article was cited most often, with 125 appearances. And for #reclaimosu, it was a link, tweeted out 117 times, from Afrikan Black Coalition, that listed the movement’s demands.


There are, I’m sure, hundreds of questions you can answer using this data. One question that comes to mind — if you act fast enough — might be the identification of the user who created the hashtag and / or the user that appears most often within the hashtag.

I look at this work from the perspective of an archivist most often. For instance: if we can identify a link that appears over and over within a hashtag, how do we make sure that the link lives on and remains accessible both tomorrow and years from now? And, you know, if we can identify trends as they appear on twitter, can we expect that similar content is being created outside of the Internet? And does a common theme within a hashtag lead us to look for these instances? Are there physical documents that reflect the conversation happening online? Videos? Photographs?

If so, how do we track them down?

As someone entering the field within the next few months, I recognize the need to be an archivist who engages with the source, the movement, the creator. When we wait for content, and people, to come to us, we barely get half the story. When we wait, we get content from those who already exist within the narrative and the archive: the privileged, the white, the rich, the well off. But by venturing out of our stale spaces, we meet new voices, and new people, and new histories, and we become part of the history as it is told by those who are living it — not by those who rewrote it, and not by those act as gatekeepers.

The role of the archivist is changing, I think. And hope. I don’t envision a future where archivists work like they did fifty — or ten — years ago. People create too much information, too quickly, for us to continue to operate like that, right? And so I hope that the profession turns outward: that our role becomes less custodial in nature and, instead, more educational. That archivists no longer protect what they think should live on, but instead teach communities to assure that the things they create outlive them — if they want them to. That our archives are no longer confined to a handful of institutions, but instead exist widely: at libraries, at community centers, within families. And I don’t think this threatens the field, or our institutions, but instead strengthens them.


To wrap, I’d be remiss if I didn’t bring your attention to Documenting the Now. Everything I’ve written about are ideas that the Documenting the Now team has been thinking about, and working toward, for a long time. And I am so excited to watch the work they do, and the conversation they’ve started, as it develops.

On poetry (always)

My sleep schedule is all out of sorts. I left a twenty-hour-a-week-job — where the workday started at eight in the morning — for three part-time jobs that don’t start until eleven or twelve. I had grand morning plans: I’d wake up early, I’d eat a real breakfast. Maybe I’d ride my bike, or read for pleasure. But, it’s 11:36 right now. Yesterday — today? — I went to bed at 2:30 in the morning. I woke up around 9. So much for those grand plans.

My nights have been productive, though. And perhaps this is how graduate students function? I’ve been working — hard, and a lot, this semester — on an independent study. It’s been a fascinating process. And not just the research. I’ve learned a lot about who I am as a student: disorganized, for one, and a little jumpy. I’ve learned a lot about the way I work: kind of in a panic, right before deadline. And I’ve learned a lot about how to be better at both of those things: start earlier, and keep your notes in one place. (It’s week seven. I have a rough draft due on Friday. And, for the first time this semester, it’s already in progress. And I still have two days.)

The whole thing is a work in progress, but I think the end product is going to be something really special. (And, if it isn’t, it won’t be because I didn’t try.)
All that said: Mary Oliver released a new book of poetry today. I’ve read it cover-to-cover twice. Here’s my favorite from the new collection, Felicity:

Nothing Is Too Small Not to Be Wondered About

The cricket doesn’t wonder
if there’s a heaven
or, if there is, if there’s room for him.

It’s fall. Romance is over. Still, he sings.
If he can, he enters a house
through the tiniest crack under the door.
Then the house grows colder.

He sings slower and slower.
Then, nothing.

This must mean something, I don’t know what.
But certainly it doesn’t mean
he hasn’t been an excellent cricket
all his life.