Mirrors of ~89 Tor-Bitcoin darknet markets & boards 2011-2015, mega market darknet and associated materials.

Internet archiving, R, DNM Archives, Hydra DNM

- Download- Research - Possible Uses- Works Using This Dataset- Citing- Donations

- Overall Coverage- Interpreting & Analyzing- Individual Archives - Aldridge & Décary-Hetu SR1- AlphaBay2017 (McKenna & Goode)- DNStats- Grams- Kilos- Information Leaks - Diabolus/Crypto Market- Simply Bear- TheRealDeal

- SR1F

- SR2Doug

- Crawler Wishlist

[Warning: JavaScript Disabled!]

[For support of key webpage features (hyperlink annotation popups/popovers & transclusions, collapsible sections, backlinks, tablesorting, picture zooming, sidenotes etc), you could enable JavaScript.]

Dark Net Markets (DNM) are on-line markets sometimes hosted as Tor hidden companies offering escrow companies between consumers & sellers transacting in Bitcoin or different cryptocoins, usually for drugs or different unlawful/regulated goods; probably the most famous DNM was Silk Road 1, which pioneered the business mannequin in 2011.

From between 2013-2015, I scraped/mirrored on a weekly or day by day foundation all present English-language DNMs as a part of my analysis into their usage, lifetimes/traits, & authorized riskiness; these scrapes coated vendor pages, suggestions, photos, and many others. As well as, I made or obtained copies of as many other datasets & paperwork associated to the DNMs as I might.

This uniquely complete assortment is now publicly launched as a 50GB (~1.6TB uncompressed) collection covering 89 DNMs & 37+ related boards, representing <4,438 mirrors, and is available for any research.

This page documents the download, contents, interpretation, and technical methods behind the scrapes.

Dark net markets have thrived since June 2011 when Adrian Chen published his well-known Gawker article proving that Silk Road 1 was, contrary to my assumption when it was announced in January/February 2011, not a scam and was a totally-functional drug market, a brand new type dubbed "dark net markets" (DNM). Fascinated, I soon signed up, made my first order, and started documenting how to make use of SR1 and then a number of months later, began documenting the primary recognized SR1-linked arrests. Monitoring DNMs was straightforward because SR1 was overwhelmingly dominant and BlackMarket Reloaded was a distant second-place market, with a number of irrelevancies like Deepbay or Sheep and then the flashy Atlantis.

This idyllic interval ended with the raid on SR1 in October 2013, which ushered in a new age of chaos by which centralized markets battled for dominance, the would-be successor Silk Road 2 was crippled by arrests and turned right into a ghost-ship carrying scammers, and the multisig breakthrough went begging. The tumult made it clear to me that no market or discussion board could possibly be counted on to last so long as SR1, and analysis into the DNM communities and markets, or even merely the reminiscence of their historical past, was threatened by bitrot: already in November 2013 I was seeing pervasive myths spread all through the media-that SR1 had $1 billion in sales, that you can buy baby pornography or hitmen providers on it, that there were multiple Dread Pirate Roberts-and different harmful beliefs in the community (that use of PGP was paranoia & pointless, markets may very well be trusted to not exit-rip-off, that FE was not a recipe for catastrophe, that SR2 was not infiltrated despite the workers arrests & even media protection of a SR1 mole, that guns & poison sellers were not extraordinarily risky to buy from, that buyers had been by no means arrested).

And so, starting with the SR1 boards, which had not been taken down by the raid (to help the mole? I wondered at the time), I began scraping all the new markets, doing so weekly and typically day by day starting in December 2013. These are the outcomes.

Download

The complete archive is obtainable for obtain from the Internet Archive as a .torrent1 (merchandise web page; full file listing).

A public rsync mirror is also obtainable:

For a single file (eg. the 2 Grams exports), one can obtain like thus:

(If the obtain doesn't start, it could also be a Torrent client drawback associated to Getright-webseeding-assist; if the torrent doesn't work, all recordsdata could be downloaded normally over HTTP from the IA item page, but if possible, torrents are beneficial for reducing the bandwidth burden & error-checking.)

Research

Possible Uses

Listed below are some suggested makes use of:

- providing info on distributors across markets like their PGP key and feedback rankings

- identifying arrested and flipped sellers (eg. the Weaponsguy sting on Agora)

- particular person drug and class reputation

- whole sales per day, with consequent turnover and commission estimates; correlates with Bitcoin or DNM-associated search visitors, subreddit site visitors, Bitcoin worth or quantity, and so forth

- vendor lifetimes, rankings, over time and by product offered

- losses to DNM exit scams, or vendor exit scams

- reactions to exogenous shocks like Operation Onymous

- survival evaluation, and predictors of exit-scams (early finalization quantity; site downtime; new distributors; etc)

- subject modeling of forums

- compilations of discussion board posts on lab assessments estimating purity and safety

- compilations of discussion board-posted Bitcoin addresses to study the effectiveness of market tumblers

- stylometric evaluation of posters, particular site staff (what's employees turnover like? do any markets ever change fingers?)

- deanonymization and knowledge leaks (eg. GPS coordinates in metadata, usernames reused on the clearnet, valid emails in PGP public keys)

- security practices: use of PGP, lifetime of individual keys, unintentional posts of non-public fairly than public keys, malformed or unusable public keys, and so forth

- anthologies of actual-world images of explicit medicine compiled from all sellers of them

- merely searching previous listings, remembering the good times and dangerous times, the fallen and the free

Works Using This Dataset

See the DNM-Archives-user tag.

Citing

Please cite this useful resource as:

- Gwern Branwen, Nicolas Christin, David Décary-Hétu, Rasmus Munksgaard Andersen, StExo, El Presidente, Anonymous, Daryl Lau, Sohhlz, Delyan Kratunov, Vince Cakic, Van Buskirk, Whom, Michael McKenna, Sigi Goode. "Dark Net Market archives, 2011-2015", 2015-07-12. Web. [access date] /DNM-archives

@miscdnmArchives, writer = Gwern Branwen and Nicolas Christin and David Décary-Hétu and Rasmus Munksgaard Andersen and StExo and El Presidente and Anonymous and Daryl Lau and Sohhlz, Delyan Kratunov and Vince Cakic and Van Buskirk and Whom and Michael McKenna and Sigi Goode, title = Dark Net Market archives, 2011-2015, howpublished= \urlhttps://gwern.net/dnm-archive, url = https://gwern.web/dnm-archive, sort = dataset, year = 2015, month = July, timestamp = 2015-07-12, be aware = Accessed: DATE

Donations

A dataset like this owes its existence to many parties:

- the DNMs couldn't exist with out volunteers and nonprofits spending the money to pay for the bandwidth utilized by the Tor community; these scrapes collectively represent terabytes of consumed bandwidth. Should you want to donate in direction of keeping Tor servers running, you may donate to Torservers.web or the Tor Project itself

- the Internet Archive hosts numerous superb resources, of which that is but one, and is an distinctive Internet resource; they accept many forms of donations

- collating and creating these scrapes has absorbed an unlimited quantity of my time & energy as a result of the necessity to resolve CAPTCHAs, launch crawls on a each day or weekly basis, debug subtle glitches, work round site defenses, periodically archive scrapes to make disk area obtainable, provide internet hosting for some scrapes launched publicly and so forth (my arbtt time-logs recommend >200 hours since 2013); I thank my supporters for their patience during this long undertaking.

There are ~89 markets, >37 forums and ~5 different websites, representing 43,596,420 files in ~49.4GB of 163 compressed recordsdata, unpacking to >1548GB; the largest single archive decompresses to <250GB. (It can be burned to 3 25GB BDs or 2 50GB BDs; if the former, it may be worth generating additional FEC.)

These archives are xz-compressed tarballs (optimized with the kind-key trick); typically each subfolder is a single date-stamped (YYYY-MM-DD) crawl using wget, with the default listing/file format. The vast majority of the content is HTML, CSS, and pictures (usually photographs of merchandise listings); photographs are space-intensive & omitted from many crawls, however I feel that photos are helpful to allow shopping the markets as they have been and could also be highly helpful in their own proper as analysis material, so I tried to collect photos the place applicable. (Child porn is just not a concern as all DNMs & DNM boards ban that content material.) Archives sourced from other individuals observe their own explicit conventions. Mac & Windows customers could possibly uncompress using their constructed-in OS archiver, 7zip, Stuffit, or WinRAR; the PAR2 error-checking will be accomplished utilizing par2, QuickPar, Par Buddy, MultiPar or others relying on one’s OS.

For those who don’t wish to uncompress all of a selected archive, as they are often massive, you possibly can try extracting particular information using archiver-specific options; for example, a SR2F command concentrating on a selected previous forum thread:

Kaggle variations:

- "Dark Net Marketplace Data (Agora2014-2015): Includes over 100,000 unique listings of drugs, weapons and more"

- "Drug Listing Dataset: Drug listing dataset from a number of darknet marketplaces", Mun Hou Won (CSV of 1776 / Abraxas / Agora / Evolution / Nucleus / Outlaw Market / Silk Road 2 / The Marketplace)

- Other: "Exploration and Analysis of Darknet Markets", Daniel Petterson

Overall Coverage

Most of the material dates 2013-2015; some archives sourced from other individuals (earlier than I began crawling) may date 2011-2012.

Specifically:

- Markets:

- Agape

- Amazon Dark

- Anarchia

- Area51

- Armory2

- Atlantis

- Black Goblin

- BlackMarket Reloaded

- Black Services Market

- Bloomsfield

- Blue Sky Market

- Breaking Bad

- BuyItNow

- Cannabis Road 1

- Cantina

- Crypto Market / Diabolus

- Darklist

- DBay

- Deepzon

- Drugslist

- FreeBay

- Freedom Marketplace

- Free Market

- Horizon

- Ironclad

- Onionshop

- Pirate Market

- Poseidon

- Silk Road Reloaded (I2P)

- Silkstreet

- Simply Bear

- The BlackBox Market

- The Marketplace

- Topix 2

- TorBay

- Tortuga 2

- Vault43

- White Rabbit

- Zanzibar Spice

Forums:

- Black Market Reloaded

- BlackBank Market

- Cannabis Road 2

- Cannabis Road 3

- DarkBay

- Darknet heroes

- Doge Road

- Gobotal

- GreyRoad

- Havana/Absolem

- Kingdom

- Kiss

- Mr Nice Guy 1

- Outlaw Market

- Panacea

- Pigeon

- Project Black Flag

- Revolver

- Silk Road 1

- The Cave

- The Hub

- The Majestic Garden

- The RealDeal

- TorEscrow

- TorBazaar

- Tortuga 1

- Underground Market

- Unitech

- Utopia

Miscellaneous:

- Assassination Market

- Cryuserv

- DNM-associated documents3

- DNStats

- Grams

- Pedofunding

- SR2doug’s leaks

Missing or incomplete

- BMR

- SR1

- TorMarket

- Deepbay

- Red Sun Marketplace

- Sanitarium Market

- EXXTACY

- Mr Nice Guy 2

Interpreting & Analyzing

Scrapes will be troublesome to research. They're large, complicated, redundant, and highly error-prone. They cannot be taken at face-worth.

Irrespective of how a lot work one places into it, one will never get an actual snapshot of a market at a particular immediate: listings will go up or down as one crawls, distributors can be banned and their whole profile & listings & all feedback vanish immediately, Tor connection errors will cause a nontrivial % of page requests to fail, the site itself will go down (Agora particularly), and Internet connections are imperfect. Scrapes can get bogged down in a backwater of irrelevant pages, spend all their time downloading a morass of on-demand generated pages, the person login expire or be banned by site directors, etc. If a page is current in a scrape, then it in all probability existed at some point; but if a web page just isn't present, then it might not have existed or existed however did not get downloaded for any of a myriad of reasons. At best, a scrape is a lower sure on how much was there.

So any evaluation must take critically the incompleteness of each crawl and the very fact that there is lots and at all times might be a lot of lacking data, and do issues like focus on what can be inferred from ‘random’ sampling or explicitly mannequin incompleteness through the use of markets’ category-rely-listings. (For instance, if your download of a market claims to have 1.3k objects however the categories’ claimed listings sum to 13k objects, your download might be highly incomplete & biased towards certain classes as effectively.) There are lots of delicate biases: for instance, there will be upward biases in markets’ average overview rankings as a result of sellers who transform scammers will disappear from the market scrapes when they are banned, and few of their customers will return and revise their rankings; similarly if scammers are concentrated in particular categories, then using a single snapshot will result in biased outcomes as the scammers have already been eliminated, whereas uncontroversial sellers last a lot longer (which could result in, say, e-ebook sellers seeming to have many more gross sales than anticipated).

The contents can't be taken at face-value both. Some distributors interact in evaluate-stuffing using shills. Metadata like classes can be wrong, manipulated, or deceptive (a category labeled "Musical instruments" might include listings for prescription medicine-beta blockers-or modafinil or Adderall may be listed in each a "Prescription drugs" and "Stimulants" category). Many things stated on forums are lies or bluffing or scams. Market operators could intentionally deceive customers (Ross Ulbricht claiming to have bought SR1, the SR2 group participating in "psyops") or conceal info (the hacks of SR1; the second SR2 hack) or assault their customers (Sheep Marketplace and Pandora). Different markets have completely different traits: the commission charge on Pandora was unilaterally raised after it was hacked (inflicting sales volume to fall); SR2 was a infamous scammer haven attributable to inactive or overwhelmed staff and missing a working escrow mechanism; and so forth. There is no substitute right here for domain information.

Knowing this, analyses ought to have some strategy to deal with missingness. There are a pair tacks:

- attempt to use "ground truths" to explicitly mannequin and cope with varying degrees of missingness; there are a variety of floor-truths accessible in the type of leaked vendor knowledge (screenshots & data), databases (leaked, hacked), official statements (eg. the FBI’s quoted numbers about Silk Road 1’s total gross sales, variety of accounts, variety of transactions, etc)

For one validation of this set of archives, see Bradley2019’s "On the Resilience of the Dark Net Market Ecosystem to Law Enforcement Intervention", which is ready to compare the SR2 scrapes to information extracted from SR2 by UK law enforcement post-seizure, and finds that any scrape is incomplete (as anticipated) but that scrapes usually appear to be incomplete in related ways and usable for evaluation. For another try at validating, see Soska & Christin2015’s "Measuring the Longitudinal Evolution of the online Anonymous Marketplace Ecosystem", which compares crawl-derived estimates to SR1 sales data produced at Ross Ulbricht’s trial (CSV/discussion), gross sales figures in the Blake Benthall SR2 criminal complaint, and a Agora seller’s leaked vendor profile; in all cases, the estimates are moderately close to the bottom-truth.

- assume lacking-at-random and use analyses insensitive to that, focusing on things like ratios

- work with the information as is, writing outcomes such that the biases and decrease-bounds are specific & emphasised

Individual Archives

Among the archives are unusual and should be described in additional detail.

Aldridge & Décary-Hetu SR1

The September SR1 crawl is processed information stored in SPSS .sav Data Files. There are various libraries accessible for studying this format (in R, using the foreign library like library(international); sellers <- read.spss("Sellers---2013-09-15.sav", to.data.frame=TRUE).)

AlphaBay2017 (McKenna & Goode)

A crawl of AlphaBay2017-01-26-2017-01-28 and data extraction (using a Python script) supplied by Michael McKenna & Sigi Goode. Additionally they tried to crawl AB’s historic inactive listings along with the usual reside/energetic listings, reaching many of them.

Because of IA add problems, at present hosted individually.

DNStats

DNStats is a service which periodically pings hidden providers and data the response & latency, generating graphs of uptime and permitting customers to see how lengthy a market has been down and if an error is more likely to be transient. The proprietor has supplied me with three SQL exports of the ping database as much as 2017-03-25; this database may very well be helpful for evaluating downtime across markets, inspecting the impact of DoS attacks, or regressing downtime against things like the Bitcoin alternate fee (presumably if the markets nonetheless drive greater than a trivial amount of the Bitcoin economic system, downtime of the most important markets or market deaths ought to predict falls in the alternate price).

For example, to graph a median of site uptime per day and highlight as an exogenous occasion Operation Onymous, the R code would go like this:

The service is an useful one and accepts donations: 1DNstATs59JANuXjbpS5ngWHqvApAhYHBS.

Grams

Grams (http://grams7enufi7jmdl.onion/) (subreddit) was a service primarily specializing in looking out market listings; they can pull listings from API exports supplied by markets (Evolution, Cloud9, Middle Earth, Bungee54, Outlaw), or they may use their own custom crawls (the remaining). They have generously given me near-daily CSV exports of the current state of listings in their search engine, ranging 2014-06-09-2015-07-12 for the primary archive and 2015-07-14-2016-04-17 for the second. Grams coverage:

1. first:

- 1776

- ADM

- BlackBank

- Bungee54

- Cloud9

- NK

- Silk Road 2

- TPM

2. second archive:

- Dream Market

- Hansa

- Oasis

- RealDeal

- Silkkitie

- Tochka

- Valhalla

The Grams archive has three virtues:

1. whereas it doesn’t have any raw information, the CSVs are easy to work with. For instance, to learn in all of the Grams SR2 crawls, then depend & graph complete listings by day in R:

DIR <- "blackmarket-mirrors/archive/grams" # Grams's SR2 crawls are named like "grams/2014-06-13/SilkRoad.csv" gramsFiles <- list.files(path=DIR, pattern="SilkRoad.csv", all.files=TRUE, full.names=TRUE, recursive=TRUE) # schema of SR2 crawls eg: ## "hash","market_name","item_link","vendor_name","price","name","description","image_link","add_time", \ ## "ship_from", ## "2-11922","Silk Road 2","http://silkroad6ownowfk.onion/items/220-fe-only-tw-x-mb","$220for28grams", \ ## "0.34349900", "220 FE Only TW X MB","1oz of the same tw x mb as my other listing FE only. Not shipped \ ## until finalized. Price is higher for non FE listing.","","1404258628","United States",... # most fields are self-explanatory; 'add_time' is presumably a Unix timestamp # read in each CSV, note what day it is from, and combine into a single data-frame: grams <- data.frame() for (i in 1:length(gramsFiles)) log <- read.csv(gramsFiles[i], header=TRUE) log$Date <- as.Date(gsub("/SilkRoad.csv", "", gsub(paste0(DIR,"/"), "", gramsFiles[i]))) grams <- rbind(grams,log) totalCounts <- aggregate(hash ~ Date, length, data=grams) summary(totalCounts) # Date hash # Min. :2014-06-09 Min. : 2846.00 # 1st Qu.:2014-07-05 1st Qu.: 9584.25 # Median :2014-08-26 Median :10527.50 # Mean :2014-08-21 Mean : 9651.44 # 3rd Qu.:2014-09-29 3rd Qu.:11165.00 # Max. :2014-11-07 Max. :19686.00 library(ggplot2) qplot(Date, hash, data=totalCounts) # https://i.imgur.com/ucPMvJQ.png

Other included datasets which are in structured formats that may be easier to deal with for prototyping: the Aldridge & Décary-Hétu 2013 SR1 crawl; the SR1 sales spreadsheet (original is a PDF but I’ve created an usable CSV of it); the BMR feedback dumps are in SQL, as is DNStats and Christin et al 2013’s public data (but note the last is so heavily redacted & anonymized as to support few analyses); and Daryl Lau’s SR2 work may be in a structured format.

2. the crawls were conducted independent of other crawls and they can be used to check each other

3. the market data sourced from the APIs can be considered close to 100% complete & accurate, which is rare

The main drawbacks are:

- the largest markets can be split across multiple CSVs (eg. EVO.csv & EVO2.csv), complicating reading the data in somewhat

- the export each time is of the current listings, which means that different days can repeat the same identical crawl data if there was not a successful crawl by Grams in between

- exports are not available for every day, and some gaps are large. The 2015-01-09 to 2015-02-21 gap is due to a broken Grams export during this period before I noticed the problem and requested it be fixed; other gaps may be due to transient errors with the cron job:

@daily ping -q -c 5 google.com && torify wget --quiet --continue "http://grams7enufi7jmdl.onion/gwernapi/$SECRETKEY" -O ~/blackmarket-mirrors/grams/`date '+\%Y-\%m-\%d'`.zip

so if my Internet was down, or Grams was down, or the download was corrupted halfway through, then there would be nothing that day.

Kilos

The owner of Kilos, a DNM search engine much like Grams, released a CSV on 2020-01-13 of 235,668 review scraped from 6 DNMs (Apollon, CannaHome, Cannazon, Cryptonia, Empire, & Samsara):

The data is in the format

Site, vendor, and comment are strings. Site and vendor are both alphanumeric, while comment may have punctuation and whatnot. Line breaks are explicit "
" in the comment field, and the comment field has quotation marks around it to make it easier to sort through. All the data uses Latin characters only, no unicode. timestamp is an integer indicating the number of seconds since the Unix epoch. Score is 1 for positive review, 0 for neutral review, and −1 for negative review. value_btc is the bitcoin value of the product being reviewed, calculated at the time of the review.

There are some slight problems with the data set as a result of the pain that is scraping these marketplaces. All reviews from Cryptonia market have their timestamp as 0 because I forgot to decode the dates listed and just used 0 as a placeholder. Cryptonia reviews’ score variable is unreliable, as I accidentally rewrote all scores to 0 on the production database. To correct for this, I rewrote the scores to match a sentiment analysis of the review text, but this is not a perfect solution, as some reviews are classified incorrectly. eg. "this shit is the bomb!" might be classified negatively despite context telling us that this is a positive review.

There are a decent number of duplicates, some of which are proper (eg. "Thanks" as a review appears many many times) and some of which are improper (detailed reviews being indexed multiple times by mistake).

Information Leaks

Diabolus/Crypto Market

Diabolus/Crypto Market are two markets run by the same team off, apparently, the same server. Crypto Market had an information leak where any attempt to log in as an existing user revealed the status bar of that Diabolus account, listing their current number of orders, number of PMs, and Bitcoin balance, and hence giving access to ground-truth estimates of market turnover and revenue. Using my Diabolus crawls to source a list of vendors, I set up a script to automatically download the leaks daily until the hole was finally closed.

Simply Bear

Upon launch, the market Simply Bear made the amateur mistake of failing to disable the default Apache /server-status page, which shows information about the server such as what HTML pages are being browsed and the connecting IPs. Being a Tor hidden service, most IPs were localhost connections from the daemon, but I noticed the administrator was logging in from a local IP (the 192.168.1.x range) and curious whether I could de-anonymize him, I set up a script to poll /server-status every minute or so, increasing the interval as time passed. After two or three days, no naked IPs had appeared yet and I killed the script.

TheRealDeal

TheRealDeal was reported on Reddit in late June 2015 to have an info leak where any logged-in user could browse around a sixth of the order-details pages (which were in a predictable incrementing whole-number format) of all users without any additional authentication, yielding the Bitcoin amount, listing, and all Bitcoin multisig addresses for that order. TRD denied that this was any kind of problem, so I collected order information for about a week.

Modafinil

As part of my interest in the stimulant modafinil, I have been monthly collecting by hand scrapes of all modafinil/armodafinil/adrafinil listings across the DNMs; the modafinil archive contains the saved files in MHT or MAFF format from 2013-05-28 to 2015-07-03. Sampled markets include:

- Abraxas

- Agora

- Alpaca

- AlphaBay

- Andromeda

- Black Bank

- Blue Sky

- Cloud-Nine

- Crypto/Diabolus

- Diabolus

- Dream

- East India Company

- Evolution

- Haven

- Hydra

- Middle Earth

- Nucleus

- Outlaw

- Oxygen

- Pandora

- Sheep

- SR2

- TOM

Pedofunding

A crowdfunding site for child pornography, "Pedofunding", was launched in November 2014. It seemed like possibly the birth of a new DNM business model so I set up a logged-out scrape to archive its beginnings (sans any images), collecting 20 scrapes from 2014-11-13 to 2014-12-02, after which it shut down, apparently having found no traction. (A followup in 2015 tried to use some sort of Dash/Darkcoin mining model; it’s unclear why they don’t simply use Darkleaks, or how far it got before it too vanished.)

Silk Road 1 (SR1)

- "Down the silk rabbit hole" (source), Delyan Kratunov

- appendix to Van Buskirk et al

- "Traveling the Silk Road: Datasets" (recompressed), supporting information for Christin et al 2013, "Traveling the Silk Road: A measurement analysis of a large anonymous online marketplace"

- 2013 scrape provided me by anonymous

- source data for "Not an ‘Ebay for Drugs’: The Cryptomarket ‘Silk Road’ as a Paradigm Shifting Criminal Innovation", Aldridge & Décary-Hétu2014

SR1F

Files: silkroad1-forums-20130703-anonymous.tar.xz, silkroad1-forums-20131103-gwernrasmusandersen.tar.xz, silkroad1-forums-anonymous.tar.xz, silkroad1-forums-stexo.tar.xz, silkroad1-forums.tar.xz.

These archives of the Silk Road 1 forums is composed of 3 parts, all created during October 2013 after Silk Road 1 was shut down but before the Silk Road 1 forums went offline some months later:

1. StExo’s archive, released anonymously

This excludes the Vendor Roundtable (VRT) subforum, and is believed to have been censored in various respects such as removing many of StExo’s own posts.

2. Moustache’s archived pages

Unknown source, may be based on StExo archives

3. My & qwertyoruiop’s consolidated scrape

After the SR1 bust and StExo’s archiving, I began mirroring the SR1F with wget, logged in as a vendor with access to the Vendor Roundtable; unfortunately due to my inexperience with the forum software Simple Machines, I did not know it was possible to revoke your own access to subforums with wget (due to SMF violating HTTP standards which require GET requests to be side-effect-free of things like ‘delete special permissions’) and failed to blacklist the revocation URL. Hence the VRT was incompletely archived. I combined my various archives into a single version.

Simultaneously, qwertyoruiop was archiving the SR1F with a regular user account and a custom Node.js script. I combined his spider with my version to produce a final version with reasonable coverage of the forums (perhaps 3⁄4s of what was left after everyone began deleting & censoring their past posts).

David Décary-Hetu has contributed a processed SQL database of SR1F posts (mirrors: Dropbox; Mega).4

SR2

Sources:

- in January 2014, Sohhlz made & distributed a scrape of SR2 vendor pages akin to StExo’s SR1 vendor dump

- "Analyzing Trends in the Silk Road 2.0" (source), Daryl Lau

SR2Doug

In 2015, a pseudonym claiming to be a SR2 programmer offered for sale, using the Darkleaks protocol, what he claimed was the username/password dump and SR2 source code. The Darkleaks protocol requires providing encrypted data and then the revelation of a random fraction of it. This archive is all the encrypted data, decryption keys, and revealed usernames I was able to collate. (The auction did not seem to go well as the revealed data was not a compelling proof, and it’s unclear whether he was the genuine article.)

Copyright

The copyright status of crawls of websites, particularly ones engaged in illegal activities, is unclear.

- to the extent I hold any copyright in the contents, I release my work under the Creative Commons CC0 "No Rights Reserved" license

- the SR1 Christin et al 2013 dataset is licensed under the CC BY-NC

- other authors may reserve other rights

(I request that users respect the spirit of this archive and release their own source code & derived datasets to public, but I will not legally demand it.)

Previous Releases

Some of these archives have been released publicly before and are now obsoleted by this torrent:

- SR2 market

- Evolution forums

- Evolution market

Verification

PAR2 archives are provided for error-correction, and PGP signatures for strong integrity checking, should that be an issue.

Integrity of the archive can be verified using PAR2: par2verify ecc.par2. Up to 10% of file damage/loss can be repaired using the supplied PAR2 files for FEC and par2repair; see the man page for details.

Signed SHA-256 hashes of the archives:

How To Crawl Markets

The bulk of the crawls are my own work, and were generally all created in a similar way.

My setup was a Debian testing Linux system with Tor, Privoxy, and Polipo installed. For browsing, I used Iceweasel; useful FF extensions included LastPass, Flashblock & NoScript, Live HTTP Headers, Mozilla Archive Format, User Agent Switcher & switchproxytype, and RECAP. See the Tor guides.

1. when a new market opens, I learn of it typically from Reddit or The Hub, and browse to it in Firefox configured to proxy through 127.0.0.1:8123 (Polipo)

2. create a new account

The username/password are not particularly important but using a password manager to create & store strong passwords for throwaway accounts has the advantage of making it easier to authenticate any hacks or database dumps later. (Given the poor security record of many markets, it should go without saying that you should not use your own username or any password which is used anywhere else.)

3. I locate various ‘action’ URLs: login, logout, ‘report vendor’, ‘settings’, ‘place order’, ‘send message’, and add the URL prefixes (sometimes they need to be regexps) into /etc/privoxy/user.action; Privoxy, a filtering proxy running on 127.0.0.1:8118, will then block any attempt to download URLs which match those prefixes/regexps

A good blacklist is critical to avoid logging oneself out and immediately ending the crawl, but it’s also important to avoid triggering any on-site actions which might cause your account to be banned or prompt the operators to put in anti-crawl measures you may have a hard time working around. A blacklist is also invaluable for avoiding downloading superfluous pages like the same category page sorted 15 different ways; Tor is high latency and you cannot afford to waste a request on redundant or meaningless pages, which there can be many of. Simple Machine Forums are particularly dangerous in this regard, requiring at least 39 URLs blacklisted to get an efficient crawl, and implementing many actions as simply HTTP links that a crawler will browse (for example, if you have managed to get access to a private subforum on a SMF, you will delete your access to it if you simply turn a crawler like wget or HTTrack loose, which I learned the hard way).

4. where possible, configure the site to simplify crawling: request as many listings as possible on each page, hide clutter, disable any options which might get in the way, etc.

Forums often default to showing 20 posts on a page, but options might let you show 100; if you set it to display as much as possible (maximum number of posts per page, subforums listed, etc), the crawls will be faster, save disk space, and be more reliable because the crawl is less likely to suffer from downtime. So it is a good idea to go into the SMF forum settings and customize it for your account.

5. in Firefox, I export a cookies.txt using the FF extension Export Cookies. (I also recommend NoScript to avoid JavaScript shenanigans, Live HTTP Headers to assist in debugging by showing the HTTP headers and requests FF is actually sending to the market, and User Agent Switcher to lock your FF into showing a consistent TorBrowser user-agent)

6. with a valid cookie in the cookies.txt and a proper blacklist set up, mirrors can now be made with wget, using commands like thus:

alias today="date '+%F'" # prints out current date like "2015-07-05" cat ~/blackmarket-mirrors/user-agent.txt ## Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/30.0 cd ~/blackmarket-mirrors/cryptomarket/ grep -F --no-filename '.onion' ~/cookies.txt ~/`today`/cookies.txt > ./cookies.txt http_proxy="localhost:8118" wget --mirror --tries=5 --retry-connrefused --waitretry=1 --read-timeout=20 --timeout=15 --tries=10 --load-cookies=cookies.txt --keep-session-cookies --max-redirect=1 --referer="http://cryptomktgxdn2zd.onion" --user-agent="$(cat ~/blackmarket-mirrors/user-agent.txt)" --append-output=log.txt --server-response 'http://cryptomktgxdn2zd.onion/category.php?id=Weed' mv ./cryptomktgxdn2zd.onion/ `today` mv log.txt ./`today`/ rm cookies.txt

To unpack the commands:

- the grep -F invocation minimizes the size of the local cookies.txt and helps prevent accidental release of a full cookies.txt while packing up archives and sharing them with other people

- wget:

- we direct it to download only through Privoxy in order to benefit from the blacklist. Warning: wget has a blacklist option but it does not work, because it is implemented in a bizarre fashion where it downloads the blacklisted URL (!) and then deletes it; this is a known >12-year-old bug in wget. For other crawlers, this behavior should be double-checked so you don’t wind up inadvertently logging yourself out of a market and downloading gigabytes of worthless front pages.

- we throw in a number of options to encourage wget to ignore connection failures and retry; hidden servers are slow and unreliable

- we load the cookies file with the authentication for the market, and in particular, we need --keep-session-cookies to keep around all cookies a market might give us, particularly the ones which change on each page load.

- --max-redirect=1 helps deal with a nasty market behavior where when one’s cookie has expired, they then quietly redirect, without errors or warnings, all subsequent page requests to a login page. Of course, the login page should also be in the blacklist as well, but this is extra insurance and can save one round-trip’s worth of time, which will add up. (This isn’t always a cure, since a market may serve a requested page without any redirects or error codes but the content will be a transcluded login page; this apparently happened with some of my crawls such as Black Bank Market. There’s not much that can be done about this except some sort of post-download regexp check or a similar post-processing step.)

- some markets seem to snoop on the "referer" part of a HTTP request specifying where you come from; putting in the market page seems to help

- the user-agent, as mentioned, should exactly match however one logged in, as some markets record that and block accesses if the user-agent does not match exactly. Putting the current user-agent into a centralized text file helps avoid scripts getting out of date and specifying an old user-agent

7. logging of requests and particularly errors is important; --server-response prints out headers, and --append-output stores them to a log file. Most crawlers do not keep an error log around, but this is necessary to allow investigation of incompleteness and observe where errors in a crawl started (perhaps you missed blacklisting a page); for example, "Evaluating drug trafficking on the Tor Network: Silk Road 2, the sequel", Dolliver2015, failed to log errors in their few HTTrack crawls of SR2, and so wound up with a grossly incomplete crawl which led to nonsense conclusions like 1-2% of SR2’s sales were drugs. (I speculate the HTTrack crawl was stuck in the ebooks section, which was always clogged with spam, and then SR2 went down for an hour or two, leading to HTTrack’s default behavior of quickly erroring out and finishing the crawl; but the lack of logging means we may never know what went wrong.)

8. once the wget crawl is done, then we name it whatever day it terminated on, we store the log inside the mirror, and clean up the probably-now-expired cookies, and perhaps check for any unusual problems.

This method will permit somewhere around 18 simultaneous crawls of different DNMs or forums before you begin to risk Privoxy throwing errors about "too many connections". A Privoxy bug may also lead to huge logs being stored on each request. Between these two issues, I’ve found it helpful to have a daily cron job reading rm -rf /var/log/privoxy/*; /etc/init.d/privoxy restart so as to keep the logfile mess under control and occasionally start a fresh Privoxy.

Crawls can be quickly checked by comparing the downloaded sizes to past downloads; markets typically do not grow or shrink more than 10% in a week, and forums’ downloaded size should monotonically increase. (Incidentally, that implies that it’s more important to archive markets than forums.) If the crawls are no longer working, one can check for problems:

- is your user-agent no longer in sync?

- does the crawl error out at a specific page?

- do the headers shown by wget match the headers you see in a regular browser using Live HTTP Headers?

- has the target URL been renamed?

- do the URLs in the blacklist match the URLs of the site, or did you log in at the right URL? (for example, a blacklist of "www.abraxas…onion" is different from "abraxas…onion"; and if you logged in at an onion with www. prefix, the cookie may be invalid on the prefix-free onion)

- did the server simply go down for a few hours while crawling? Then you can simply restart and merge the crawls.

- has your account been banned? If the signup process is particularly easy, it may be simplest to just register a fresh account each time.

Despite all this, not all markets can be crawled or present other difficulties:

- Blue Sky Market did something with HTTP headers which defeated all my attempts to crawl it; it rejected all my wget attempts at the first request, before anything even downloaded, but I was never able to figure out exactly how the wget HTTP headers differed in any respect from the (working) Firefox requests

- Mr Nice Guy 2 breaks the HTTP standard by returning all pages gzip-encoded, whether or not the client says it can accept gzip-encoded HTML; as it happens, wget cannot read gzip-encoded HTML and parse the page for additional URLs to download, and so mirroring breaks

- AlphaBay, during the DoS attacks of mid-2015, began doing something odd with its HTTP responses, which makes Polipo error out; one must browse AlphaBay after switching to Privoxy; Poseidon also did something similar for a time

- Middle Earth rate-limits crawls per session, limiting how much can be downloaded without investing a lot of time or in a CAPTCHA-breaking service

- Abraxas leads to peculiarly high RAM usage by wget, which can lead to the OOM killer ending the crawl prematurely

See also the comments on crawling in "Measuring the Longitudinal Evolution of the Online Anonymous Marketplace Ecosystem", Soska & Christin2015, and Turk et al 2020.

Crawler Wishlist

In retrospect, had I known I was going to be scraping so many sites for 3 years, I probably would have worked on writing a custom crawler. A custom crawler could have simplified the blacklist part and allowed some other desirable features (in descending order of importance):

- CAPTCHA library: if CAPTCHAs could be solved automatically, then each crawl could be scheduled and run on its own.

The downside is that one would need to occasionally manually check in to make sure that none of the possible problems mentioned previously have happened, since one wouldn’t be getting the immediate of noticing a manual crawl finishing suspiciously quickly (eg. a big site like SR2 or Evolution or Agora should take a single-threaded normal crawl at least a day and easily several days if images are downloaded as well; if a crawl finishes in a few hours, something went wrong).

- supporting parallel crawls using multiple accounts on a site

- optimized tree traversal: ideally one would download all category pages on a market first, to maximize information gain from initial crawls & allow estimates of completeness, and then either randomly sample items or prioritize items which are new/changed compared to previous crawls; this would be better than generic crawlers’ defaults of depth or breadth-first

- removing initial hops in connecting to the hidden service, speeding it up and reducing latency (does not seem to be a config option in Tor daemon but I’m told something like this is done in Tor2web)

- post-download checks: a market may not visibly error out but start returning login pages or warnings. If these could be detected, the custom crawler could log back in (particularly with CAPTCHA-solving) or at least alert the user to the problem so they can decide whether to log back in, create a new account, slow down crawling, split over multiple accounts, etc

Other Datasets

One publicly available full dataset is:

- Sarah Jamie Lewis2016, "Dark Web Data Dumps" (Valhalla Marketplace scrapes, as of 2016-12-11)

A number of other datasets are known to exist but are unavailable or available only in restricted form, including:

- law enforcement scrapes (see the Force briefing), seized server images

- Interpol (eg. "Pharmaceutical Crime on the Darknet: A study of illicit online marketplaces", February 2015; based on monthly scrapes by INTERPOL IGCI June 2014-December 2014; this is probably drawing on their ongoing comprehensive scraping activities)

- Europol (eg. European Monitoring Centre for Drugs and Drug Addiction’s (EMCDDA) 2017 report "Drugs and the darknet: Perspectives for enforcement, research and policy" which also draws on Christin)

National Drug and Alcohol Research Centre (NDARC) in Sydney, Australia; Australian vendor focused crawls, non-release may be due to concerns over Australian police interest in them as documentation of sales volume to use against the many arrested Australian sellers

unknown Princeton grad student

Christin CMU group: uncensored SR1 crawls (available on request via IMPACT), large number of other markets crawled 2012-2015 (see Soska & Christin2015; Europol; Bone & Cleveland2018; possibly Hutchings2018; "Plug and Prey? Measuring the Commoditization of Cybercrime via Online Anonymous Markets", van Wegberg et al 2018; and "Enabling Learning in Resilient Adaptive Systems: From Network Fortification to Mindful Organising", Georgiadis2019 )

The Soska & Christin2015 dataset is available in a censored form publicly, and the uncensored dataset is available on request to qualified researchers via IMPACT. Similarly, there are anonymized & non-anonymized versions of their in-depth AlphaBay crawls used in 3 papers. The group notes

Upcoming data (as of July 2018): We are monitoring a number of other markets as of 2018. We expect to make this data available in 2019, with a six-month to one-year delay.

Tai et al 2019 ("Adversarial Matching of Dark Net Market Vendor Accounts") uses additional data from "Dream, Berlusconi, Valhalla, and Traderoute". Gãnán et al 2020 use the IMPACT dataset to study "Agora, Alphabay, BlackMarket Reloaded, Evolution, Hydra, Pandora, Silk Road 1 and Silk Road 2 from 2011 to May 2017, and consists of 44,671 listings and 564,204 transactions made on digital goods, grouped in 17 categories."

"Analysis of the supply of drugs and new psychoactive substances by Europe-based vendors via darknet markets in 2017-2018", Christin & Thomas2019 uses a rewritten crawler and does analysis of presumably the same dataset but gives a time period:

…we collected 35 scrapes of four markets-Dream Market, Traderoute, Valhalla, and Berlusconi Market-between summer 2017 and summer 2018

Digital Citizens Alliance (?)

Dolliver2015 (claimed NDA prevents sharing SR2 crawls, despite serious anomalies & absurd results in published analysis)

Hardy & Norgaard2015/Hardy2019: HTTrack-based scrapes of SR2 marijuana listings November 2013 to October 2014

Marin et al 2016

Janetos & Tilly2017

Celestini et al 2016 (AlphaBay/Nucleus/East India Company, monthly crawls for 4 months in 2015)

Project CASSANDRA: Wadsworth et al 2017, Wadsworth et al 2018 (22 DNMs, every 2 months from October 2015 to 2016)

"The Economic Functioning of Online Drug Markets", Bhaskar et al 2017 (CEP crawls: SR1 2013-08-01, SR2 2013-12-02-November 2014, Agora December 2013, Evolution January 2014, Nucleus November 2014)

Dittus et al 2017, "Platform Criminalism: The ‘Last-Mile’ Geography of the Darknet Market Supply Chain") (AlphaBay/Hansa/Traderoute/Valhalla, June-July 2017

"Dark Market Regression: Calculating the Price Distribution of Cocaine from Market Listings", David Everling (2017-07-14-2017-07-21, Dream Market; used in Chun2019)

"Sex, Drugs, and Bitcoin: How Much Illegal Activity Is Financed Through Cryptocurrencies?", Foley et al 2019 (active DNMs2016-2017)

"Challenging the techno-politics of anonymity: the case of cryptomarket users", Bancroft & Reid2017 (1 major but unspecified DNM forum, crawled March-May 2015 covering ~2 years of forum posts)

DATACRYPTO: Paquet-Clouston2016/Paquet-Clouston et al 2018 (AlphaBay: September 2015-February 2016); Martin et al 2018: "31 cryptomarkets in operation between 2013-09-12 and 2016-07-18, including all the largest English language sites (Alphabay, Nucleus, Dream market, Agora, Abraxas, Evolution, Silk Road 2 (Silk Road Reloaded), and SR1)."

Bakken et al 2017, "Coordination problems in cryptomarkets: Changes in cooperation, competition and valuation": SR2 seller profiles snapshot, 2014-09-15

Tzanetakis2018, "Comparing cryptomarkets for drugs. A characterisation of sellers and buyers over time" (AlphaBay: September 2015-August 2016)

Hutchings2018, "Leaving on a jet plane: the trade in fraudulently obtained airline tickets" (unknown "blackmarket", "December 2014 to August 2016", but probably using the Christin/Soska crawl and AlphaBay)

Hayes et al 2018, "A Framework for More Effective Dark Web Marketplace Investigations"

Hutchings2018, "Flying in Cyberspace: Policing Global Travel Fraud": unspecified large DNM 2014-2016 (Evolution?)

Rolando & Beccaria2018, "This place is like the jungle: discussions about psychoactive substances on a cryptomarket" (July 2016, single AlphaBay forum scrape)

Baravalle & Lee2018, "Dark Web Markets: Turning the Lights on AlphaBay" (June-September 2017 AlphaBay market scrapes)

Du et al 2018, "Identifying, Collecting, and Presenting Hacker Community Data: Forums, IRC, Carding Shops, and DNMs" (51 forums/13 IRC channels/12 DNMS (0day/Alphabay/Apple Market/Dream Market/French Deep Web/Hansa/Minerva/Russian Silk Road)/26 carding shops, 2016-2018)

Evangelista2018, "Darknet Markets: Competitive Strategies In The Underground Of Illicit Goods" (Berlusconi Market (2018-06-02, 2018-06-29-2018-06-30), Dream Market (2018-03-06-2018-04-06, 2018-06-30-2018-07-03, 2018-08-19-2018-08-20), Empire Market (2018-06-06), & Olympus Market (2018-06-06))

Rossy et al 2018, "Drogues sur Internet: Etat des lieuxsur la situation en Suisse" (Google Translate: "This is data collected by several police forces during the seizure and closing of two important crypto markets of the Silk Road 2.0 era and Pandora. The data on Swiss buyers was transmitted to us anonymously. The data concerns 724 purchases made between November 26, 2013 and August 12, 2014. 11 of them are from the Pandora platform, while the other 713 come from the Silk Road cryptomarket 2.0.")

Ball et al 2019, "Data Capture & Analysis of Darknet Markets", Australian National University’s Cybercrime Observatory; Apollon Market, 2018-12-17-2019-02-25.

Zhang et al 2019, "Your Style Your Identity: Leveraging Writing and Photography Styles for Drug Trafficker Identification in Darknet Markets over Attributed Heterogeneous Information Network": Valhalla/Dream Market, "weekly snapshots from June 2017 to August 2017"

Magnúsdóttir2019, "Darknet Drug Markets In A Swedish Context: A Descriptive Analysis Of Wall Street Market And Flugsvamp 3.0": "Wall Street Market and Flugsvamp 3.0, in March of 2019"

Wu et al 2019, "Python Scrapers for Scraping Cryptomarkets on Tor": one-off scrapes of 7 markets (Dream/Berlusconi/Wall Street/Valhalla/Empire/Point Tochka/Silk Road 3.1)

Espinosa2019, "Scamming and the Reputation of Drug Dealers on Darknet Markets": Hansa, March 2017

Červený & van Ours 2019, "Cannabis Prices on the Dark Web" (AlphaBay: first two weeks of "early October 2015")

Du et al 2019, "Identifying High-Impact Opioid Products and Key Sellers in Dark Net Marketplaces: An Interpretable Text Analytics Approach" (AlphaBay2016-2017 & Dream Market2016-2018)

Bradley2019, "On the Resilience of the Dark Net Market Ecosystem to Law Enforcement Intervention" (SR2, raided server images; Bradley received copies of the SR2 server data from an unspecified UK LE agency); Bradley & Stringhini2019, "A Qualitative Evaluation of Two Different Law Enforcement Approaches on Dark Net Markets" (Reddit datasets)

Yannikos et al 2019, "An Analysis Framework for Product Prices and Supplies in Darknet Marketplaces" ("Dream Market, Wall Street Market, and Tochka Market…We performed the data collection over a duration of a little more than 7 weeks, starting from September 18, 2018 until November 19, 2018")

Zheng et al 2019, "Identifying Hidden Buyers in Darknet Markets via Dirichlet Hawkes Process" (Dream Market, Wall Street Market, & Empire Market; single 2019 crawl?)

Bancroft et al 2019, "Producing Trust Among Illicit Actors: A Techno-Social Approach to an Online Illicit Market" (The Majestic Garden; selected forum posts, 2017-2018?)

Kwon et al 2020, "Knowledge Sharing Network in a Community of Illicit Practice: A Cybermarket Subreddit Case" (/r/AlphaBay scrape from a "cybersecurity firm", June 2016-July 2017)

Smith & Frank2020, "Dishing the Deets: How dark-web users teach each other about international drug shipments" ("Data collected for this research was obtained from two forums and one cryptomarket between the period of November 2017 and April 2018.": /r/DNM, Dread, & Dream Market respectively)

Ubbink2019, "Characterization of illegal dark web arms markets" (Berlusconi, weapon lists, May-June 2019)

Norbutas et al 2020, "Reputation transferability across contexts: Maintaining cooperation among anonymous cryptomarket actors when moving between markets" (aside from using this & Soska, "We collected data from AlphaBay in June and July 2017, shortly before the cryptomarket was seized.")

Arps & Christin2020, "Open Market or Ghost Town? The Curious Case of OpenBazaar" (OpenBazaar crawls: June 25, 2018-September 3, 2019)

Zhou et al 2020, "A Market in Dream: the Rapid Development of Anonymous Cybercrime" (Dream Market: 2018-10-30-2019-03-01)

Yang et al 2019, "Anonymous market product classification based on deep learning" ("In order to conduct research on the anonymous trading market, a one-month crawler was used, and anonymous market data was collected by OnionScan.")

Rhumorbarbe et al 2018, "Technical Note: Characterising the online weapons trafficking on cryptomarkets" ("Weapons related webpages from nine cryptomarkets were manually duplicated in February 2016…The selected markets are: Aflao marketplace (AFL), AlphaBay (ALB), Dr D’s multilingual market (DDM), Dream market(DMA), French Darknet (FRE), The Real Deal (TRD), Oasis (OAS), Outlaw market (OUT), Valhalla (aka Silkkitie) (VAL).")

Cheung2019, "‘We must work together for the good of all’: An examination of conflict management on two popular cryptomarkets" (Tochka Free Market (TFM)/Wall Street Market (WSM) forum posts, vendor profiles, reviews, and market rules, unspecified 2019 (?) date)

Veringmeier2019, "Repeat Buying Behavior of Illegal Drugs on Cryptomarkets" (single scrape, AlphaBay July 2017?)

Recon2020: announcement (Grams DNM search engine successor?)

Broadhurst et al 2020, "Fentanyl availability on darknet markets" ("Data were collected over 84 days (from 2 January to 2019-03-27) from 64 ‘scrapes’ of six omnibus darknet markets: Berlusconi, Dream Market, Empire, Tochka, Valhalla (‘Silkkitie’) and Wall Street.")

- Broadhurst et al 2021, "Impact of darknet market seizures on opioid availability" ("Data were collected over 352 days, from 2 January to 20 December 2019 [2019-01-02-2019-12-20] (excluding weekends), combining 251 scrapes from initially 8 darknet markets: Apollon, Empire, Dream, Nightmare, Tochka (also known as Point), Berlusconi, Valhalla (also called Silkitie), and Wall Street. In April three ‘new’ markets (Agartha, Dream Alt and Samsara) were added after Wall Street and Valhalla were seized by law enforcement and Dream voluntarily closed. In July Cryptonia was added as a substitute for Nightmare, which closed in an exit scam (where a business stops sending orders but continues to accept payment for new orders). Cryptonia operated until a planned (voluntary) closure in November.")

Ladegaard2020, "Open Secrecy: How Police Crackdowns and Creative Problem-Solving Brought Illegal Markets out of the Shadows"

To estimate the scale of encryption-signing, information hub activity, and seller migration, I downloaded and extracted data from key original sources using python and wget. For the encryption-signing analysis, I collected data from the discussion forums associated with 5 cryptomarkets: Silk Road and Silk Road 2; BlackMarket (another early cryptomarket); and the 2 largest cryptomarkets in 2014-2015: Agora and Evolution. I supplemented collected files with data from public archives (Branwen2016). For the analysis of information hub activity, I collected data from 3 market-independent forums, and visitor data from 2 additional websites were shared with me by their operators. Last, I collected data on post-intervention trade and seller migration from the 3 largest markets after Silk Road was shut down: Silk Road 2, Evolution, and Agora. I collected these data daily, from October 2014 until September 2015. Agora lasted throughout the period, but Silk Road 2 was shut down in early November 2014, and Evolution closed in medio March 2015. (Most of these data are available at darkdata.bc.edu or upon request.)

Silfversten et al 2020, "Exploring the use of Zcash cryptocurrency for illicit or criminal purposes" (uses DWO, RAND’s ongoing "Dark Web Observatory")

Turk et al 2020, "A tight scrape: methodological approaches to cybercrime research data collection in adversarial environments" ("Concretely, we have been crawling various cybercrime communities for more than four years, including web forums…We have scraped 26 forums (described in Table 1), around 300 chat channels across Discord and Telegram, and an archive of files.")

Lamy et al 2020, "Listed for sale: Analyzing data on fentanyl, fentanyl analogs and other novel synthetic opioids on one cryptomarket" (eDarkTrends scrape of Dream Market: 2018-03-22-2019-01-26)

Duxbury & Haynie2020, "The responsiveness of criminal networks to intentional attacks: Disrupting darknet drug trade" ("Data for our study come from one of the largest currently operating darknet drug markets, Silk Road 3.1. They contain information on 16,847 illicit drug transactions between 7,126 buyers and 169 vendors, representing the entire population of drug transactions on the Silk Road 3.1 during its first 14 months of activity")

Moeller et al 2020, "Illicit drug prices and quantity discounts: A comparison between a cryptomarket, social media, and police data" ("Data from Flugsvamp 2.0 was collected in collaboration with the DATACRYPTO project (see Décary-Hétu & Aldridge2015) between May and September in 2018, yielding 826 advertisements. Flugsvamp 2.0 provided specified categories for drug types and prices, but we also verified and coded them manually.")

Bradley2020, "Essays in Demand Estimation: Illicit Drugs and Commercial Mushrooms" (Agora, 2014-11-04-2015-09-05)

Barr-Smith & Wright2020, "Phishing With a Darknet: Imitation of Onion Services" (spidering of all Tor hidden services, May-July 2019)

Holt & Lee2020, "A Crime Script Analysis of Counterfeit Identity Document Procurement Online"

Samreen & Alafi2020, "Voting for Authorship Attribution Applied to Dark Web Data" (forums: DNM Avengers, The Majestic Garden, The Hub, Dread; ~2019-10-2019-12)

Bracci et al 2021, "Dark Web Marketplaces and COVID-19: before the vaccine" (2020-01-01-2020-11-16; Flashpoint Intelligence commercial crawls of Atshop, Black Market Guns, CanadaHQ, Cannabay, Cannazon, Connect, Cypher, DarkBay, DBay, DarkMarket, Darkseid, ElHerbolario, Empire, Exchange, Genesis, Hydra, MEGA Darknet, MagBO, Monopoly, Mouse In Box, Plati.market, Rocketr, Selly, Shoppy.gg, Skimmer Device, Tor Market, Torrez, Venus Anonymous, White House, Willhaben, Yellow Brick)

Crowder & Lansiquot2021, "Darknet Data Mining-A Canadian Cyber-crime Perspective" (early 2020-07: EliteMarket, Icarus, AESAN)

Brunelle et al 2021, "Introducing A Dark Web Archival Framework" (MITRE, ongoing?)

van Waardenberg2021, "Reputation in AlphaBay: the effect of forum discussions on the business success of cryptomarket sellers" (unpublished Dutch AlphaBay dataset)

Furumoto et al 2021, "Extracting Threat Intelligence Related IoT Botnet From Latest Dark Web Data Collection" (ASAP, DarkMarket, DarkFox; Dread; 2021?)

Maras et al 2023, "Keeping Pace With the Evolution of Illicit Darknet Fentanyl Markets: Using a Mixed Methods Approach to Identify Trust Signals and Develop a Vendor Trustworthiness Index" (2020-2022: Vice City, Versus, Cartel, ASAP)

Goonetilleke et al 2023, "Hydra: Lessons from the World’s Largest Darknet Market" (Hydra: 1 April 2020-2 May 2020; uses the Christin scrapes and an anonymous third-party scrape)

External Links

- Discussion:

- Reddit: 1, 2, 3, 4

- HN

"You Can Now Download a Copy of Pretty Much Every Dark Web Market Ever Made"

"There’s a searchable cache of the web’s darkest corners of the anonymous internet"

1. Internet Archive Upload Limits

Something that might be useful for those seeking to upload large datasets or derivatives to the IA: there is a mostly-undocumented ~25GB size limit on its torrents (as of mid-2015).

Past that, the background processes will no longer update the torrent to cover the additional files, and one will be handed valid but incomplete torrents. Without IA support staff intervention to remove the limit, the full set of files will then only be downloadable over HTTP, not through the torrent.