Download all contents of a Shared Link exceeding 15 GB
The US Department of Agriculture makes official data (specifically, the National Agricultural Imagery Program data, consisting of satellite imagery of the continental US) available only through the following Box.com Shared Link (also available from the USDA website, click on the big link for "National Ag. Imagery Program County Mosaic").
I want to download all the files inside a given directory, which exceeds 15 GB in size. I am looking for a way to *automatically* download these files, meaning, I don't want to sit there and manually download potentially hundreds of files. Non-solutions include WebDAV, which doesn't follow links to Shared Links, nor FTP, which is disabled for free users. I can't seem to easily get a list of URLs out of the box.com web single-page app, which I could then download in with wget or curl. If I disable JavaScript, Box.com will show me a list of files but not let me download any, saying "The requested page does not exist"—apparently JavaScript is required to download this public data.
I'm looking into the Box CLI—will it let me download a local copy of all files in Shared Links, or will it be another non-solution?
If Box can't enable straightforward downloading of US government public data, I'll have to lobby the USDA to use another service.
-
Hi ,
Welcome to the Box Community and thanks for your first post in the forum!
Have you checked out the BOX CLI here: https://developer.box.com/docs/box-cli
As for downloading content from Box, have you had more success downloading your content in parts? It may help to download subfolders at a time, but if you have more info, please share!
Be sure to read our guidelines, Subscribe to content you like, and complete your profile on the community.
-
Can the Box CLI download Shared Links? Or is it limited to only downloading your own Box files?
About downloading in parts: ~1 TB of data would require way too much labor to download manually, even fifteen gigs at a time. I'm really looking for an automatic solution where I can download all the contents of a leaf directory in the Shared Link I posted above.
-
I have the same problem but for a different reason. I uploaded thousands of pictures to share with family and friends, most in single folders exceeding 15 gb. One folder has ~4,000 pictures spread out over 40 pages as Box limits a page to 100 MAX.
Looks like I have to download 40 times???
Box reminds me of when Bill Gates limited RAM saying "640K ought to be enough for anybody."
Box is shortsighted hear - time to get another storage vendor who understands current customers needs.
Please sign in to leave a comment.
Comments
4 comments