Recursive file Copy - Python Box SDK

Answered

Follow

New post

shynee

• October 25, 2018 08:35

Hello,  I am new to box SDK. I need to copy millions of user files (using as_user). Here is how I understand it works with python SDK

 

• Check if a folder or sub-folder exists, if not create it. Return the folder_id for the next step
• Check if the file exists. If not copy files to the folder
• Apply retention policies (did not test this yet)

I did test and I can copy test files fine. But I have to copy large number of files. The process will be slow. It will take a long time to copy. Wondering if there is a better way to do this.

 

I need to repeat the copy process to move any changed files during cut-over.

 

0

• 

  mwiller

  • October 25, 2018 10:12

   As documented, the Folder Copy API is already recursive and will copy over the contents of the folder for you — is there a specific reason for wanting to copy files and subfolders individually?

  0

  Comment actions Permalink
• 

  shynee

  • October 25, 2018 10:52

  Thank you Willer. Totally Overlooked Copy Folder API(client.folders.copy(folderID, destinationFolderID)). I wanted to copy one by one and check SHA1 for file integrity.  But that could be a separate function. Appreciated.

  0

  Comment actions Permalink
• 

  shynee

  • October 25, 2018 11:05

  We have large Isilon Shares with millions of files. We are trying to move some of the files which were accessed in the last one year. That was the main reason for copying file by file. I did a test copy against around 250 files (total 18MB)  and it took 15 minutes. if I copy file be file, it will take months. Copying folders seems the way to go. Then apply expiration policy to take care of the files which are not accessed in a long time.

  0

  Comment actions Permalink
• 

  Mxolisi

  • December 11, 2019 01:25

  Hi  and , I don't know if this is possible but I want to copy files from a shared folder to a folder on my own account using a python script.
  
  Right now I can only get the metadata of the files and folders shared but I can't download them so I thought maybe I should go the copy route.
  
  Here is the function I'm using:
  
  

  def download_from_link(self, links😞

  extensions = ['jpg', 'jpeg', 'png']

  try:

       for link in links:

            _shared_folder = self.client.get_shared_item(link)

            print(_shared_folder.type)

             if _shared_folder.type == 'file':

                   # downloading file

             elif _shared_folder.type == 'folder':

                    self.client.folders.copy(_shared_folder.id,destination_id)

                    for _file in _shared_folder.get_items(limit=1000😞

   

                          if "Original Files" in _file.name:

                                for _file_ in _file.get_items(limit=1000😞

                                      ext = _file_.name.split('.')

                                      if ext[-1] in extensions:

   

                                      print(_file_.name)

                                      output_file = open(_file_.name, 'wb')

                                      print(output_file)

                                      download_file = self.client.file(_file_.id).get()

                                      self.client.file(_file_.id)

                                      download_file.download_to(output_file)

   

  I'm sorry about the Indentation I had to do it manually since I the text editor can't format a python script.

  0

  Comment actions Permalink

Please sign in to leave a comment.