transfer large amount of files

Status
Not open for further replies.

Zack Ibraham

New member
Jan 15, 2022
4
0
0
Visit site
I have about 3 million files and folders sized (240G) on a local PC . Once the main job is synced only a thousand or so files change every day.

What I am trying to do is to sync the entire contents to a webhost via ftp and keep it updated as the files change on my local PC.

So what I need , is to transfer only the changes that happens on the files in the same time it changed on the destination .

I tried using GoodSync for this but it takes 15 hrs to analyses and by that time a number of files have changed.

Is there any software or string of batch files that would do this for me and utilize multiple connections for a ftp username?

- Greetings
Zack
 

HelloNNNewman

Administrator
Moderator
Oct 30, 2012
2,761
87
48
Visit site
Hi Zack,

I've started replying to this a few times, but everything I look at isn't what you are describing. Most have the time issues you are experiencing with GoodSync.

Comparitech had an interesting article a while back about syncing using Box services. It covers a few different areas and may give you an idea or two (hopefully). Software like what you've tried has limitations as you have found. Dunno... there are a few things out there where people claim to do it with scripts, but those are something I can't point you to directly as they simply seem unsafe and/or not on point with what you are looking to achieve.
 

Phillip Remaker

New member
Dec 29, 2021
3
0
0
Visit site
Is there a way you can use something else other than FTP? I know that it'll always be fairly slow via FTP as it's hard to check so many files to see if they have changed.
Clouds like Onedrive, Google Drive , Azure blob or any of the big name offerings will be a good destinations .
And as per my experince with Onedrive , be careful using OneDrive on active files because it's designed for storage not realtime sync... If a user tries to access/change files that OneDrive is currently syncing then the file gets locked or corrupted , OneDrive should not be used in Lu of network devices as active file sharing between multiple users but if you want to sync between multiple stations for a single user i.e from desktop to laptop and back for taking work home it's very good .

Then depending on the cloud destination system you could use something like rsync , Carbonite , GsRichcopy360 or you can check again with GoodSync ,all work fine in Windows 11
 

Zack Ibraham

New member
Jan 15, 2022
4
0
0
Visit site
Thanks, HelloNNNewman and Phillip Remaker , Your helpful info led me to end up transferring the data with the backup tool Gs Richcopy 360 and using OneDrive as a destination. This seems to have worked
 
Status
Not open for further replies.

Members online

No members online now.

Forum statistics

Threads
322,736
Messages
2,242,597
Members
427,978
Latest member
Duouser3